for a commercial job I’ve found a very interesting Soundfingerprinting Library (like Shazam - you give it a piece of audio, it tells you from which track it is). Of course like many other opensource libs out there, this one has a very thin documentation. What can be patched very easily is working with the files, but when it comes to the live realtime data, the documentation fails short. I’ve asked the Author of the lib on the Github about it and he answered immediately with a short and understandable example.
Now comes the first question:
is it possible to patch this example in VL or are there some features in VL that are missing to do so?
The second question:
How can I collect samples in this BlockingCollection?
My Idea is to collect samples from the microphone into a fixed length (about 2 sec) Queue and feed the whole queue (every second or so) into the lib to see if there will be a match. My question is not Audio-related, but how to patch such a queue in Async scenario (I can’t just use the Queue node, right?) and how to supply the samples from this queue into the BlockingCollection the lib needs.
For the realtime capture from the Microhone I’m patching directly with NAudio C# nuget. The samples (just float numbers) are arriving via an observable.
Here is an example from the Lib I want to patch:
And here is my question and the answer from the Author on the GitHub:
Of course, once the thing is working I’ll wrap the library for VL and upload it as a nuget. So hopefully I’ll open a WIP topic pretty soon.
this is probably possible to patch, but if time is short, I would just wrap the C# code in some static methods and import them.
if you want to patch everything, I would start with creating one AsyncTask region per C# method. This provides the CancellationToken and does the same as a Task. Note that you have things like Task in Task here, so it’s region in region and such things.
BlockingCollection is also available in VL.
If you have a running patch with a demo setup it’s easier to suggest improvements.
thank you for your answer.
Unfortunately I don’t understand how to patch this code, so I took your advice and started to (blindnessly) wrap every method in its own AsyncTask region. I don’t understand how should Task.Run work inside of another Task and how to do Task.WhenAny and so on. And also the execution order, how to ensure, that we first put samples in the BlockingCollection (in GetRealtimeData) and then call QueryRealtimeData. I understand it in non-async scenario, but not here.
Unfortunately I don’t have a running patch with a demo setup. I have now only the capturing part, but it relates to the second part of my question about how to queue samples. It will then go into GetRealtimeData, as far as I understand. Let’s do it in the second step.
So for now I have a VL doc with the referenced c# nuget.
Here is the screenshot and the VL doc is attached.
I guess 1. and 2. can be easily joined into one ForEach.
What I’m missing now are cancellation tokens, but I don’t know how important they are.
Unfortunately I can’t just test it easily, so just from the patch point of view, how this translation from the code to patch looks like? What is wrong and what is the way to go?
If you use reactive nodes instead of tasks, the patch is not async anymore. what .NET tasks with async/await are doing is that they do not block the call stack.
the reactive chain you patched is fully blocking, there are no tasks introduced anywhere. so you probably want to put a ToBackground node into your observable pipeline where the C# code had an await keyword, to not block the observable from upstream.
especially after the region that gathers the samples (first one in your patch), because you don’t want to block the audio thread with the workload of the song analysis.
Ok, I see. Thank you for clarification.
The samples are arriving from NAudio using the DataAvailable Event, which is getting converted into Observable by VL already. Right after that I should add ToBackground, right?
so here is the test setup for the listening for the samples and making a realtime query for the fingerprinting.
The patch includes 3 ways I’ve tried it:
Query is a static: c# method in dll
Query is patched: not reactive and just got the data from the HoldLatest
Query is patched: reactive
All three approaches fail.
The first two fail like that: as soon as I’m firing them, the vvvv goes into the pause mode with nothing pink and no errors.
The last one - fully reactive, but CPU goes incrementally high, on every observable that is pushed a new query is started (and fails), but stays running (that’s how I understand it).
Unfortunately I don’t understand how to debug this in Gamma and what I’m doing wrong.
Attached is the ZIP with the VL doc, 2 dlls (one for the static query method, another one for the resampler), test WAV file. When you start VL, the missing nuget dependencies should be installed.
maybe don’t put it right after the samples arrive, it could mess up the order of samples. rather when you are finished gathering a block of samples and you want to process them with the sound fingerprinting library.
the operation of queuing a block of samples should be quite fast because it doesn’t do any heavy calculation. so it shouldn’t block the audio input thread for too long. also, the audio in event can have a very high rate, creating a task per event is probably less performant than doing it synchronously.
the process of finding an audio match is clearly something else and could take much longer, so this is a good place to make the cut between the threads.
in your patch the ToBackground is mostly on the the end, which doesn’t do anything, it doesn’t influence the observable above, it only has an effect on workload that happens after the node in the observable.
So in your case the ToBackground should be after ProcessAndDownsample.
do you know the Debug node in reactive? it gives you information about how often an observable has fired and on which thread ID. this could help here to understand whats going on.
thank you for the explanation and the hint.
Yes, I was trying to place the ToBackground all over the place without any success, now it’s under the ProcessAndDownsample.
Below is the updated VL doc. Also the ForEach in ProcessAndDownsample is reworked a bit. As it strangely didn’t counted up the “Count” pad, but it was doing so, when everything was in the Application and not in the Processnode. Unfortunately I couldn’t replicate this in the simple scenario, so no bug report here.
I’ve placed the Debug node right at the end, after the Query.
Looks not so helpful, maybe because the vvvv pauses before something happens?
As soon as the first observable is pushed from the ProcessAndDownsample gamma goes into the Pause.
The same happens in the first scenario where I’m holding the arriving Observables by HoldLatest and the trigger the static C# method manually or by the bang coming from the HoldLatest.
I’ve checked your patch and there have been some bugs in the audio preparation:
in the ToMono there was some mixup between float and byte count.
and when adding the samples, it seems you have to call CompleteAdding to allow the fingerprinting to read from the collection. not 100% sure on that, but this patch seems to do something now:
Cool, thank you @tonfilm.
Will investigate further and report.
Thanks for the ToMono bug. Made a wrong link there.
I’ve seen, that you’ve replaced the FromSequence with GetInternalArray when converting a Spread into the MutableArray. So is this the way? What is the difference between those two in this case?
Wow, I’ve never thought of using FrameDelay inside the ForEach (Reactive) in order to reset the Counter.
yes, it avoids a copy of the data because it directly accesses the data in the spread. and since the spread is immutable and you will only read, everything is fine.
just wanted to report, that the lib is working.
There was one more minor bug in the ToMono, and I’ve reverted some changes you’ve made to the enqueuing part, because I want to have a continuous “queue” (maybe it’s a wrong term) where I’m pushing new samples, removing the older ones, but the queue is never cleared (only manually on bang), like the normal vvvv queue.
I’ll post the VL doc and also step by step walkthrough of the converting the Async C# sample code into the Observables. A bit later, as I have to fight further for the project.
Will not mark this thread as Solved. Will do so, when I’ll post the documentation of the findings.
what I’m still failing to understand:
In the example C# code, the QueryRealtimeData is made as a long running task. It looks at the BlockingCollection and processes the items, but the task itself doesn’t end, it’s still sitting there and looking at the collection. The task ends only if the cancellation token is set or if the BlockingCollection isCompleted.
In the patches above, ForEach new item in the Observable a new Command is created and the Query is executed. Even if I’ll move the creation of the Command outside of the ForEach and put it on the Create and leave only the call to the Query method in the ForEach, still, the Query will be called for each pushed value.
How is it possible to patch the long running task, which sits there and works on the items in the BlockingCollection?
In your first reply you’ve suggested:
I would start with creating one AsyncTask region per C# method. This provides the CancellationToken and does the same as a Task.
But AsyncTask says: ‘Runs the given task once in a background thread’, so it’s not about long running task, right?
the region doesn’t know how long it should run, it depends on when the code inside returns. for example, if you wait for other tasks inside the AsyncTask region, it will finish whenever the awaited task finishes.
