Hey, I just played around with gamma and my (very) old code for making generated sounds in a fresh, strange, (silly?) way… ;-)
I recently came up with Dataflow (TPL), probably the best library your not using.
It is basically a set of few classes to create dataflow systems with .NET/C#. Sounds like a good fit considering VL is dataflow on steroids.
Now, sound (especially realtime sound) is a bit awkward, since you notice every framedrop of more then 5ms - maybe even fewer.
To counter that, you normally use buffers (the bigger, the fewer framedrops), but so you introduce lag - which is equally annoying for realtime use.
However, for me the weirdest part is, the common way to solve all those problems. What you normally do, is to tell the soundcard to call you when more sound data is needed. Then your code must deliver the new sound, as fast as possible (and you will hear it immediately when you fail).
So, when you design a graph based sound system, every node in the system needs to keep track of timings and have own small buffers to prevent stutters and noises. And you always calculate from the sound output back to the start - not in the natural way of thinking.
Why I am writing about this?
I tried to implement a sound system in VL using TPL, based on my older experiments (VL.NewAudio). And it looks very promising so far.
What it does is building a directed graph based on nodes you place in VL, where you can push small audio buffers into, from generating nodes (ie sound inputs) to the sound outputs. Every node in this graph takes a buffer in, does some calculations and hands the buffer to next node. Every node is a Task and can run in parallel, on different cores if needed.
Okay, if you read this far, great you can try it out (VL.NewAudio 0.3.1-alpha). It works so far (and has some quite some nice features like FFT and IFFT and many many more bugs). But I currently cant figure out, what happens in some strange and rare cases, when this stops working.
It looks like out of nowhere, the whole dataflow system takes a break of several milliseconds, which breaks everything.
Here is what it looks like:
[50][BlockingSampleProvider35] READ 0 1024, buffer: 5120, w=2048, r=13312
[50][AudioGenerator29] ACTION Received Request for 1024 samples
[50][AudioFlowSource] Offer Message 1, False -> Accepted
[13][AudioFlowSource] receiving 1024 0, buffered: 0
[13][AudioFlowSource] OnDataReceived 1024 at 66560
[13][AudioFlowSource] OnDataReceived 1024 at 66816
[13][SinGen23] Received 512 at 66560 in=1 out=0
[13][SinGen23] Received 512 at 66816 in=0 out=1
[50][BlockingSampleProvider35] READ 0 1024, buffer: 4096, w=2048, r=14336
[50][AudioGenerator29] ACTION Received Request for 1024 samples
[50][AudioFlowSource] Offer Message 1, False -> Accepted
[50][BlockingSampleProvider35] READ 0 1024, buffer: 3072, w=2048, r=15360
[50][AudioGenerator29] ACTION Received Request for 1024 samples
[50][AudioFlowSource] Offer Message 1, False -> Accepted
[50][BlockingSampleProvider35] READ 0 1024, buffer: 2048, w=2048, r=0
[50][AudioGenerator29] ACTION Received Request for 1024 samples
[50][AudioFlowSource] Offer Message 1, False -> Accepted
[50][BlockingSampleProvider35] READ 0 1024, buffer: 1024, w=2048, r=1024
[50][AudioGenerator29] ACTION Received Request for 1024 samples
[50][AudioFlowSource] Offer Message 1, False -> Accepted
[50][BlockingSampleProvider35] READ 0 1024, buffer: 0, w=2048, r=2048
[50][AudioGenerator29] ACTION Received Request for 1024 samples
[50][AudioFlowSource] Offer Message 1, False -> Accepted
[50][BlockingSampleProvider35] Underrun, requested 1024, actual 0
[50][BlockingSampleProvider35] READ 0 1024, buffer: 0, w=2048, r=2048
[50][BlockingSampleProvider35] READ 0 1024, buffer: 0, w=2048, r=2048
[50][BlockingSampleProvider35] READ 0 1024, buffer: 0, w=2048, r=2048
[50][BlockingSampleProvider35] READ 0 1024, buffer: 0, w=2048, r=2048
[50][BlockingSampleProvider35] READ 0 1024, buffer: 0, w=2048, r=2048
[50][BlockingSampleProvider35] READ 0 1024, buffer: 0, w=2048, r=2048
[50][BlockingSampleProvider35] READ 0 1024, buffer: 0, w=2048, r=2048
[50][BlockingSampleProvider35] READ 0 1024, buffer: 0, w=2048, r=2048
[50][BlockingSampleProvider35] READ 0 1024, buffer: 0, w=2048, r=2048
[50][BlockingSampleProvider35] READ 0 1024, buffer: 0, w=2048, r=2048
[50][BlockingSampleProvider35] READ 0 1024, buffer: 0, w=2048, r=2048
[50][BlockingSampleProvider35] READ 0 1024, buffer: 0, w=2048, r=2048
[27][AudioFlowSource] receiving 1024 0, buffered: 0
[27][AudioFlowSource] OnDataReceived 1024 at 67072
[27][AudioFlowSource] OnDataReceived 1024 at 67328
[27][AudioFlowSource] receiving 1024 0, buffered: 2
[27][AudioFlowSource] OnDataReceived 1024 at 67584
[27][AudioFlowSource] OnDataReceived 1024 at 67840
[27][AudioFlowSource] receiving 1024 0, buffered: 4
[27][AudioFlowSource] OnDataReceived 1024 at 68096
[27][AudioFlowSource] OnDataReceived 1024 at 68352
[27][AudioFlowSource] receiving 1024 0, buffered: 6
[27][AudioFlowSource] OnDataReceived 1024 at 68608
[27][AudioFlowSource] OnDataReceived 1024 at 68864
[27][AudioFlowSource] receiving 1024 0, buffered: 8
[27][AudioFlowSource] OnDataReceived 1024 at 69120
[27][AudioFlowSource] OnDataReceived 1024 at 69376
[27][SinGen23] Received 512 at 67072 in=9 out=2
[27][SinGen23] Received 512 at 67328 in=8 out=3
[27][SinGen23] Received 512 at 67584 in=7 out=4
[27][SinGen23] Received 512 at 67840 in=6 out=5
[27][SinGen23] Received 512 at 68096 in=5 out=6
[27][SinGen23] Received 512 at 68352 in=4 out=7
[27][SinGen23] Received 512 at 68608 in=3 out=8
[27][SinGen23] Received 512 at 68864 in=2 out=9
[27][SinGen23] Received 512 at 69120 in=1 out=10
[27][SinGen23] Received 512 at 69376 in=0 out=11
[5][AudioFlowSink] Offer Message 262, True -> Accepted
[5][AudioFlowSink] Offer Message 263, True -> Accepted
[41][AudioFlowSink] receiving 512 66560, buffered: 0
[41][AudioFlowSink] Received 512 at 66560, buf w=2560 r=
[41][AudioFlowSink] Received 512 at 66560, buf w=2560 r=2048 source count 0
[41][AudioFlowSink] receiving 512 66816, buffered: 0
[41][AudioFlowSink] Received 512 at 66816, buf w=3072 r=2048 source count 0
[5][AudioFlowSink] Offer Message 264, True -> Accepted
In the first couple of lines you see it working properly. The BlockingSampleProvider reads and write from/to a ring buffer (and does not block anymore anything) that is fed into the sound card, AudioGenerator generates proper timed silence, SinGen adds some waves. You also can see the used block sizes (512 samples for 2ch audio).
Then, suddenly, the processing stops.
The BlockingSampleProvider still requests more sound, those messages are accepted, but you now can just see the ring buffer size shrinking, until its empty and you will hear it… It then recovers really fast, as soon as the dataflow starts again, however, that cracking noise has already been produced.
This happens very seldom, which is a bit frustrating - having a system which can produce sound in real time, using super short buffers and with (almost) any blocking code. But the breaks now and then, are inacceptable and make the whole concept useless.
I already monitored the GC, because that was my first guess (coming from the java world, its ALWAYS GC’s fault). But no, GC does nothing here.
I also checked several times the last occurance of a lock statement (for reusing audio buffers, to minimize GC work) - in fact its disabled for now, because not really needed. The whole thing needs barely any CPU or memory, even with multiple FFTs and remixing channels, I cant reach noticable load (okay, not really forced it).
Could it be, that some kind of I/O can make the complete .NET VM stop all (or most) threads? Does VL recompiling or optimizing parts of patches? Sometimes I can see loading HDE tooltip influencing the render thread, but this does not always lead to any sound problems.
Any ideas?
I’ve attached a full log (bit outdated) for anyone interested. VL.NewAudio.zip (60.2 KB)