VL.MediaFoundation

Microsoft Windows has a lot of multimedia capabilities built-in which can be accessed through the Media Foundation SDK (https://docs.microsoft.com/en-us/windows/win32/medfound/about-the-media-foundation-sdk).

After scanning through a bunch of C++ and SharpDX (which provides the bindings for .NET) examples I’ve finally managed to put all pieces together for a nice and easy to use VideoIn node (as well as its reactive counter-part in case one wants to process data in the video thread).
It outputs a B8G8R8A8 image which can easily be made available to other nodes from Skia or OpenCV.
Hardware acceleration can be turned on if needed. I didn’t see much of a difference though with a 720p stream, maybe it’s different with higher resolutions. Would be eager to get some feedback on that topic.

Once VL.Xenko/VL.Stride gets available we can also think about making the video frames directly available as textures (by-passing the CPU completely) because the used DirectX11 device for hardware acceleration can be fed from outside.

The repository and install instructions can be found here: https://github.com/vvvv/VL.MediaFoundation

10 Likes

Worked out of the box using a Kinect V2
image

Use Hardware Acceleration doesn’t seem to make any difference.

When using the reactive version it takes a while (~2,5 s) before the image is shown in the tooltip.

Maybe the tooltip could show the current FPS?

1 Like

would also make a nice videoplayer, no?

@bjoern
The ~2.5s you see when using the reactive version is because only when you hover over the pin the camera capture gets activated. The node is in that sense lazy. Other camera device nodes are patched in the same way. Reason being mainly about lifetime management.
I did however also notice the tooltip on the observable being a little sluggish and when looking in the task manager the CPU usage was much higher than going the manual route through DrawImage/Renderer - not sure what’s the reason for that, maybe something is off in the tooltip itself…

@sebl
I’ll definitely also have a look at normal video playback. Apart from having to install GStreamer it also never was very stable. So maybe Media Foundation is easier to handle in that regard. We’ll see.

Something that I tried to test a while back, but I couldn’t get to work with VL at the time was this library, essentially a cross-platform rewrite of media foundation based on FFMPG, but you were exploring Gstreamer so it seem unnecessary to discuss it.
https://github.com/unosquare/ffmediaelement.

On closer inspection

Target Framework must be 4.6.1 or above, or .Net Core 3.0 or above.

so I’m not sure if that excludes Gamma at the moment

Just uploaded version 0.0.2 which adds VideoPlayer and VideoPlayer (Reactive) nodes.

3 Likes

hei there, tried to export an app using that nuget. the patch works fine (nothing fancy, just a VideoIn drawn on a rectangle) but soon as I try to run the exported app, it throws this exception :

System.IO.FileNotFoundException: Could not load file or assembly 'SharpDX.MediaFoundation, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b4dcf0f35e5521f1' or one of its dependencies. The system cannot find the file specified.

(full stack here)

I manually added SharpDX.MediaFoundation and SharpDX dlls in the lib folder next to the .exe (it would complain for both), the app starts but it shows a blank rectangle instead of my webcam feed (works fine in the editor).

any hints?

here’s a screenshot of the patch i’m exporting … nothing fancy :)

image

cheerz!

Ah yes indeed, I think I can fix SharpDX not being referenced. Regarding the other error, you sure you stopped the patch? Access to camera is exclusive what I know. At least here it would show a white quad as long as the patch was still running.

hey elias, thanks for the fix and yep, the patch was stopped. the small LED next to my webcam was off when I launched the .exe, and never turned on.

Do you have multiple devices? Maybe it’s related to the dynamic enum I wonder. I just uploaded a new package (0.0.3) which should fix the reference issue. Maybe some other file was missing? Hard to say why you don’t get anything, it worked for me having one camera.

I can confirm exported applications works fine with 0.0.3 ! Also for some reason my webcam is now showing up there.

Thanks again for the prompt fix !

Works fine here too, in editor and after being exported. Only with Preview Gamma version, adding reference get stuck and never finish but nuget packet is fine.

As of 2021.4 the VL.Video.MediaFoundation nuget is now shipping with vvvv. And there are even more options for video playback.