After scanning through a bunch of C++ and SharpDX (which provides the bindings for .NET) examples I’ve finally managed to put all pieces together for a nice and easy to use VideoIn node (as well as its reactive counter-part in case one wants to process data in the video thread).
It outputs a B8G8R8A8 image which can easily be made available to other nodes from Skia or OpenCV.
Hardware acceleration can be turned on if needed. I didn’t see much of a difference though with a 720p stream, maybe it’s different with higher resolutions. Would be eager to get some feedback on that topic.
Once VL.Xenko/VL.Stride gets available we can also think about making the video frames directly available as textures (by-passing the CPU completely) because the used DirectX11 device for hardware acceleration can be fed from outside.
The ~2.5s you see when using the reactive version is because only when you hover over the pin the camera capture gets activated. The node is in that sense lazy. Other camera device nodes are patched in the same way. Reason being mainly about lifetime management.
I did however also notice the tooltip on the observable being a little sluggish and when looking in the task manager the CPU usage was much higher than going the manual route through DrawImage/Renderer - not sure what’s the reason for that, maybe something is off in the tooltip itself…
I’ll definitely also have a look at normal video playback. Apart from having to install GStreamer it also never was very stable. So maybe Media Foundation is easier to handle in that regard. We’ll see.
Something that I tried to test a while back, but I couldn’t get to work with VL at the time was this library, essentially a cross-platform rewrite of media foundation based on FFMPG, but you were exploring Gstreamer so it seem unnecessary to discuss it. https://github.com/unosquare/ffmediaelement.
hei there, tried to export an app using that nuget. the patch works fine (nothing fancy, just a VideoIn drawn on a rectangle) but soon as I try to run the exported app, it throws this exception :
System.IO.FileNotFoundException: Could not load file or assembly 'SharpDX.MediaFoundation, Version=220.127.116.11, Culture=neutral, PublicKeyToken=b4dcf0f35e5521f1' or one of its dependencies. The system cannot find the file specified.
I manually added SharpDX.MediaFoundation and SharpDX dlls in the lib folder next to the .exe (it would complain for both), the app starts but it shows a blank rectangle instead of my webcam feed (works fine in the editor).
here’s a screenshot of the patch i’m exporting … nothing fancy :)
Ah yes indeed, I think I can fix SharpDX not being referenced. Regarding the other error, you sure you stopped the patch? Access to camera is exclusive what I know. At least here it would show a white quad as long as the patch was still running.
Do you have multiple devices? Maybe it’s related to the dynamic enum I wonder. I just uploaded a new package (0.0.3) which should fix the reference issue. Maybe some other file was missing? Hard to say why you don’t get anything, it worked for me having one camera.