Thanks to the new (super cool) .NET libraries vl’s feature I try to use the libreRealsense library. The lib seems to works great with vl. I’ve access to streams, few options and functionality. My issue is I can’t pass the depth stream to vvvv in his orginal/similar format “Z16” . For now I have to use the “colorized” depth texture instead, which ends up to vvvv in “R8G8B8A8_UNorm”. I guess this colorized depth texture is designed for visualisation and not to transmit the maximum depth resolution possible.
seems like you put a step on unknown terrain. pointclouds and vl is a combination I haven’t come across yet. this could just as well be a WIP and not a Question (ironically there is no explicit question, not lost on me).
care to share some patches, so the forum can come along for the ride?
Thanks for interest. I link the patch. The pointcloud is generated in vvvv by pointcloudbuffer for kinect node …which is actually coming from your team?
Sorry my question is if it’s possible to pass the “depthframe” from vl to vvvv in its orignal format of 1 channel of 16bit (like kinect’s depth texture) ? The Toimage openCv node seems to be incompatible with 16bit format.
It’s my first shot with vl and it’s dirty patched. Often it needs to restart vvvv after modifying the vl patch and it needs to be refreshed once to fill “create” fields in vl patch.
Also the patch is not linked with a real device but with a playback file provided by the library. Streams coming from it are slow but I guess it will perform at normal speed with a real device, as this playback file behaves the same in the Realsense Viewer sofware. Hope at least, let’s keep the surprise for later.
It requires vvvv36x64, dx11.pointcloud, opencv package for vl and to add OpenCvSharp.dll assembly (from vvvv_50beta36_x64\lib\packs\OpenCvSharp3-AnyCPU.3.4.1.20180319\lib\net46)
Patch
Playback file can be find here it’s the stairs one
please have a look at my take on getting depth out of the RealSense, I’ve simplified a bit your example, so now it works without OpenCV and DX11.PointCloud.
For now I’m creating a DynamicBuffer, where 16bit depth values are packed into 32bit integers (thanks to @tonfilm) and the shader picks individual depth values from there.
The straight way of getting Z16 texture directly out of VL is still to come and the things will be even more easy in the future.
I’m not a shader-guru, probably it can be done way more efficiently, but it is not the point.
The result looks like yours and I think everything is ok with the Depth resolution. At least the output looks much like in the RealSense Viewer app from the SDK.
Thank you very much it works great! Thanks for made it understandable by commenting and for cleaning the patch too.
I added the xy coordinates adjustment according to the camera’s intrinsics and put your shader code into “bsprite” shader from Instance Noodles.RS-08_05_2018.zip (2.4 MB)
I will let you know how it behaves with a real device when I’ll receive mine at the end of the month.
Hi, after over a one month delivery delay and few struggles, I finally have a decent pointcloud. The patch is running with two d430 modules set at 848x480 90fps without slowing down as it was with the playback file. The main issue was the two cameras came out of calibration and needed to be recalibrated using realsense calibration software. It’s quick and easy to use but I didn’t achieve to not get small bends on edges of the pointcloud. Have to manually correct it in vvvv. On vvvv side the previous patch actually picked the wrong corresponded stream’s intrinsics. Also it’s running smooth only if node “Perfmeter2” from Microdee is connected. No idea why Realsense_d430_pointcloud.zip (37.2 KB) (Need to add Intel.RealSense.dll to vl assemebly)
Cheers
Thanks for sharing! Cloud is working, the Root_d430_1 node is red - Inside the VL patch the Async Loop and WaitForFrames are flashing purple. And the Output Pin from “ToBufferDescription” is red.
TTY: “00:27:49 ERR : Exception occured in TMPluginWrapperNode.Evaluate
00:27:49 ERR : One or more errors occurred.”
OK, color me clueless, but I have not (yet) fooled around with VL, and the DX11 nodes are red. Where the heck is the VL DX11 pack? I have a nice new D430 and am anxious to give it a spin!
Answering my own problem, and maybe @schlonzo too. I was using 50beta35.8; 50beta36 seems to have the missing DX11 nodes.
And just to make it explicit, you need to install the RealSense SDK, and copy two files from the SDK install folder C:\Program Files (x86)\Intel RealSense SDK 2.0\bin\x64, Intel.Realsense.dll and realsense2.dll, into the rs_assembly folder createdby the zip file.
It then works for me, but the VVVV.d430.vl node is still red; inside I see the following red lines, but I do not know enough (yet!) about VL to diagnose it:
Edit: And yes I am seeing the same halts in display without Perfmeter2. Something to do with those error lines in the wait for frames area?
Edit2: If I open the Root_d430 patch, and move my mouse around in it, the display updates smoothly as long as I move my mouse! Whaddupwiddat?
Hi, as a vl beginner I’m not sure but dark red/brown as on @mediadog screenshot seems to stand up for nodes assign to “destroy”, not for errors. The vl node may stay red while the patch is working because of “pin is marked as hidden” on “TobufferDescription” node, if so you can right click on the node->open->middle click on Output pin (text not box)->check visible. But this error doesn’t seem to affect the patch anyways.
No ideas about vl patch messing when open it. I have similar issue when I use two cameras. Once it’s setup I restart it and let it close. @schlonzo is it running smooth in realsense viewer? the camera is sometimes recognize as usb2 instead of usb3. Also the patch is set for d430 module, stream’s index, available resolution/fps might change with other models
could you give me some pointers on how to stream out the position data PosW as a dynamic buffer, so that I can directly feed it onto a pointcloud emitter? Or what would be the most efficient way of emitting these within a dx11.particles system?
I know it’s an old topic but did you guys managed to get it working properly? I’m trying to work with a D435 and it only works for a few seconds than freezes. I’m guessing i’ts some kind of memory leak but can’t find it so far.
