Forum

Realsense depth stream z16

vl

#1

Hello

Thanks to the new (super cool) .NET libraries vl’s feature I try to use the libreRealsense library. The lib seems to works great with vl. I’ve access to streams, few options and functionality. My issue is I can’t pass the depth stream to vvvv in his orginal/similar format “Z16” . For now I have to use the “colorized” depth texture instead, which ends up to vvvv in “R8G8B8A8_UNorm”. I guess this colorized depth texture is designed for visualisation and not to transmit the maximum depth resolution possible.

Here is the pointcloud generate from the colorized depth texture
https://imgur.com/BiKFeUa

Here is how the depth stream travel to vvvv
https://imgur.com/JuW3Frm
https://imgur.com/nJ8yGQg
https://imgur.com/ZQfV2FR

Thanks


#2

seems like you put a step on unknown terrain. pointclouds and vl is a combination I haven’t come across yet. this could just as well be a WIP and not a Question (ironically there is no explicit question, not lost on me).

care to share some patches, so the forum can come along for the ride?


#3

Thanks for interest. I link the patch. The pointcloud is generated in vvvv by pointcloudbuffer for kinect node …which is actually coming from your team?

Sorry my question is if it’s possible to pass the “depthframe” from vl to vvvv in its orignal format of 1 channel of 16bit (like kinect’s depth texture) ? The Toimage openCv node seems to be incompatible with 16bit format.

It’s my first shot with vl and it’s dirty patched. Often it needs to restart vvvv after modifying the vl patch and it needs to be refreshed once to fill “create” fields in vl patch.
Also the patch is not linked with a real device but with a playback file provided by the library. Streams coming from it are slow but I guess it will perform at normal speed with a real device, as this playback file behaves the same in the Realsense Viewer sofware. Hope at least, let’s keep the surprise for later.

It requires vvvv36x64, dx11.pointcloud, opencv package for vl and to add OpenCvSharp.dll assembly (from vvvv_50beta36_x64\lib\packs\OpenCvSharp3-AnyCPU.3.4.1.20180319\lib\net46)

Patch
Playback file can be find here it’s the stairs one


#4

Hi @david_sh,

please have a look at my take on getting depth out of the RealSense, I’ve simplified a bit your example, so now it works without OpenCV and DX11.PointCloud.

For now I’m creating a DynamicBuffer, where 16bit depth values are packed into 32bit integers (thanks to @tonfilm) and the shader picks individual depth values from there.

The straight way of getting Z16 texture directly out of VL is still to come and the things will be even more easy in the future.

I’m not a shader-guru, probably it can be done way more efficiently, but it is not the point.

The result looks like yours and I think everything is ok with the Depth resolution. At least the output looks much like in the RealSense Viewer app from the SDK.

Hope this helps a bit.

p.s. Unpack the ZIP into your example.
RealSense - DepthToBuffer.zip (18.5 KB)

Best,
Anton


#6

Thank you very much it works great! Thanks for made it understandable by commenting and for cleaning the patch too.

I added the xy coordinates adjustment according to the camera’s intrinsics and put your shader code into “bsprite” shader from Instance Noodles.RS-08_05_2018.zip (2.4 MB)

I will let you know how it behaves with a real device when I’ll receive mine at the end of the month.

Thanks again
David


#7

Hi, after over a one month delivery delay and few struggles, I finally have a decent pointcloud. The patch is running with two d430 modules set at 848x480 90fps without slowing down as it was with the playback file. The main issue was the two cameras came out of calibration and needed to be recalibrated using realsense calibration software. It’s quick and easy to use but I didn’t achieve to not get small bends on edges of the pointcloud. Have to manually correct it in vvvv. On vvvv side the previous patch actually picked the wrong corresponded stream’s intrinsics. Also it’s running smooth only if node “Perfmeter2” from Microdee is connected. No idea why
Realsense_d430_pointcloud.zip (37.2 KB) (Need to add Intel.RealSense.dll to vl assemebly)
Cheers


#8

Thanks for sharing! Cloud is working, the Root_d430_1 node is red - Inside the VL patch the Async Loop and WaitForFrames are flashing purple. And the Output Pin from “ToBufferDescription” is red.

TTY: “00:27:49 ERR : Exception occured in TMPluginWrapperNode.Evaluate
00:27:49 ERR : One or more errors occurred.”


#9

if something is red in vl, you get error messages. what are they saying?
also which vvvversion are you using?


#10

50beta36. in VL within “Order” there are two red outlined objects:

State Output: Root_d430_1[VVVV.Rs] This process definition has errors.
Update: VL.Model.Fragment(99836) The fragment has errors

within “Members” the Destroy Object is red.

Were you referring to theseor is there a seperate tty for VL?


#11

OK, color me clueless, but I have not (yet) fooled around with VL, and the DX11 nodes are red. Where the heck is the VL DX11 pack? I have a nice new D430 and am anxious to give it a spin!


#12

Answering my own problem, and maybe @schlonzo too. I was using 50beta35.8; 50beta36 seems to have the missing DX11 nodes.

And just to make it explicit, you need to install the RealSense SDK, and copy two files from the SDK install folder C:\Program Files (x86)\Intel RealSense SDK 2.0\bin\x64, Intel.Realsense.dll and realsense2.dll, into the rs_assembly folder createdby the zip file.

It then works for me, but the VVVV.d430.vl node is still red; inside I see the following red lines, but I do not know enough (yet!) about VL to diagnose it:

Capture-1

Capture-2

Edit: And yes I am seeing the same halts in display without Perfmeter2. Something to do with those error lines in the wait for frames area?

Edit2: If I open the Root_d430 patch, and move my mouse around in it, the display updates smoothly as long as I move my mouse! Whaddupwiddat?


#13

Hi mediadog,

did you download all nugets and add the according dependencies within VL? (click on VVVV.d430.vl -> Dependencies -> Manage Nugets -> Commandline

“nuget install VL.OpenCV -prerelease”

After that I have to manualla add the realsense.dll unter Dependencies -> Files.
voila


#14

Hi, as a vl beginner I’m not sure but dark red/brown as on @mediadog screenshot seems to stand up for nodes assign to “destroy”, not for errors. The vl node may stay red while the patch is working because of “pin is marked as hidden” on “TobufferDescription” node, if so you can right click on the node->open->middle click on Output pin (text not box)->check visible. But this error doesn’t seem to affect the patch anyways.
No ideas about vl patch messing when open it. I have similar issue when I use two cameras. Once it’s setup I restart it and let it close.
@schlonzo is it running smooth in realsense viewer? the camera is sometimes recognize as usb2 instead of usb3. Also the patch is set for d430 module, stream’s index, available resolution/fps might change with other models


#15

@david_sh

its running super smooth within the viewer and vvvv.


#16

@david_sh it runs smooth in the RealSense viewer. @schlonzo, it runs smooth in vvvv if you remove the Perfmeter2 node?


#17

Open the VL patch, there is an async Loop with a framedelay (5000ms). Lower that to increase framerate.


#18

@robotanton
@david_sh

could you give me some pointers on how to stream out the position data PosW as a dynamic buffer, so that I can directly feed it onto a pointcloud emitter? Or what would be the most efficient way of emitting these within a dx11.particles system?


#19

hey, maybe not a big help, but I’m streaming out PosW data like this with an Orbbec Astra:

image

The key here was to copy the data in the async loop, so they are still available when the buffer is uploaded in vvvv.


#21

I know it’s an old topic but did you guys managed to get it working properly? I’m trying to work with a D435 and it only works for a few seconds than freezes. I’m guessing i’ts some kind of memory leak but can’t find it so far.


#22

tested it for ~48h on an udoo. no crashes