VL.Devices.RealSense

[robotanton]

Hello,

we’ve started to work on Intel RealSense Depth Cameras support for VL.

The nodeset will give you:

• Color Image
• Colorized Depth Image
• Point Cloud (incl. filters provided by SDK)
• Depth Pipet
• Full control over RGB and Depth sensors
• Full info about the Device, Streams, Sensors and Intrinsics

If you have any ideas, comments or wishes, please let us know.

The development is sponsored by wirmachenbunt.
How cool is that! Thank you very much.

Github repo with instructions how to use the package (in alpha or gamma):

GitHub - vvvv/VL.Devices.RealSense: A package for using RealSense depth...

Updates:

12.07.19:

• Camera runs async
• Added Motion (IMU) data for D435i cams
• Added GetIntrinsics node
• If you’ve missed it: the package is available via nuget.org (check the Github page for instructions)

Best,
Anton

Related topics:

Realsense depth stream z16

Hello Thanks to the new (super cool) .NET libraries vl’s feature I try to use the libreRealsense library. The lib seems to works great with vl. I’ve access to streams, few options and functionality. My issue is I can’t pass the depth stream to vvvv in his orginal/similar format “Z16” . For now I have to use the “colorized” depth texture instead, which ends up to vvvv in “R8G8B8A8_UNorm”. I guess this colorized depth texture is designed for visualisation and not to transmit the maximum depth …

NUItrack port (urgent!)

looking at the meshed depth cam feed I can see why - it comes with 10cm noise across the board this cam is probably great for registering presence, and quantifying it on a per-person scale, but it is far from the fidelity of the late kinect seems to me intel did a good job to invent a cool cam for obstacle avoidance, but it will not be able to “see” a dance, in case the obstacle is actually human

[guest]

Hooray! So looking forward to this. How to test? And will this include the IMU data for D435i?

[robotanton]

Hello guest,

IMU data should be easy to get as soon as we have stable pipeline.
But we don’t have D435i to test its IMU, so I would ask you to test it when we’ll reach the point.

Best,
Anton

[robotanton]

Some progress:

[robotanton]

And… now for the brave:

vvvv/VL.Devices.RealSense

Please test and report.
We are WIP.
(IMU data, multiple devices, multithreading, etc… are still not touched)

[u7angel]

@robotanton , thing is we need the realsense in vvvv beta and don’t care much about how it is implemented in VL. it would be nice to have an easy way to test the thing instead of this 4 steps instruction inlcuding having to compile things and making references somewhere (which doesnt work).

can you please provide a zip with a working project ?

[robotanton]

Hi @u7angel,

please check the Github repo again.
Now we have a ready-to-install nuget tested with the current alpha and gamma.

Hope it works for you.

Best,
Anton

[Aurel]

Hi, I have some difficulty to install. It is always saying the same :

[joreg]

@Aurel what exactly did you try to get this error?

[Aurel]

I open any girlpower VL patch and after having installed as explain in the Github :

nuget install VL.Devices.RealSense -prerelease!

So then I try to use Realsense device selecting the Vl Nugets. here come the error.

Aurel

[robotanton]

Hi @Aurel,

it looks like you’re using the beta.
Would you please try it with the current alpha instead:

alphas

Best,
Anton

[Aurel]

Thanks Anton,
I tried with Alpha 64 & 32 but no menu in VL to go in Commandeline.
When I go to User…\Documents\vvvv\beta-preview\Packages> nuget install VL.Devices.RealSense -prerelease

Nuget is not recognize…? is this normal
In User…\Documents\vvvv\
there is only beta-preview…

Don’t see how to do.
A

[ravazquez]

Hi @Aurel, CommandLine menu in alpha was moved to the quad menu:

Hope that helps.

[Aurel]

Ok it’s works, Thanks for your help.

[timpernagel]

Great work @robotanton ! Works out of the box and as aspected. At least the vvvv Gamma part. :)

The range for the D435 and D415 are specified at ~11m. Well… didn’t expected good results in this distance, but my cam (D435) seems so be limited already at a depth of 3,7m. Tried different settings and light-situation but nothing really changed. What did you exprienced @robotanton or @u7angel, what was your maximum z-depth with the RealSense?

[timpernagel]

Just found the threshold for the depth-clipping. So: Everything is awesome!

[timpernagel]

Any chance, to get a “raw” depth image coresponding to the Kinect-depth image with floats in the R-channel for the distance in mm?

Searched fo it and tried several node-combinations and channel-settings for the depth-stream inside the “Config”-settings of the RealSense-node. But no success. Also the “Depth (Raw)”-node seems not work as expected.

Good news though: We managed to add the PointCloud-output as pos-buffer. Which works out of the box with DX11.Particlepack. So, fast homebrewn gpu-pointcloud-visualizer: ✔
But would be nice to also approach this texture-based, as described above. Any ideas?

[Aurel]

@timpernagel Could you please provide an example or just explain the way to have a homebrewn gpu-pointcloud-visualizerGpu-Pos-Buffer in V4 with Particlepack ?

[timpernagel]

@Aurel will look into the realsense-project by the end of the week and will post the steps what to do here.

[SuperflysiNZ]

I just received my D435i, super tidy little camera! I have loaded up the demo patch, but not sure how to get it out into V4, and then how to pull that data into V4 to manipulate it, be that the pointcloud, or jhow to pull the video feed out into a dx11 renderer. Pretty exciting stuff!