Forum

Visualising a Kinect Point cloud in a Stride Shader

Kinect Point Cloud.zip (341.0 KB)
As I’m working with a depth texture, I need to create the positions in a shader, I have a view cone visualised of my kinect, I expected to be able to use the same transform to warp the points to the view cone in the shader, but no matter how I’ve wrangled it, they keep to a rectanlge. I’m sure I’ve done this in v4, but can’t find the patch :D. What am I missunderstanding?
( I know I could use the same technique as the skia3d point cloud, but it seems less efficient to turn into bytes, and I’m trying to use really low power processors on the kinect end)

image

In DX11 using the KinectView perspective, gives me a warped cone, so why doesnt it work in stride?

psInputTextured VS_Textured(vsInputTextured input)
{
psInputTextured output;
float4 pos = float4(input.posObject.xy , inputTexture.SampleLevel(linearSampler, input.uv.xy,0).r *Amount,1 );
pos = mul(pos,KinectView);
output.posScreen = mul(pos,mul(tW,tVP));
output.uv = mul(input.uv, tTex);
return output;
}
image

override stage void VSMain() 
{
     // trying to get world apce of depth pixels

    streams.displace =  float3((streams.Position.xy) , Texture0.SampleLevel(LinearSampler, streams.TexCoord,0).r *Amount );  

    streams.displace = mul(float4(streams.displace,1), KinectView); //Why not view cone distortion?

    streams.TexCoord = streams.TexCoord;
}

With the viewprojection happening in the GS in stride

PointCloud.zip (340.7 KB)
Kind of works, but the depth is now non linear and the pointsprites no longer lookat the camera, as I have to worldviewprojection after the look at to get it work at all.
Any help gratefully recieved. I’m guessing it is stride doing weird overides. Why does World not work in the VS? What is it that I dont understand?
image

there were no hidden overrides or so, it was mainly a question of doing things in the right order. I’ve added some comments in the shader but probably messed up the Kinect transform a bit, you might have to re-adjust it:
PointCloud2.zip (338.9 KB)

also, the sprite technique you were using could be improved. instead of creating the sprites in world space and then inverting the view to rotate them towards the camera, you can create the sprites in view space. like that, they will automatically face the camera. and in the PS you can save a sqrt op by comparing the squared length to a squared distance.

1 Like

That looks great, but why was it when I tried using the World transform in the VS it simply didn’t work! Everything got moved to the GS becuase that was the only way I could get anything. Just about the only thing I didnt try was mul(kinect, World) which I guess must be the missing link! Thanks so much!
Ah I was doing some * transforms outside as well, they were probably the bugger!

not sure, didnt try it…

I wouldn’t say it was wrong, I just find the * for matrix counter-intuitive, the transform node is often easier to understand, it displays better which matrix transforms which.

anyway, glad it works now.