Skinned Skeleton and Kinect

@synth The contribution from me never realy worked with either openNi or the microsoft one. the problem with the joint rotations was never solved at that time. the workaround for everyone i knew having projects like this was to cut the mesh into pieces and transform the pieces one by one… but i hope you guys can do it the proper way now :)

@tekcor thanks for clarifying this :)

Ok so the Collada file i attached up might actually work.
problem that i met with the joint rotations is that they have to be rotated and rotation applied before skinning the mesh otherwise it all twists and becomes ugly. I`m not much of an rigger but simple skeleton like this should be easier ( my weights might be a bit ugly tho) . Did some tests with Blender last night and it might be doable even tho i am still unsure of Blenders Collada expeorter as it seems not to work properly ?
Also found Better Collada addon for Blender from GoDot game engine that might do the job.

sry for delay closest update prolly gonna be around monday…

ok well another 5 hours of nothing… this shit…
It’s just retarded i played around with joint orients in maya, not much helps.
I’m suspecting this due to the fact that with such orientation not every joint is translated by Y axis.
Second one since i’m pretty sure that mesh has negative Z, turning it around 180 gonna result negative X.
The only thing is left to do is to collapse amount of joint on ur patch so i can finally understand what remaping doing… And why create joint don’t behave same as setjoint with base transform…

a pitty to hear that it seems to be such a nightmare to get some joints oriented in a specific way. so i think another option could probably be to take the mesh as is and offer a way to manually rotate each joint within vvvv before applying the kinect transforms…

think i spent past week at list 30 hours on this crap, just need few days off

Any developments here? I’m also working on a solution and would appreciate any further suggestions

for my understanding what is needed is either/or:

  • a suitable mesh
  • a further transformation per slice (at the right transform stage to be identified) in the patch i uploaded above

So, I’ve been trying my best to get this stuff running and I think I am almost™ there. I’m also assuming that nobody was able to come up with a sattisfying sollution yet, even so the thread is kinda deserted!?

My goal here is:
A DX11 version of controlling a skinned collada character with Kinect v2. My current version is a mashup of Joreg’s Kinect2Skinner and “skinning dx11 instanced by microdee”. I also attached a blender file with the rig and collada model.

At the moment there seem to be three problems:

  1. The orientations from the Kinect 2 Skeleton node seem to be incorrect, or at least I cannot comprehend why for example the z-axis of the head is inverse to the neck.

  2. The orientations of my collada rig are still not correct. But I think this can be fixed by playing around with the rig in blender a bit more.

  3. Lastly and most importantly the method of using the skeleton nodes to apply the skinning transforms seems to have some central conceptual fault.
    For example: The ShoulderLeft bone of the collada rig is given the transformation matrix of the ShoulderLeft bone created with the skeleton nodes. But this does not contain the correct rotations for this bone to behave as expected. The ShoulderLeft bone would rather need the rotation of the vector pointing from ShoulderLeft to ElbowLeft. Or the rotation of ElbowLeft and the translation of ShoulderLeft.
    As you can see in the example patch this is illustrated when offsetting the skinning transforms. (Which is no solution and breaks the rig at the outer most bones)
    I guess this is what Joreg tried to account for with the remapping of the skinning transforms. But no matter what remapping combinations I tried, it never seemed to work out.
    In the end my guess is that the way the vvvv skeleton rig works does not produce the correct skinning transforms for a collada rig.

Maybe the attached files will inspire some further ideas.

vvvv project, blender file, collada file (601.5 kB)