ok, so i hope attached patch can help to get this sorted.
it allows you to view your mesh next to live kinect data and compare joint names, orientations and degrees.
i’d say the first goal should be to get all the arrows of your mesh’s skeleton point in the same directions as the kinect ones. that should help with further steps…
EDIT:fixed attachment
skeleton2.zip (10.6 kB)
quick update…
found extra an issue… inverting axis in maya results skin twists in dx11
seems assimp dx11 way more perky then dx9 collada. Need some extra test on dx9 here no clue if that prolitoriat_3 is working on dx9…
Anyways pain and suffering only way here…
ai, can you upload proletariat3?
already fixed some aligments however i don’t think raw rotatins will work here…
try this one…
gonna try to use plane T pose for riggin
pro_4.zip (278.4 kB)
ok, i’m afraid the first thing we’d still have to get right is that 180° rotation around Y of the whole model.
so when loading the model in vvvv it has to look towards negative Z. note: it doesn’t help to change the Coordinate configuration settings in ColladaFile (EX9.Geometry) because those somehow don’t affect the joints.
only when that rotation is fixed it makes sense to try and get the individual joints line up with the orientations coming from the kinect.
also for simplicity i’d concentrate on DX9/collada for a start, once we have that it should be easier to make a version for DX11.
mm in 11 version it actually properly oriented i’m afraid this the negative Z vvvv uses by default… witch actually can be fixed by multiplying root joint with scale 1,1,-1 however rotating the rig won’t be a quick job, since to make it that way i need to redoo whole skeleton thing otherwise it will be just rotated from root transform
here is quickly rotated version
pro_4_rot.zip (274.6 kB)
btw from my tests on dx11 single joint works pretty ok not on whole thing tho
and since i had some sucess on a 2d version, i’m pretty sure i know how to fix that thing…
here is also an 11 version
Kinect2_Char_dx11.7z (291.1 kB)
ouright, i think this is getting somewhere, see here: kinect2skinner-directx-renderer and attached. the guy is moving all his limbs according to my movements. i am still not sure how to exactly deal with the extremities (feet, handtips, thumbs) correctly but i think the rest is at least correct concerning the SkinningTransforms.
you see that i had to rotate the mesh by 180° around Y and you see some of the joints seem twisted, but i think it would be best to fix those in the mesh-skeleton. you can do this anto!
Kinect2Skinner.zip (253.1 kB)
Just a quick question … is there going to be similar node/contribution for Kinect 1 ?
I ask because the one we have ( AnimatedCharacter (AssetsDB) ) now is for OpenNI and at least i didn’t manage (don’t have the skills) to make it work properly with MS SDK. OR am I doing something wrong ?
Kinect1 one still quite usefull i think , as it can work on low end /old machines …
i haven’t looked at the orientation-data we get for kinect1 yet. from what i read it is different, so probably the method used here cannot directly be applied. anyway first step: we’d need a skinned mesh with bones according to the kinect1 bones. scroll down here to see the requirements: kinect#kinect v1
I have skinned mesh with animation already :)
the file that i attached few posts up complies with the Kinect1 naming in the document you linked to. Just checked it in Blender and the names are a match.
I am asking because i spend a week or so trying to make it work and i have no success whatsoever .
EDIT: Ok may be not, my bones seems very similar to the scheme but i am not sure yet. Searching fro reference from MS
@synth The contribution from me never realy worked with either openNi or the microsoft one. the problem with the joint rotations was never solved at that time. the workaround for everyone i knew having projects like this was to cut the mesh into pieces and transform the pieces one by one… but i hope you guys can do it the proper way now :)
@tekcor thanks for clarifying this :)
Ok so the Collada file i attached up might actually work.
problem that i met with the joint rotations is that they have to be rotated and rotation applied before skinning the mesh otherwise it all twists and becomes ugly. I`m not much of an rigger but simple skeleton like this should be easier ( my weights might be a bit ugly tho) . Did some tests with Blender last night and it might be doable even tho i am still unsure of Blenders Collada expeorter as it seems not to work properly ?
Also found Better Collada addon for Blender from GoDot game engine that might do the job.
sry for delay closest update prolly gonna be around monday…
ok well another 5 hours of nothing… this shit…
It’s just retarded i played around with joint orients in maya, not much helps.
I’m suspecting this due to the fact that with such orientation not every joint is translated by Y axis.
Second one since i’m pretty sure that mesh has negative Z, turning it around 180 gonna result negative X.
The only thing is left to do is to collapse amount of joint on ur patch so i can finally understand what remaping doing… And why create joint don’t behave same as setjoint with base transform…
a pitty to hear that it seems to be such a nightmare to get some joints oriented in a specific way. so i think another option could probably be to take the mesh as is and offer a way to manually rotate each joint within vvvv before applying the kinect transforms…
think i spent past week at list 30 hours on this crap, just need few days off
Any developments here? I’m also working on a solution and would appreciate any further suggestions
for my understanding what is needed is either/or:
So, I’ve been trying my best to get this stuff running and I think I am almost™ there. I’m also assuming that nobody was able to come up with a sattisfying sollution yet, even so the thread is kinda deserted!?
My goal here is:
A DX11 version of controlling a skinned collada character with Kinect v2. My current version is a mashup of Joreg’s Kinect2Skinner and “skinning dx11 instanced by microdee”. I also attached a blender file with the rig and collada model.
At the moment there seem to be three problems:
The orientations from the Kinect 2 Skeleton node seem to be incorrect, or at least I cannot comprehend why for example the z-axis of the head is inverse to the neck.
The orientations of my collada rig are still not correct. But I think this can be fixed by playing around with the rig in blender a bit more.
Lastly and most importantly the method of using the skeleton nodes to apply the skinning transforms seems to have some central conceptual fault.
For example: The ShoulderLeft bone of the collada rig is given the transformation matrix of the ShoulderLeft bone created with the skeleton nodes. But this does not contain the correct rotations for this bone to behave as expected. The ShoulderLeft bone would rather need the rotation of the vector pointing from ShoulderLeft to ElbowLeft. Or the rotation of ElbowLeft and the translation of ShoulderLeft.
As you can see in the example patch this is illustrated when offsetting the skinning transforms. (Which is no solution and breaks the rig at the outer most bones)
I guess this is what Joreg tried to account for with the remapping of the skinning transforms. But no matter what remapping combinations I tried, it never seemed to work out.
In the end my guess is that the way the vvvv skeleton rig works does not produce the correct skinning transforms for a collada rig.
Maybe the attached files will inspire some further ideas.
vvvv project, blender file, collada file (601.5 kB)