Kinect project - Questions


Im new to vvvv, but im working straight to realize one specific project, an interactive art installation. its an project for university, i am studying communicationdesign in hamburg, germany.
The idea is to create somekind of an interactive “painting” (or better say a projection), which changes its appearance by the behaviour of the visitors. the changes should be very subtle, so that the visitors dont think on the first sight, that they control the installation. we hope that it seems that the installation has its own life and that the people try to understand the reaction of the installation or have at least that strange feeling, the apprehension that the visuals are not randomly generated but a consequence of the visitors presence. we hope to achieve an aesthetic that is some kind of amorph, organic, maybe fluid-like. we get the data about the behaviour with the help of a kinect. maybe the word behaviour is too much, at least with my skills, it would be also ok to say: there is something direct in front of the kinect, make the color more vivid, or on the other hand, there is just something/someone in the back, make the color more bluish. the people should not wave infront of it or making jumping jacks. the idea is that the pure presence is sufficent to change the installation. its more about changing the atmosphere of the room and with that, the feelings of the people who enter the room. it could be possible, that, if the room is full of people, the installation starts radiating a bright red color, like a pulse. the room would change its whole appereance and the feeling of being cornered would be stronger.

so far so good, that would be the idea behind it. of course it is not an easy project, so thats why i need some help. what would be the best way to achieve a prototype of that idea? the visuals are at this point of realization not that important, making it looking good would be one of the last parts. i would be very happy, if at least a rudimentary version, which works only with colors or shapes, came into existence.

i already set up the kinect, i got access to the information of the depth (via a pointcloud shader), but now i stuck in the part where i convert the data into useful information, so that i can say vvvv, that, if something is very close, make this or that brigter, slower, higher or something.

in the end, there should be visuals which are based on the kinect information, not a representation of the kinectdata itself.

any ideas? is it good way to use the pointcloudtransformationdata for that purpose or is the a more easy way?

i would really appreciate any help! if there are some germans here, the vernissage is in the beginning of july in the gängeviertel.

i hope you understood me,
thank you very much

Hello zwickmueller,

I like the idea very much and the way you described focussing on the visual rather just kinect data.

The easy part is to to extract kinect data and the difficult yet fun part is manipulating those info into interesting abstract form of graphics.

Kinect help patches has almost all the information you want to play around, from that point make a plan how you want to affect your visual based on the received data. for instance users joint information against time itself will unlock many possible abstraction art. There is a lot to explore and all upto you and your wildest thoughts ;)

Maybe just start with the player skeleton data you recieve. You can take the info about head position for example and map that data to some coulour change. The kinect tells you also how many humans it can see, so that data can drive something else too. Pointcloud that in fact is already a representation of the kinect data seems to me bit overkill and not really useful regarding that you are aiming just for some subtle interaction.

hi, i like the project too, and as urbankind i like the way you described it.
there are many ways to use kinect data for this project and probably the best is to combine different data for different pourposes. here some hints, hope it helps.

  • track the users number and use it as a variable (more users or less)
  • track the position of users (the z is particularly interesting becase that tells you if a user is near and how much)
  • take the depthmap and analyze it with a pipet (so you can understand how much the whole scene is full of people and how near (a brighter value is a near pixel, and you can have a lot of informations to mean)
  • track the motion of the scene (for example with a trautner) so you can know if the scene is static or dinamic

and the best thing is to use all this and more combined, every variable has a little impact on the whole scene.

kinect works with 2 skels only. Track closet users and do a hittest.

thank you all very much, it helped me alot!
i already made a basic prototype which changes colors based on the depth and player information!

so, now the fun part will start. as i said before, i want to achieve an amorph and organic aesthetic. for example i think the work of andreas fischer is very inspiring ( -> i dont want to make it look like that, but i like the feeling about it. are there any nodes which could help me? i experimented some days ago with the waveshaper and combined with lfo and sine shape i got a nice organic movement, like particles in water. any other hints or ideas for that purpose? the thing is, i have something in my mind which i want to create, but there are 1350 other nodes which i dont know yet, maybe there is a node which fits perfectly and i just dont know that it exists. thats why im asking.

@antokhio: thanks for the information about the skels, but do you know how many users i can track with the kinect player node (microsoft)? and what is a hittest?

it’s two users whatever you are using
hit test just checking if you hit something

so, its not over yet, this ist a version i made a week ago:

pw: zzzz

the visuals and the colors are nice, i like them already. but there are some problems:

  • at 0:38 you can see that, if no one i tracked, the visuals kinda break down. i like the effect, but after that its buggy. is it possible to bypass the time until someone is tracked? i tried it with OR and SWITCH nodes, but i encountered a bug (at least i think so): when i connect an input pin to the OR node i could not connect the output pin. if there is no input pin, i can connect the output pin, but then, no input pin. thats strange. did i do something wrong? did i miss something? or is it a bug?

  • this prototype works way to fast, the flow is too precise, i want to slow everything down. i used Damper (Animation) at the most important part, but still its too fast. are there other nodes which can slow down an input?

thank you all very much!

Can you be a bit specific what value you get when no one is tracked? does that goes nul? to slow things down it’s best to use map node to set what you want and also you may check mainloop.

it’s just nil when you don’t have points, use avoid nil to bypass that or S+H and count combination, to connect something on feedback loop use framedelay since you can’t have the value resulting in this frame as and input for same frame evaluation. but it’s better to avoid framedelays since you can get an unexpected result from feedback loops if your spread count changes so better to check that straight away or use it with something like OR (Spectral)

Thank you all very much for your help!
the final outcome can be seen the next 3 days in the Gängeviertel in Hamburg, Germany:

i will post footage as soon as possible.

how did you create the fluid look? looks really good ;)

i used the liquish fx from here sourceshaderpack-(ex9.texture-source)