Forum

Smile detection

vvvv beta question
#1

hey guy and girls,
i am very interested in experimenting with “vision based emotion recognition”.

  • Maybe some of you know Christian Möllers project “Cheese”.
  • Also many cameras nowadays have something like a smile detection (e.g. sony)

Personally i want to head towards a “mood-font” - a dynamic typeface influenced by certain variables; like the amount of smiling / happiness would be great to capture and process.

i wonder if there already exist any libraries for smile detection within vvvv or processing.

best regards,
daniel

#2

hi daniel,

as far as I know there is no such library available for vvvv. if you wanna realize one you should have a look how to use opencv inside a freeframe plugin.

gonna go and google for “mood detection with opencv” - it’s for sure not a beginner task!

good luck, and keep us up to date.

#3

hhhm, just thinking:

retrieving the face itself using FreeFrameDetectObject and afterwards using Pipet (EX9.Texture) to analize the known region of mouth using ~9 coordinates.

1 2 3
4 5 6
7 8 9

then check for the indices of the darkest colors.

1 5 3
could be a smile

4 5 6
neutral

7 5 9
“surely not a vvvv-user”

somehow naif, he?
you can expect problems with beards.
and it won’t be precise enough for use in medical surgery…

just some thoughts

#4

thanks for the responses!

i am very aware that this is neather a beginner task nor that it would be easy…

i know about opencv and freeframe but as far as i know it does not go beyond face-detection. fear that anything else would be custom but still hope someone already did something familiar.

there is a smile-detection by fraunhofer institute called “Shore”
http://www.iis.fraunhofer.de/bf/bv/kognitiv/biom/dd.jsp

anyone experience with that one? there is a free demo version…

cheers

#5

hi guys,

I’m working for another project of mine on a smile detection - it’s basically a c++ project … maybe I find some time in the future to implement a freeframe version for vvvv, but I can’t promise that (free time is spare atm).

I also tried the spore library, seems to be a nice one - but the lib is not open :(

#6

If you gonna use OpenCV facedetector be aware, that this kind of software is a racist one. Im not kidding, it just works for kauskasian humans as its looking for eyebrows and mouth to detect a face. It needs contrast, which isnt high enough at asian faces (eyes region) and black people (eyes and mouth). Ive read this at OpenCv mailing list - a indian guy had troubles with that piece.

May be you get access of mouth region in face detection. And analyse shape up or down. But it will be hard to get subtile emotions beyond smile or antipathy. You’ll need eyes emotion as well. Just have a look at smiley’s, how they express emotions. Without eyes it wont work. No idea how eyes-emotion-tracking is working…

#7

I’m using AAM for the task of emotion recognition, head rotation and stuff like that … to be more correct, I’m trying to use it - still doesn’t work atm ;)

#8

well of course “face-detection” and most of all things called “vision based emotion recognition” are just models and can not be absolute correct, especially in nuances . (thank goodness!)

to make things like this work you need very complex algorithms. most of these tools have been developed in the western world and i think therefore more tailored to “the white man” - of course this is very limited…

The engine i am using is pretty sophisticated and uses something like a huge pool of face-templates for comparison. other might just use eyes like you mentioned…

++

lets face it guys - we should be happy that these things don’t yet work superprecise…
could do a lot of damage in the hand of the wrong people… :(

#9

yes, let’s face it! :D
(sorry, couldn’t resist)

#10

:D
lets all be happy - and there will be no need for smile detection…

#11

@morgentau: are you using a commercial application for that task, or a opensource based one?

#12

hey m9dfukc,
the application i am using is not opensource…
i have done a lot of research but could not find anything doing “smile detection” or similar available for free… (guess it will still take some time)

depending on the nature of your project you might consider contacting a commercial provider for cooperation? also there a are quite a few universities with departements working in the field of computer vision…

#13

yeah, I know that there is no ready to go software on the market atm. But it’s possible to bake something in opencv - I already did some work in the right direction (using aam algorithms) but now I’m running out of time and considering to use some commercial software (maybe shore). What software do u use atm?

#14

@m9dfukc: i am using shore engine. works nice…

#15

ah, did you manage to use jpgs insteaf of pgm’s ? was to lazy to implement a jpg to pgn conversation, last time I used shore.

#16

no. i did a conversion via java…
did you ever look at raw data of the pgm file? think they use it for its specific structure…

but maybe you can change via c++?

#17

funny, doing a project for the fraunhofer IIS (who made shore) right now, but different department…

© vvvv group