Dynamic 3D Avatar Creation from Hand-held Video Input (SIGGRAPH 2015)

SIGGRAPH 2015 Technical Paper
by Alexandru E. Ichim, Sofien Bouaziz, and Mark Pauly
webpage: lgg.epfl.ch/publications/2015/AvatarsSG/index.php
We present a complete pipeline for creating fully rigged, personalized 3D facial avatars from hand-held video. Our system faithfully recovers facial expression dynamics of the user by adapting a blendshape template to an image sequence of recorded expressions using an optimization that integrates feature tracking, optical flow, and shape from shading. Fine-scale details such as wrinkles are captured separately in normal maps and ambient occlusion maps. From this user- and expression-specific data, we learn a regressor for on-the-fly detail synthesis during animation to enhance the perceptual realism of the avatars. Our system demonstrates that the use of appropriate reconstruction priors yields compelling face rigs even with a minimalistic acquisition system and limited user assistance. This facilitates a range of new applications in computer animation and consumer-level online communication based on personalized avatars. We present realtime application demos to validate our method.

[@veggiet2009]

This is probably one of the most exciting Siggraph videos I've watch for this year. I'd say the quality of the output surpasses conventional motion capture of years ago.

[@veggiet2009]

***** Oh I most certainly agree. I think even modern full suit capture is "missing a soul." In motion capture that looks good I'd be very surprised to not find an artists hand tweaking the motions ever so slightly. I was just comparing this technique to full body suit capture from years ago. The fact that we can achieve with a single camera quality that was done with a full body suit 10 years ago is very encouraging. It still needs exceptional materials if you want it to look realistic, multiple layers of skin effects the light bouncing off of. But you'll still want an artists hand going back through and fine tuning the motions to make them as lifelike as possible

[@MemorableMedia]

The detail here is PHENOMENAL! Was the animation in the end really synthesized using the one camera angle? The quality was astounding and overcame the uncanny valley in a way that thousands of dollars worth of recording equipment could never dream of.

[@VirtualRealityReview]

This is super impressive.

[@MytreeTv]

#treetastic this will take #VR to the next level

[@davidcarpenter6385]

I want this.

[@KaltatheNobleMind]

outide the usual venues like telepresense gaming and the like i can see this tech take the place of the blurry face when people wish to remain anonymous. all the expressions can easily be mapped n a very non descript and generic head preserving the anonymoty of the person while lettign them be fully epressive.

[@veggiet2009]

+Meir Teichman That is a great idea!

[@Jakob7539511]

So THAT'S how zuckerberg did it

[@steves5476]

What the hell are you guys doing? Get off your asses and market this to the public. You could make a lot of money from something like this. Please don't let this be another one of those awesome research projects that just collects dust on a shelf in some university.

[@steves5476]

And your point is? :P It would still be a crying shame if this just ended up being hoarded away in some university's intellectual property vault.