This AI Learned How To Generate Human Appearance

Pick up cool perks on our Patreon page:
› / twominutepapers
The paper "A Variational U-Net for Conditional Appearance and Shape Generation" is available here:
compvis.github...
Crypto and PayPal links are available below. Thank you very much for your generous support!
› PayPal: www.paypal.me/...
› Bitcoin: 13hhmJnLEzwXgmgJN7RB6bWVdT7WkrFAHh
› Ethereum: 0x002BB163DfE89B7aD0712846F1a1E53ba6136b5A
› LTC: LM8AUh5bGcNgzq6HaV1jeaJrFvmKxxgiXg
Earlier NVIDIA episode: • NVIDIA's AI Dreams Up ...
We would like to thank our generous Patreon supporters who make Two Minute Papers possible:
313V, Andrew Melnychuk, Angelos Evripiotis, Anthony Vdovitchenko, Brian Gilman, Christian Ahlin, Christoph Jadanowski, Dennis Abts, Emmanuel, Eric Haddad, Eric Martel, Evan Breznyik, Geronimo Moralez, John De Witt, Kjartan Olason, Lorin Atzberger, Marten Rauschenberg, Michael Albrecht, Michael Jensen, Milan Lajtoš, Morten Punnerud Engelstad, Nader Shakerin, Owen Skarpness, Raul Araújo da Silva, Rob Rowe, Robin Graham, Ryan Monsurate, Shawn Azman, Steef, Steve Messina, Sunil Kim, Thomas Krcmar, Torsten Reil, Zach Boldyga.
/ twominutepapers
Thumbnail background image credit: pixabay.com/en...
Splash screen/thumbnail design: Felícia Fehér - felicia.hu
Károly Zsolnai-Fehér's links:
Facebook: / twominutepapers
Twitter: / karoly_zsolnai
Web: cg.tuwien.ac.a...

[@TwoMinutePapers]

Pick up cool perks on our Patreon page: www.patreon.com/TwoMinutePapers

[@paultremblay4836]

Is there a AI that could transform a 2d picture of me in to a 3D animation mod that I can use on blender or Daz 3d?

[@MobyMotion]

Wow, this takes the dangers of deep fakes to a whole new level. One picture of someone and you can make it look like they’re doing anything

[@dArKoMeGa89]

We're welcome to the fakeable era

[@Olodus]

I read a proposition the other day that said that maybe cameras would crypto-sign images and video they produce. I don't know if that is the final solution, but when you look at things like this I really feel we need to atleast make more strives to solve the problems these cool applications also create. Can we really trust video evidence in court anymore? Probably, but I would like them to tell me what camera produced that video first.

[@1ycx]

@@Olodus Exactly.

[@sankhyohalder97]

@@Olodus I saw similar concerns in an earlier discussion, so I'll just throw in my 0.0002 BTC as to why this isn't as big a threat as we fear: Prior to the advent of photography, the best evidence for events you didn't see yourself is oral or written evidence, hearsay or drawings. Only recently in the last 200 years did we create the idea of objective and absolutely trustworthy means of dissemination of information/evidence. And this was a false notion from the start. Photobrushing is as old as photography itself. Know that famous picture of young Stalin? Well it's shopped to hide the terrible smallpox scars he actually had. In a way, we're back to a situation that the human race has known and tolerated for millenia, but in fact it's better than that. In those days, you could reasonably vet information by cross referencing various sources, and by informal use of reputation networks. If A trusts B, and you trust A, then you can assign some credence to what B says. Now, we've got cryptography techniques to help with that, I'd imagine companies will strive to manufacture and market hardware and cameras as tamper proof as they can make them. And there will be a constant stream from the IOT of things that would have dozens of independent observations of any given event. You could then do some calculations on the probability of the evidence being true by simple Bayesian means, after assigning suitable weights. If some device is easily hacked, discount it for the purposes mentioned and try to get other sources. Just like asking multiple people what they saw or heard! There's going to be teething pains, but nothing here is fundamentally impossible, and it'll become an outright necessity should be want to keep modern day levels of trust in our recordings, or at least modern day before this video came out ;)

[@sentinel9651]

@@Olodus Doesn't matter if you have evidence or not. The courts are so rigged that they already discard or destroy evidence they don't want used.

[@MichaelSHartman]

Photographic evidence will never be used again. Take a photo (possibly colorized) from 120 years ago and have the people, landscape, or urban area become "alive". What biographical videos might be made, or old scenes revisited. Imagine seeing Anastasia and her family "alive" again, or common people shopping, talking, and walking, in 1900 New York.

[@jasonnicolini3934]

That sounds so awesome! I would love to walk around reanimated 1900 London with my VR, or perhaps something even better, entering a reanimated simulation.

[@mesharial6676]

man every time I see a new paper I just get overwhelmed by the number of applications, our life will never be the same after AI matures.

[@florisr9]

To me it looks like the network just generates generic human poses with some of the colors of the input applied to it. The head and face structure between all examples is pretty much the same. This can be seen especially at 1:40.

[@florisr9]

They are all basic templates that are not really made on the fly, but pre-generated, and filled in with colors.

[@cryan9137]

@@florisr9 I noticed this myself. Seems like this method will need to generate new presets for different kinds of clothing types, and body types. They only used clothes that stick to the body and don't flair out. The more characteristics you try to control for, the more unfeasible or costly this method becomes, it seems, as the amount of effort to upscale multiplies. Still, will be useful for designers as a starting point.

[@1ycx]

@@cryan9137 Just the starting. May become near perfect after several rounds of optimization.

[@selftransforming5768]

This is amazing, imagination meets technology. Imagine the possibilities.

[@mathematicalninja2756]

Great. So we can pose estimate through walls using Hotspot and using just a simple photo of a person, we can accurately simulate what he / she is upto behind walls, all using a simple hotspot.

[@cooleobrad]

I imagine this could be used for VR videos. Allowing people and eventually objects to have 3 dimensions would finally allow users to walk around and lean into VR videos, at least to some degree or in some scenes. This could also be great for future VR avatars!

[@TheMullela]

This is the one and only channel to have a bell of mine. Keep up the good work!!!

[@komugi7136]

Get ready guys, the whole anime AI community is about to blow up.

[@creeperlamoureux]

Lol

[@Dave-rd6sp]

You predicted the future.

[@simoncarlile5190]

The fact that it can generate so many new angles and vantage points of an object _using only one image as a reference_ is sincerely awe-inspiring.

[@mykulpierce]

Imagine concept art being generated on a 3D models. this would be amazing for a lot of different formats especially games and film

[@tempname8263]

Wait 200 years and we'll have whole games with render engines completely replaced by neural nets, all looking like trippy liquid dreams. Honestly, this idea feels cool and terrifying at the same time.

[@nikitakhutornyy3086]

Temp Name more like 20 years

[@Jone952]

We need to start putting hashes of videos on the blockchain to prove their existence is from the Pre-fake era

[@baumkuchen6543]

Finally we can create our own dystopia

[@NoNTr1v1aL]

No, that was not me who jumped from building to building screaming "Just your friendly neighborhood Spiderman".

[@VeeraBun]

It's like the computer is dreaming up people. I wonder when its going to wake up!

[@K9JT]

Maybe we are the dream...

[@Olodus]

No, don't wake it! You see this is the solution to the Control Problem. You keep it sleeping and hope that its sleep-ramblings are useful for us. Right now all we are getting from it is fake celebrity porn though. Must be a wet dream...

[@JorgetePanete]

it's*

[@joewandy8012]

Am I the only one who's annoyed by the 'dream' and 'thought vector' terminologies lol. The computer doesn't do such thing.

[@LyubomirIko]

@Joe Wandy Of course we on the edge with that. Yes machines dont dream etc - but for your perspective a lot of thing dont even exist - yet they shape you everyday. Your/mine consciousness is very narrow strip of the Unimaginable Universe. we are living in total darkness even for our own processes - in the body, and mind as well. The science is talking more and more of the concepts such as "emergent Universe". Basically what we are experiencing might be similar to what might happening from inside of a process such a deep learning AI. Take the concepts such numbers and math. We dont know what those ideas are - we see their ends (axioms) - sinking beyond our realm - similar to how internal computer process is divided from the atoms and the hardware - the barrier seems impenetrable - yet they are one whole.

[@lorenzorossi2000]

This is on an entire new level, that's amazing!

[@nononono3421]

Great to optimize crowds in real-time rendering if you have the memory to store renders of the individual characters in a few different views, but it seems to do well with very few source images anyway so memory footprint could be minimal. If you want to skip bone processing needed to animate the stick-figure, you could store the stick-figure animations in flip-books.

[@dynamicgecko1213]

What is this sorcery! My God, it's amazing.

[@mykulpierce]

That would be amazing for video games. instead of developing whole texture packs just have the AI produce them once during development

[@kkkkjjjj4517]

Now you can have Elvis as one of the pedestrians in GTA

[@mickelodiansurname9578]

Great for online clothing stores where you can now see what the product will look like on you before you buy. In fact better than a shop since you can't see what you'd look like cycling or running or with friends etc. in a store mirror.

[@constantinosschinas4503]

you cannot feel size, fabric and fit. nothing can replace trying it on store. even 3D scan of yourself at the moment of purchase and realtime physics with interactive body suit. maybe a brain chip. just maybe.

[@kayrosis5523]

As always with these AI papers, 2 parts exciting, 3 parts terrifying...

[@krishnamohan2351]

I guess in 10 - 15 years, all I'll need is a computer to make a Hollywood movie.

[@philtrem]

3:08 Now I know how they made that golf game we played on Windows 3.1.

[@selfreference2]

If we want to revolutionize the field, all we need is an AI that takes neural network style outputs and makes them look photorealistic. Not just upscalinging, but understanding what the generally blurry neural network outputs are going to look like (they feel like they have all the same issues -- things bleed into each other, output is blurry, no hard edges). Once a human can no longer detect these images aren't from reality, it will change everything.

[@cryan9137]

You're right. I think what you're asking for is an AI with a stronger understanding of reality. Or, an AI with agency that can do research to understand what it needs to solve these small errors. Like a human being or designer would do when they don't understand something. That's the problem in itself; if we, with immensely superior models of realities, of how things look and work, need to perform research to create (paint) images, I don't see how AI will reach our level without doing so. When we don't understand something, we understand that and fill in our understanding. The AI is in the dark, unable to understand what it doesn't understand, unable to fill in the holes, as is.

[@selfreference2]

I don't think it's that complicated. These errors look like compression artifacts. Try using a neural style transfer without using a reference image. It will have these artifacts I listed, the same artifacts, every time for every image. It will be blurry, have bleed over and have trouble with specifically hard edges. All we really need is a good solution for guessing occlusion and a neural network trained to separate out parts of objects (like a person and their shirt) or identify when an object can't be easily separated into parts (like an elbow and a forearm). Currently, it handles the latter very well but the former very poorly. So you have shoe fabrics that bleed into each other like human skin, or human skin that bleeds into the landscape behind it.

[@jordanscarrott3749]

Károly, you should just start making videos about what these new algorithms can't do xD. Thanks for the amazing content!

[@JorgetePanete]

This is a mega combination of NNs and i love it

[@ingusmant]

HOW CAN IT FORGET THE SOCKS?!! Joking, it's actually pretty impressive.

[@helloansuman]

You are awesome. Hope you can teach how the algorithm works as well in details.

[@devincory9695]

As these algorithms get faster, I'm looking forward to true photorealistic video games. It would be an entirely different render pipeline from what we use now, but the results could be more real than we could ever imagine.

[@WildAnimalChannel]

A-M-A-Z-I-N-G. I would like to use neural networks to generate 3D rigged animal models for my game from photos.

[@AntiCookieMonster]

First photography obliterated need for realistic painting, now AI will take care of the rest of visual art. That being said imagine, one proficient technical artist with a supply of processing power will be able to create amounts of content in days that used to require a well coordinated army of artists to create in months.

[@royromano9792]

This will be a godly tool for game developers.

[@marcopederzoli4939]

all this new techniques will bring us home made high quality movies created out of nothing

[@not_a_human_being]

Great as usual! Are we nearing our limits?

[@kayrosis5523]

Nope, the ride has only just begun

[@liquidmasl2675]

i never knew you could wait for a video before it actually goes live :D

[@TwoMinutePapers]

New feature, we are testing this out too...

[@sevencolours5014]

It's not AI that learns, it's an algorithm that people programmed to do specific calculations.

[@mariospanker]

nEuRaL NEtWorKs aRe JUsT a LoT oF If sTaTEmEnTs

[@dankmemelord100]

This has unlimited meme potential

[@Terrafire123]

2:50 In 30 years from now, I hope to be able to click on any of the pictures on this list, and have a machine create the pair of shoes I have chosen. *Get working, shoe companies.*

[@pachinko-0707]

Thank you so much!

[@fergusmoffat8926]

We will probably have human level AGI within 18 months. To build one we simply need to reverse engineer the building block of the brain the cortical column ( cortical columns have 100 000 neurons each and 1 billion connections), there about 950 000 of thess building blocks in the brain and scientists think they all work in basiclly the same way . If we can reverse engineer a single one then we would basiclly understand how learning , reasoning , memory and intelligence works. By the end of the year the Allen institute will release the full connectome and neuron firing data from a mouses cortical column as part of the MICRONS project ( 60 000 of the 100 000 neurons have already been released, portal.brain-map.org/). Starting in 2019 programmers at IARPA and other teams at the Blue Brain Project and the Human Brain Project , Deep Mind and other places will simulate this cortical column and use machine learning to reverse engineer its activity, so within as little as a few months we could have understood its function fully , and then replicate it into an artificial cortical column. These artificial cortical columns would be significantly more sophisticated and powerful than the artificial neural nets we have today. Artificial cortical columns may allow us to solve the 5 major AGI challenges remaining in 1 go. Including catastrophic forgetting and one shot learning. As proof of concept that this reverse engineering method works, Programmers at MIT just used machine learning to reverse engineer the neural circuitry of a worm and built an artificial neural network from it that they then them to park a car. This ANN was actually totally different from the standard ANNs since it used time as a computational dynamic , perhaps the missing factor we need to make our artificial neural nets as efficient and dynamic as biological ones and as a unifying way to scale up complexity simply. www.i-programmer.info/news/105-artificial-intelligence/12243-nematode-worm-parks-a-car.html Then we could run these new artificial cortical cloumns on the prodigy tachyum 34 exaflop supercomputer that a Big AI company will be turning on in 2020, you could run an AGI brain at 600x the speed of a human meaning you could do 20 years of learning in 10 weeks. Elon Musk is also releasing his neuralink that will merge the human mind with a computer at the end of 2018, accelerating the speed of human evolution by billions of times. Human 2.0 by 2022 ? We will soon know if AGI is just around the corner or 20 years away , if its the former were looking at 10 000 years of human progress by the end of the decade.

[@iAnasazi]

How is this supposed to solve catastrophic forgetting?

[@fergusmoffat8926]

dynamic memory @@iAnasazi

[@iAnasazi]

@@fergusmoffat8926 Please explain.

[@kayrosis5523]

'22 is a bit early. Narrow AI will certainly be way more impressive, but an AGI is not going to be a simple matter of making enough neurons. Kurzweil predicts 29, but I suspect we'll have Turing grade AI a few years before that.

[@techniek3422]

oh my lord, the possibilities are endless!

[@ONDANOTA]

this is the second time it happens! youtube did not notify me of your videos (my bell is on)

[@TwoMinutePapers]

Ouch. :( Noted, thank you for the feedback!

[@jcair4017]

same

[@Aaron565]

This is perfect, now we just need to make it do video, and then use it to fool other AI designed for psycho-graphic analysis or interviews.

[@Kram1032]

Beautiful work. I'm really curious how far they'll take these techniques.

[@kayrosis5523]

A computer scientist I know says that he would be shocked if by 2023 we were not able to fully animate any photo ever taken, to the point of Harry Potter pictures. Do you want the 1896 Harvard rowing team to do a dance routine or have an orgy? Done. Make yourself a backup dancer for Micheal Jackson's Thriller music video? Done. Have a video of you making out with Beyonce? Done. Turn your baby pictures into movies? Done!

[@agnel47]

The future is gonna be amazing.

[@kamenwaticlients]

The beginning of AI imagination.

[@mintybin]

Wow this is superb 🤩

[@guilhermecaeirodemattos]

Does somebody know a survey, a blog post or anything that lists the main papers on face and/or scene generation?

[@sergeyltk]

You didn't believe that it forgot to put socks on? :)

[@storerestore]

Time to make a new Mortal Kombat game

[@skolex3121]

The dude in the bottom left at 1:08 lol

[@luck3949]

How is it different from Everybody Dance Now?

[@ironmonger4498]

​ @Two Minute Papers Your method is good only 5 minutes waiting. I have some creators subscribed who have DAYS of waiting. That feels bad coz u know the creator already has the video but is withholding it from us for the hype. I personally feel 1 hr for premier waiting and chat with creator is ok but not more than that

[@TwoMinutePapers]

This is a new feature, I am still learning how to use it properly. Glad you're enjoying this too! For now, this few minute thing feels great, and also I can talk to you Fellow Scholars a little in the meantime. Loved it!

[@richardede9594]

Amazing!

[@nextlevelstart]

HOLY SHIT THaTs COOL As HELL

[@lykenth08]

this is both amazing and scary at the same time - for design creation and blending of objects this if the future - its seems more creative than a 200 people - or create a film/series or game in a very short amount of time and at a fraction of the cost - the Matrix is real haha. on the flip side, in the wrong hands this can be used to place people - with video evidence, maybe a sound version of this coming soon? to create false evidence or depict a person doing something they otherwise would not.. hopefully there will be a way of identifying it and very strict control of its use.

[@kayrosis5523]

Yes, but by the time this happens it will be common enough that juries will be aware of this issue and video will undergo intense scrutiny, and there will be lots of cause for reasonable doubt

[@mateusmachadofotografia8554]

Carol johnah efahel?

[@aMulliganStew]

The Models Guild is gonna be pissed.

[@nutzeeer]

oh wow... i hope this gets used responsibly.

[@AntiCookieMonster]

It won't 😉

[@kayrosis5523]

It will, by some. But we'll find a way to ruin people's lives with this soon enough

[@jacekb4057]

crazy stuff! :)

[@francoisrd]

You sound a bit like you're exhausted/overworked in this video. If that's the case, remember to take break, relax, and care for yourself

[@jasonw137]

Did anyone notice how distorted all the generated images were? The face networks were kind of ok... but the face/body networks had awful distortions. You people complaining of photorealistic fakes have a few generations of improvement before you have much to fear. In the meant time, the best attitude is www.naturist.com/ imo.

[@fanofhifi]

1:30 ad on a 3:00 video? C'mon.

[@duytdl]

RIP trust

[@Graeme_Lastname]

How bloody wonderful !! Now, if you have enough money, you can make it look like anyone has done anything. Photographic evidence ?? Ya'll stuffed now. Not me, I'm too old. ;)

[@danishcoolkhan]

Nice

[@DheemanSaha]

What’s the name of the papers

[@newspeed8000]

don't worry it won't turn into skynet, skynet is just our primal repsonse of fear of the unknown!

[@omarcusmafait7202]

No way!

[@BitcoinJake09]

I see you accept crypto :p smart guy!

[@james.angel333]

Gurney journey brought me here

[@TheSpicyPotatoe]

Lol we are fucked. Nice knowing yall

[@redmotherfive]

I guess the question I have is, so what?

[@robinranabhat3125]

I want to help this guy, But I don't have money

[@Sefferdoodle]

Steins;gate...

[@lineage13]

Actors and actresses will be out of a job!

[@Cghost-fh4hf]

It seems like ai doesn't see any difference between face and background, for it this is all the same object, like a face is the part of a background. I can make a conclusion that ai doesn't think in terms of objects, or in terms of 3d space, or in terms of objects in 3d space, something like that )

[@kaikart123]

Really good to blackmail people. I am looking at you both NSA and CIA, you don't have to look at eachother.

[@p0rt3r]

This is getting creepy. Soon videos will be meaning less as proof.

[@ayra_c3781]

Wtf

[@williambarnes5023]

But can it do nudes?

[@antoniogabrielchaconmenke83]

Pffuuaaaa

[@neurowhai]

Fake CCTV video. Oh dear...

[@user-zu1ix3yq2w]

Maybe hoax