iPhone Facial Capture with Unreal Engine | Unreal Fest Online 2020

[@PauloSamurai]

*Cries in Android... *

[@jadrianmc3420]

*cries in no phone at all*

[@xelo_516]

@@jadrianmc3420 *cries in no face*

[@TheJonathanExp]

@@jadrianmc3420 * cries in potato chips *

[@motionmx]

not as good , but you may try my app Face Mocap links on the video kzbin.info/www/bejne/iIXRZKSrqMSJa6s

[@fabricioh94]

*cries in calculator*

[@ajaxnation9504]

"If your iPhone doesn't have a home button, you should be all good" If my teachers explained lessons this way I would've gotten far.

[@yasaipicles6295]

well, its apple.. so naturally it would be missing features.

[@ajaxnation9504]

​@@yasaipicles6295 Actually I'm pretty happy with the home button removal and the notch 😅

[@beeceelad]

@@yasaipicles6295 Just like Android is missing this app

[@yasaipicles6295]

@@beeceelad hei, we shitting on apple here, dont get distracted

[@diamondwolfpack6]

@@beeceelad we shitting on apple here

[@TheGladScientist]

Was not expecting Tentacle Sync integration! Whoa, amazing! So many amazing features of UE4 Virtual Production learned in just this one video. Thanks so much.

[@psy237]

just tried it on my own custom made character, it works like a charm .had some setup trouble but now after using the gateway instead of the computers IP address it does an amazing job. if you have your arkit blendshapes set up properly this looks life like and far better than in the presentation here... incredible!!

[@TheKwiji]

so do you have to set up like 52 morph targets as reference like you have to for professional facial capture systems? Or does it just work with any head? Is there a specific rig layout? This video only clears up a tiny portion of how to do this.

[@charlesalexanderable]

Yes you have to create morph targets. This same app already has an example project you can tear apart, but it is driving the character from kite demo. Look for FaceAR on the learn section of the engine launcher.

[@TheKwiji]

@@charlesalexanderable Thank you! I did not mean to be rude, it's a good video. I guess it can't all be covered in a short video like this one.

[@roblesliemedia]

@@charlesalexanderable Thanks for that. It's nice to know how to get it all up and running.

[@vivektyagi6848]

Please make a video for teaching Installing all the require products for this Facial Mocap with Live Face App. Many Thanks.

[@raai_n]

This is really cool. Keep up the great work

[@SuperJackman117]

Really cool tech, love Unreal Engine and Epic's new upcoming tech, definitely Game Dev friendlier than most applications.

[@julianas.3352]

Would be cool if it has the ability to export frames to other 3D software like Blender for instance.

[@yusufa5429]

You could technically export animations from ue4 to blender using another add on

[@aryandhume7251]

@@yusufa5429 which addon?

[@SlenLen]

@@aryandhume7251 You dont even need an add on You can record the animation in ue4 and export the animation

[@jayverene1986]

@@SlenLen Thank you :)

[@SlenLen]

@@jayverene1986 just be sure that things like the fps counter aren't enabled

[@teslavoltagames3208]

This is fantastic for indies. A cheap(er) solution for head and facial tracking in 3D without huge amounts of equipment.

[@Jsfilmz]

be cooler is a retargeting bp

[@andyp123456]

I have a feeling this would look great if you spoke more slowly and exaggerated your expressions for the demo parts of the video. It doesn't really capture your mouth movements when you are speaking normally. Also, do you just need a recent iPhone with the right camera setup, or do you need to be running the editor on Mac too for the ARKit plugin to work? (sorry if you already mentioned this in the video - I skimmed)

[@zerocoool1111]

Vozeek is correct. Before this app you had to have a mac to build their template project to your Iphone which I dont think a lot of people were doing so they came out with this app. You can use an Iphone and a windows together to do this.

[@KriGeta]

but how can we control the eyes? can we see and rotate our eye balls?

[@colmduggan8230]

I wonder if indie game Devs could use this to create a dialogue system similar to the fallout games or the outer worlds it's really nice

[@anasdev1553]

they would I would if I had an iphone

[@Forest5]

How do you combine live link face animation with fbx body animation?

[@zmg1995]

I'm wondering is it possible to get a facial animation fbx file - that could easily be placed and moved in timeline in sequencer?

[@JoeAndolina]

This is kinda awesome! Is there a way to use this technique to control a skeletal rig?

[@sumanthdavanam8229]

Hi now news that unreal engine will not support mac support what is the future of theis software now android support software will be safe please consider , or any depth sensor camera like intel real sence or prime sensor

[@ShhheilaASMR]

could you use a kinect or not really coz it doesnt do details? are there web cams that do this?

[@HDTRNK]

would be nice if they showed how to set the character up

[@Adama_Now]

I read this in the tune of Beach Boys

[@ХиппиПаНк]

super product! Can i record on a shooting information if clamp i phone near camera and zoom to portrait format, to use captured info for example to track 3d HAT later?

[@AnimatorHeadSpace]

That's cool man! D'you happen to know how to get her hair working for the control rig? When I load the character control rig into the scene, it comes without her hair. Is there a way to connect the hair back to the rig? Using UE 5.4 currently. Thanks.

[@Alex_River]

Any way you can edit the Animation in ue4 after its recorded? So you can like tweak the mouth and eye brows etc. if you're not happy with the performance

[@mnomadvfx]

Seems unlikely - though I would be extremely surprised given Epic's recent acquisitions like Quixel and Shave/Haircut if they do not have an appropriate acquisition in mind to bring free animation software into the toolkit. Given how much they have publicised UE4 and Houdini I think there is also a possibility that SideFX may be a future acquisition.

[@AlexiosLair]

@@mnomadvfx Quixel has nothing to do with animation

[@zerocoool1111]

You can tweak the animation after. All this data gets records as frame data. You can open up each clip in Unreal and edit the keys. It will be a bit of a pain but possible.

[@CrownKing2499]

Bring this for Android?

[@badrequest5596]

although it would be awesome, iphones camera can handle this, while androids still cant. possibly because it makes phones far more expensive and most phone developers opt out. putting one on and android is not impossible, just probably expensive

[@malathomas6141]

@@HuanLeVuong no there are alot of phones with time of flight camera for 3d face scan like s10, mate 20 pro, lg g8 and so on

[@malathomas6141]

@@HuanLeVuong ok but does the phones also need a strong neural processing unit?

[@RunTheTape]

ChickenWeed why? What studio has a specific need for Android? Android is just a toy os for toy phones anyway.

[@CrownKing2499]

@@RunTheTape Well maybe for indie developers, who don't have the money for heavy/priced gear?

[@captain6255]

Imagine hooking up this to a rig that has LEDs on a mask that can display emotions

[@Tallacus]

It would be nice if we can use OTHER depth cameras

[@Nashfanfl13]

can this work with a iphone 8 max??

[@yuutonn5979]

UE... u guys are amazing. big fan

[@Jsfilmz]

only if adding morph targets was so easy hahaha still love ue4

[@jvebarnes]

It would be great if this worked with either the asus or kinect depth cameras or the intel intellisense cameras.

[@gmdzbanwic]

it would be really better if it worked with kinect as its more advanced and can be waaay better with its proper depth camera

[@Arashocky]

*Looking for the app but realize I have android* 😢

[@abinari]

im so surprised that this technology has been available for years now but unreal/epic is one of the first developers to unleash it's creative potential in terms of creative tools for users

[@BlackEagle352]

It always has been.

[@SectorZeroGP]

Hey men, i have a question... If i use this on unreal... could i export the blendshapes or bones animation (whatever this plugin uses) so i could edit them in maya and smooth the raw tracking?

[@zerocoool1111]

What you want to get is MocapX which is a app for the iphone that directly connects to Maya. There is a free version but if you want real time feedback, its 300. For how face you get good quality facial mocap. its worth it.

[@SectorZeroGP]

mark fitzpatrick the thing is that you can actually export the data from this unreal engine plugin to maya so i think its viable (the only bad thing is that i cant test it right now but at least the theory is with me). Also, i know mocapx and i know is a great plugin but tbh, if i cam get the same results for free... 🤤

[@zerocoool1111]

@@SectorZeroGP I actually use both. It takes too much time to export your model to unreal each time you make an adjust to the blendshape in Maya. It's such a back and forth process. So I start with mocap X and then once its gtg there I export over to unreal and start making adjustments.

[@saadanees7989]

How to get switchboard? The program you showed at the end

[@Kanermi]

Can Live link work on a dedicated server, or only on a package?

[@MrTravisDCook]

will the latest ipads be able to use this?

[@swiftdetecting]

what is the best ipad and iphone which works with this now ? and do u also know how to do mocap so can do full body as well ?

[@thirasheela8334]

which type of mobile we want use for a facial motion capture pls comment mobiles name

[@williamheckman4597]

When is this coming to Android ?

[@jadebrimhall9173]

The head movement just seems a little bit too jerky. Look forward to when it becomes a little smoother.

[@laplacenudoll5500]

it just because the camera doeant track the shoulder, so lt looks weird

[@jaigrant-williams7736]

I'm sure they'll fix it with more updates

[@wickywak1]

Agree and also it can't track eye movement

[@miguelgm808]

and also she doesn't cry

[@Silpheedx]

So how do you get this working from a new project?

[@khaledabdellatif9497]

Is the connection betweeen I phone and pc via wlan or can it be via Bluetooth?

[@ahmedalsalameen5898]

This is amazing 🔥🔥

[@thechuckwagon6951]

With a professional capture rig and more realistic character model, you could do all kinds of interesting stuff with this going forward. Televised debates, for example...

[@川畑貴寛]

Could you tell me reason why don’t move head motion?

[@DontfallasleeZZZZ]

This is cool, but it would be even cooler if it was real-time in-game for players.

[@seboritter]

Im sure this will happen including vocal transform. Future is kinda scary...

[@ilv1]

All this new tech, yet mouth movements are still very bad. Can we work on that first? The 3D visual industry needs better facial animations as a whole.

[@Badabing69]

How could that then be implemented in a practical manner in-game though? The CPU players would then have to have AI smart enough to verbally and contextually react to you otherwise what would be the point? 🤷‍♂️ Edit: actually this could work in a multiplayer context quite well, don't mind me.

[@geniusn8343]

It can't be done in-game, at least for more years since this technology is still using your face to recognize the facial animations. In-game would also your require your face to be recorded properly in order to do it.

[@MiregalForges]

They're already doing it in Star Citizen. Search Star Citizen FOIP on KZbin to get an idea.

[@sumanthdavanam8229]

where to purchase this plugin for unreal engine , i want to purchase it

[@f12374]

why Andriod don't have depth camera?

[@malathomas6141]

alot of them do you need it for 3d facial scan. mate 20 pro, mate 30 pro and p40 pro and lg g8 all have 3d tof ir camera for 3d facial scan so this should work on them

[@orthopraxic]

Is it only compatible with iPhone X camera and newer version of iPhone? What about iPhone 7

[@edwood972]

Android version ? and when ?

[@Hunger53]

I don't think it will be available on android. It's because newer iPhones have Face ID feature, that capable to track face very accurate.

[@oskar7063]

Huawei mate 20 pro too

[@Hunger53]

@@oskar7063 it's not that popular and I don't think it's compatible with ARKit, since it was developed by Apple.

[@oskar7063]

@@Hunger53 I know

[@chloeguennou2855]

Hey! You can try using Win and Android, if you want to: kzbin.info/www/bejne/qZXNomqsnbB0p5I Only a proof of concept but could be useful as a start.

[@matangunland4525]

What version of iPhone is compatible? And at least use iOS? Thanks

[@Supergab85]

interesting! Is there a way to sync automatically a random talking animation (facial) to an audio file? like Mass Effect

[@ToysRUsKid_Critter]

Wondering out loud of there is animation data I can look at as curves in graph editor ? Some way to filter out noise and tweak…

[@ntygamer20]

does iPad air 4 has it?

[@AgainstThisWorld]

I know people aren't interested in development but this is surreal thank you very much for the work, even though people think this wont make a difference but it can make a difference for many person who has psychological issues and will help very much on their self development

[@nunosenica1862]

Is it possible to add face capture to an existing body only mocup animation? Thank you.

[@framebyframegames]

Can Unreal use the Kinect 2.0 for mocap?

[@Sestanovich]

I want to know this too!

[@63fan]

i'm pretty sure, just not sure if there's a piece of software made for that specific use, tho

[@Demnus]

And when did you said Android version will be available? Need one ASAP!

[@chloeguennou2855]

Hey! You can try using Win and Android, if you want to: kzbin.info/www/bejne/qZXNomqsnbB0p5I Only a proof of concept but could be useful as a start.

[@mateuniverse]

Show how to do a remap asset etc. workflow inside UE

[@ChenZijian0831]

Can this project be packaged and run

[@HunterDThompson]

The app isn't downloading on my iphone, just says waiting? I must be doing something wrong...any ideas? Thx

[@EvaFrise]

Is it still available? Or Apple blocked everything from epic?

[@joystickblaze]

Incredible, but what about on android..?

[@anasdev1553]

nope

[@OurStudioco]

add a default movement so when shes not mirroring instead of freezing.

[@designmaterial8829]

Isto é a cena! Bem top.

[@NickAB94]

Would this work sending/receiving mo cap data overseas? We need to work all over the world but I don't really fancy travelling with an iphone just to get 3 minutes of mocap data ha

[@twitte0king]

look easy enough, but how do i rig the face so it is compatible with this?

[@요이요이-z9t]

Please tell me how to apply it to the 3D character I'm creating, not this character

[@actl8832]

any way to use that in android??

[@RoboMagician]

the lip movement animation looks off

[@Crick2x9]

"off" is a generous way of saying it...

[@hiraclip6694]

What about for non IPhone userslive link?

[@ddupyjj6508]

例如我有一个角色动画，没有表情，要怎么结合这个软件来用?

[@Silot847]

Is any way i can take rhe recorded livelink sequence and put it on a new level sequence where i can animate the metahuman with cotrol rig?

[@phlegios]

Can you do this with a regular camera?

[@malathomas6141]

no camera need tof camera or 3d depth sensor

[@shivdoesmusic937]

Plz bring this feature for Android too☺️

[@hotbrain4368]

It Cool . But Imposible to capture Animation Movement from iphone?

[@shanshanva1682]

Which model iPhone is used

[@josephtomczak963]

what iphone do i need for this to work EXACTLY?

[@andreiandgames9581]

Work in iphone 7 ???????????????????

[@jayverene1986]

Just bought an XR to try this

[@darksilvergames5692]

Was it worth it? I'm going to buy it too

[@matthewlane]

Any chance of getting this on Android?

[@anasdev1553]

no get an iphone That is so painful to say but it looks like ios got something that is worth the money this time

[@matthewlane]

@@anasdev1553 Yeah, that looks to be the case.

[@anasdev1553]

@@matthewlane true

[@asamangareader4132]

I love the avatar What's her name?

[@drupatel5131]

You guys should've inverted the camera or the animation. It probably would've looked cooler

[@helenawilsena5821]

Is it available for Android?

[@chloeguennou2855]

Hey! You can try using Win and Android, if you want to: kzbin.info/www/bejne/qZXNomqsnbB0p5I Only a proof of concept but could be useful as a start.

[@elainehoey2914]

Is this software compatible with using windows based Unreal, or do you need to use a MAC

[@chloeguennou2855]

Now you can try on Win using Android, if you want to: kzbin.info/www/bejne/qZXNomqsnbB0p5I only a proof of concept but could be useful as a start.

[@lucianodaluz4637]

3 minutes and the guy didnt blink once. :)

[@mitsuracer87]

Does it work on adult phones as well?

[@lunaticiousofficial7210]

yes, its for iphone. Didnt you read the title?

[@geekume5539]

@@lunaticiousofficial7210 no he means adult phones not kid phones

[@TrilightLabs]

WOW 以后玩游戏结合摄像头捕捉真的会像 电影场景描述的那样了。

[@adayexpired6370]

Lack of tongue movement... maybe certain sounds could be mapped to tongue movements intuitively? 🤷‍♂️ Amazing tho. Exactly the app I’ve always wondered why hasn’t come out yet.

[@bf5874]

Why the sound is potato?

[@williamsli3745]

UE4.25.4　here, It give me error "Blueprint Runtime Error: "Accessed None trying to read property FaceGeo". Blueprint: KiteBoyHead_JointsAndBlends_AnimBP Function: Execute Ubergraph Kite Boy Head Joints and Blends Anim BP Graph: EventGraph Node: Set OriginalTransform" also tell me my device is not supported, I am using Iphone10X+, and I am sure ip and port are set collectly. Any idea?

[@williamsli3745]

Fixed, need manually set IphoneModel you are using...

[@georgefelner]

im having issues using UE with my iphone on Live Link. I used the correct IP.. UE sees it. but when i click play it says "facetracking not supported on this device" but Im using an iPhone X. Can anyone help?

[@artended]

Lets wait for a better version of bakametai app.

[@bondell1840]

paboss bisa untuk iphone xs max ga pabossss

[@heycutie4119]

now I can be anyone I want to be

[@piattrocks]

I came here to look for an Android or PC plugin that would do this. Now that the MetaHuman creator is in that limited trial with the samples out, has anyone found a workaround? Also, unrelated, I thought Epic and Apple were beefing?

[@chloeguennou2855]

Now you can try on Win using Android, if you want to: kzbin.info/www/bejne/qZXNomqsnbB0p5I only a proof of concept but could be useful as a start.

[@visualjonas6453]

i phone 7 is not compatible !! 😢

[@WarNoodle85]

thats pretty sick!

[@SeanMaxhell]

Nothing for android?

[@moue-x]

On What engine is this ?

[@anasdev1553]

unreal engine 4?

[@alexandrebachmatiuk3591]

Hi everybody, I have an IphoneXr, but it is impossible to connect my phone with the livelink, someone have an idea ? Plugins are good installed, UDP Messaging configured, Application with the good ip...