How to get Face tracking like Codemiko - Adding ARKit Face tracking for mocap & vtubing

I also stream on Twitch: / fofamit
This video is brought to you by Private Internet Access: www.privateinternetaccess.com...
- - - Software Mentioned in Video - - -
Unity:
unity.com/
Blender:
www.blender.org/download/
VROID iPhone plugin:
kuniyan.booth.pm/items/2604269
FaceCap App:
apps.apple.com/us/app/face-ca...
ifacialmocap:
apps.apple.com/us/app/ifacial...
Waidayo:
apps.apple.com/us/app/waidayo...
My video on it: • Waidayo: Free iPhone, ...
FaceIT:
blendermarket.com/products/fa...
Autorig pro:
blendermarket.com/products/au...
Other videos:
DIY your own VTuber application like Codemiko, Ai Angel, & Projekt Melody
• DIY VTuber application...
Converting Avatars to VRM for Vtubing~
• Converting Avatars to ...
How to set up this application on your stream:
• Make Vtuber Content fo...
12 hour VOD creation iPhone ARKit Tracking for GMA:
• iPhone Face Tracking C...
VSeeFace Revisit:
• Best FREE Vtuber App -...
Face Tracking iPhone App:
• Waidayo: Free iPhone, ...
--Fofamit's Recommended VR Computer for Streaming-
A computer parts list recommendation for something that will be quiet and cool, but also be able to run stream and vr at the same time. I picked parts based on what I currently use or would want to use in the future... also picked parts that would look nice together.~
Motherboard - Gigabyte B550:
amzn.to/3afGnxO
CPU - AMD Ryzen 5600X:
amzn.to/38s9duY
GPU - RTX 2070 Super:
amzn.to/33BWiVT
RAM - 16-32GB DDR4 3200:
amzn.to/31sWELI
Power Supply - Corsair 850W 80+ Gold:
amzn.to/31s69uE
SSD - Crucial M500 1TB:
amzn.to/3aaHkXS
Hard Drive - Seagate 4TB:
amzn.to/3gKSkOk
CPU Cooler - Noctua D15:
amzn.to/3fFV0eP
Computer Case - Fractal Define R7:
amzn.to/3kszJbV
For Real-Time Face/Body Capture:
• Real Time Face & Body ...
#howto #vtuber #mocap

[@Fofamit]

Special thanks to Great Moon Aroma!

[@MAMAvsGOD]

I asked this before but how do u add Avatars face at bottom to check for previews on Unity, Thank you

[@Fofamit]

@@MAMAvsGOD kzbin.info/www/bejne/aHSYoX2ibL5sma8

[@AICandii]

talented smart amazing fofa

[@bakedbuddhist]

12 hours of stream time converted to 10 minutes . Fofa's editing skillz are over 9000 .

[@abiez4018]

more like Patience Skill

[@kaito4885]

Jesus

[@CountJulius]

Ah yes This will be useful Edit: *cries in android*

[@minikuiyosei5098]

@Bird I literally just switched from my Sony phone to get the iPhone 12 for this. Still use google chrome and Spotify nothing really changed except now I can use real time face tracking models

[@minikuiyosei5098]

@Bird like I said the same. Unplug in the morning to go to work and plug it again at night like any other phone?

[@g.sedi_]

ah noo!! don't be sad, the android equivalent of arkit is arcore (they are both built into unity). So there probably is a way to do this with android, you'll just have to do a bit more searching

[@qumefox]

You can always just buy a used old iphone off ebay to use for this purpose. Needs to be a 6s or up to work with arkit. 6s's run like ~$50, which is honestly cheaper than a decent webcam.

[@CountJulius]

@@qumefox sadly only the new iPhones have the right camera I eventually got my hands on one tho

[@Estel_]

A nice surprise to see Moon's model included! Thank you for the video~

[@2beornot2bable]

Fofamit, you're the best! So informative, entertaining, and enjoyable.

[@Orchalon]

You've just made my life a whole lot meaningful. Thank you!!!

[@RascheTheRabbit]

This is incredibly impressive!

[@RexusKing]

I finally ordered my iPhone! While wait the phone to ship, I'm brushing up on your videos about Face ID. PS: I don't know why, but you laughing at yourself (like in the intro) always gets me laughing!

[@HikariYumii]

Thank you for this tutorial! I’ve wanting to know how to create new shapekeys for FOREVER! I’m new to using Blender, I’ve tried before and it gave me such a headache... xD

[@rokioto07]

Thanks so much for the tutorial, I’ll try the hard way.💙

[@KawaiiKlau]

Thank you! 🥰

[@alwaysyouramanda]

Fofa, you had me at, “What do I do with my life!”

[@mishaelalanahbaquir7395]

Thanks fofamit, you are heaven sent xD I bought an iPhone for this 😂

[@naysae]

Thank you so much for so many amazing tutorials. We don’t even know each other but u really helped me overcome my fear of streaming. Not only that your tutorials helped me start my own journey on twitch and for that I will be forever thankful❤️ I have a question- I have a vrm model as well (made from a vrchat unity package model) At 6:00 I can see that ur avatars on the screen has a ton of different face expression sliders. (I’m not really good with unity or blender) Is it possible to get face tracking to work with a vr chat -> vrm model? ( for some reason it feels like I have less face expression than ur model on the screen) Thank you so much for your help ❤️

[@CountJulius]

Took me 5 listens to have the luppet and magic mirror part register in my brain haha Thankfully that gave me time to set up everything else

[@Agonylord52]

Hey! I love this content, very helpful! I was wondering, though, is this a good wah to go about in-video face replacement? I'm trying to learn how to just swap out my face with a 3d model of a mask, which tracks my facial expressions into the mask itself. I want to leave the rest of my real life body in the video. Only swap out my face with the 3d mask model.

[@vtuberraf]

I wish I had an iPhone for just this purpose :/ uhh xD amazing video!

[@WisKy64VT]

You are really smart and funny

[@Tybbles]

Just wondering about your thoughts on the new HTC Vive mouth tracking and eye-tracking headset combo. would that be a viable face-tracking option for Vtubers or is the iPhone the only viable way. mainly asking cuz I don't have an iPhone and I'd rather buy a whole vive setup then get one.

[@TheeBeardedNoob]

Hello Fofamit! I'm super baby newish to the Vtuber scene and have just finised having my model drawn and rigged to be used with VTube Studio. Currently I'm using a Logitech Brio for my capturing but my Rigging Design Person suggested going with either a IOS or Android as they have the IR Depth thing you speak of. My question, which I feel is dumb but I tend to overthing everything, is Does the Iphone XR have to have service even though I'm just using the Camera feature and probably having it hooked straight up to my PC? And do you maybe have a good tutorial video that goes through setting this all up?

[@tommy_tomcat_vod]

When the cooler Facetracking might take longer than the creation of the whole avatar.😥 Thank you for this awesome tutorial, I definitely will get back into it because I just want this tracking to work. Problem is I will need to manually create a lot of them since my model is based on things from Models Creators created for MMD mainly. So it's just the basics and I need to add the rest like... 42 blendshape keys. 🤣

[@5olace]

link for the apple shape key names and guide? it isn't in the description or I cant find it

[@ntygamer20]

hello, how did you make your environment..please make a tutorial.. what app did you use?

[@peachuri]

Hihi! Not sure if you'd know the answer/cause of this-- but when I add blendshapes for brows up/down, they work fine in VSeeFace (using VTubeStudio on iPhone to track my face). However, when I add things like Cheek Puff, Tongue out, etc. they all work fine BUT then the eyebrows don't work anymore. Do you know what I could be doing wrong or how to fix it? Please help if you can!

[@shaper_i-o]

Do VRoid models have enough shape keys for this process?

[@COVERTI]

Заранее Спасибо :)

[@WoongNyang_O3O]

woW niceee

[@crybb-]

my blendshape proxy script does not have an avatar set to it :(

[@hazelnutneko]

I can't seem to change the face no matter what :/ idk what i'm doing wrong i can't get the left eye by itself to move

[@GloriaTheAnimator]

how do you get SO MANY shape keys in the shape key menu list in unity? i only get the 5 default dxpressions T-T

[@lucidparalysis2794]

Does anyone know if any of these apps work with brood models??

[@user-dy3sq8pg2d]

♥️

[@qumefox]

Does anyone happen to know what she's using for hand tracking? Is this also arkit?

[@ConnorM2007]

Can you do it without a computer

[@Paintingsvr]

I don't understand the language! would it be possible to have subtitles in Portuguese?

[@garypetras]

I just saw great moon aroma

[@COVERTI]

Здравствуйте. Можно гайд как после настройки аватара начать стримить и так далее, хочу сделать всё как у вас.

[@thestupidtwins5467]

UR MODEL IT LOOKS SO REAL STOP! btw I’m on my old account LOL

[@StilKoSan]

Hey Fofa, thanks for the tutorials! Are you doing commissions? How much can it possibly cost to create those blendshapes?

[@Fofamit]

The best way to keep up to date would be to check my Twitter page: twitter.com/fofamit

[@TaffieASMR]

Hello! Would really love some help if possible, I have exported my avatar with all the apple blendshapes set with no export issues, but when I put it into waidayo it doesn't seem like it's able to use any of them, I can't find any solutions. Any input would be greatly appreciated! Thank you

[@dukkducc]

late reply but ive had the same issue the only that is visibly working is sticking my tounge out... but its still kinda bad tracking i think im gonna try switch to ifacialmocap instead

[@Isimarie]

My main issue is that I don't know what half of these blend shapes mean. Like, what does "MouthRollUpper" mean? The guide is Japanese, so the text is basically unintelligible after Google Translate, like "Cover the upper teeth with the upper lip and roll it up. The shape when mogumogu is included in the mouth."

[@Syphonosaur]

what iphone is recommended

[@Ladycydonia]

Please please please more in depth I have no idea where to get the arkit shapes or anything 😭

[@schwipsy]

the avatar with the blue hair is clearly 3d but kinda looks 2d. HOW?

[@Sponty]

Is there a link to ARKit or the reference sheet?

[@MAMAvsGOD]

Fofamit? Where's the Codemiko tutorial for Unity part 2? I'm so sad

[@thisisaname3447]

yo have some patience

[@Scultore3D]

Hey there, I can create 52 blendshapes on a custom character, like the ones on the metahuman. Hit me up if you need any.

[@activemotionpictures]

8:19 - omg! the amount of subdivided vertices! Doesn't VRChat /android stream capture get slowed down by them? I imagine 3 of those avatars in a room would make the room laggy....What's the recommended amount of polys for an Avatar? 50k? 80k? 250k?

[@Loupning]

Oh, it's cause X-Ray is enabled so it's showing the vert from behind too.

[@meat5069]

Does anyone know a good channel for learning how to use blender? I am kinda brain dead currently on using character creation programs.

[@Fofamit]

Blender Guru

[@troubaderring-do9640]

Dikko is another great one, once you have the basics down!

[@estellastardust1159]

hey fofa! im trying vmagicmirror with ifacialmocap and its not working? all the blandshapes and keys are in the name correctly. and with waidayo it works fine. im not sure whats going on. any idea?

[@Fofamit]

Did you link vmagicmirror with ifacialmocap?

[@estellastardust1159]

@@Fofamit yes and it says "sending to computer" when i click connect so the connection is there i think...

[@sslicer7545]

What is the oldest iphone model that will give you the better streaming stuff?

[@MAMAvsGOD]

iPhone 10 and up

[@jadedk9916]

I notice that your "real-time face tracking" is not in real time but the audio is slightly off sync. What causes the delay?

[@qumefox]

This one I actually know the answer to. It's because processing what the camera is seeing your face do takes a little bit of time. This slight lag is very easily overcome though either in post with a slight audio offset, or if streaming/recording using OBS, with a very slight delay on the mic audio source, which OBS can do natively.

[@ValkyrieFolf]

What about us android users can you make a video of android I would be greatly appreciated amongst us android users and I'm just starting out vtubing but haven't posted a video yet but I'm looking for an app to were I can use a fbx model because I have one already made and (I love your work keep it up Fofamit you the best vtubing guide that's why I'm subscribed)

[@Fofamit]

I can do a video on that addressing the Android face tracking.

[@ValkyrieFolf]

@@Fofamit that would be helpful thank you and your the one who has inspired me to become a vtuber and your guides are so helpful that's why I'm subscribed

[@thatoneoni]

@@Fofamit that would be very helpful as I kinda hate apple with a burning passion and was wondering if arcore is a feasible replacement for arkit.

[@H3llok1tty_]

Is Arkit on iPhone?

[@lukek.5773]

are you doing something with your voice? not sure if you are just getting more comf on the mic or practicing subtle things like enunciation. but your vocal quality seams to be a bit more consistent so if you are doing something it show :D otherwise you seem to be getting more comf on the mic

[@justrosie.]

*screams in iphone 8*

[@noway5096]

Your mouth is TINY

[@xoxmylinh]

Hard to follow T^T