Android Facial Tracking for Mocap and Vtubers

Join this channel to get access to perks:
/ @fofamit
I also stream on Twitch: / fofamit
Check me out on Patreon!
/ fofamit
Check out the creator of the application!
/ suvidriel
/ suvidriel
suvidriel.com/ (Get the application here!)
Her video on how to set it up! • Android Precision Face...
Other videos:
Waidayo Video:
• Waidayo: Free iPhone, ...
$1500 iPhone mount:
• So Comfy: Why I spent ...
Blender tips to create iPhone Face tracking:
• How to get Face tracki...
DIY your own VTuber application like Codemiko, Ai Angel, & Projekt Melody
• DIY VTuber application...
Converting Avatars to VRM for Vtubing~
• Converting Avatars to ...
How to set up this application on your stream:
• Make Vtuber Content fo...
VSeeFace Revisit:
• Best FREE Vtuber App -...
Face Tracking iPhone App:
• Waidayo: Free iPhone, ...
-Fofamit's Recommended VR Computer for Streaming-
A computer parts list recommendation for something that will be quiet and cool, but also be able to run stream and vr at the same time. I picked parts based on what I currently use or would want to use in the future... also picked parts that would look nice together.~
Amazon Links Below
Motherboard - Gigabyte B550:
amzn.to/3afGnxO
CPU - AMD Ryzen 5600X:
amzn.to/38s9duY
GPU - RTX 2070 Super:
amzn.to/33BWiVT
RAM - 16-32GB DDR4 3200:
amzn.to/31sWELI
Power Supply - Corsair 850W 80+ Gold:
amzn.to/31s69uE
SSD - Crucial M500 1TB:
amzn.to/3aaHkXS
Hard Drive - Seagate 4TB:
amzn.to/3gKSkOk
CPU Cooler - Noctua D15:
amzn.to/3fFV0eP
Computer Case - Fractal Define R7:
amzn.to/3kszJbV
For Real-Time Face/Body Capture:
• Real Time Face & Body ...
#Android #vtuber #mocap

[@Fofamit]

Be sure to check the description for the links to get the information on how to get the information on the application and the Suvi's socials!

[@sonickitch]

Hi Fofa, great video. As you mentioned this is the first version of the software so it has room to improve , and of course i wasnt expecting you to go nuts with the latest flag ship phone , but would you think some of the slow lip synching is because you were using an S8+ ? aka would the extra power of a newer say S21 or pixel phone change the performance of is that just down to the program getting better ?

[@Fofamit]

@@sonickitch If you already have a android phone it's worth just trying it if you have a better phone than what I have~. The tutorial on setting it up is on Suvi's channel~

[@protowave]

hey, it looks like the author's website has an unmatching security certificate - AKA the SSL cert on the website doesn't match the domain name of the actual website. this is a common spoofing tactic and it makes the website look pretty untrustworthy. just thought i'd let you know in case the author is either A. unaware, or B. intentionally being sketchy

[@Bleats_Sinodai]

"for whatever reason, people will just have this hate for Apple!" An iPhone costs half what I make in a year where I live, and has one of the worst customer support for repairs in the world. Meanwhile, I can get an Android device with MORE features than any one iPhone model for less than 1/4 that amount, and in my experience, it's always repairable by the manufacturer or a decent repair shop. I'm not getting a phone just for face tracking, but if I am to get a phone with that feature, I'm DEFINITELY not getting an Apple device, when I can spend less for a more capable phone that'll have a longer life because it can actually be fixed in the future.

[@dogdog1913]

feel like it's kind of weird to immediately follow up on people asking about android facial tracking with "for whatever reason people just have this hatred for iphone" like...... that is not the reason the majority of people are asking for android facial tracking...........

[@Fartingbucket]

@samara sizi ... Have you not looked at samsung prices?

[@leannechong2481]

maybe someone just has android ya never know

[@filypa101]

​@@Fartingbucket true, at least for the purpose of vtubbing (and maybe, probably, still being able to download games that would burn an android phone and play smoothly), you could get a reconditioned iphone X for a really cheap price and according to Worten, almost brand new in terms of cpu and processor, if not better than in the original launch since they actually update the drives (from what I understood). I think the only negatives were "semi-used" and "not as smooth" or slower than in launch but really... It's the stigma against apple, lets be honest (even if samsing is starting to become like apple with the A54 not coming with a charger or something like WTF SAMSUNG)

[@PrimaYuurinTM]

@@Fartingbucket since then samsung is the monopolizer of android? There are way more android phones that are cheaper than iphones and are as good as it is

[@ryuugaeisen8946]

@@filypa101 in my country, to get to hold of apple you either need to have a middle man who's in abroad, and have it delivered (of course the middle man increases the price). But android is almost everywhere. It's not stigma, it's the availability. We don't get the "line-up" in malls when Apple releases something new. Edit: I know u might not be doing it on purpose, but it's like ur airing ur privilege in front of us. And as if that's not enough, they make us poor people out here like warmongering-hate type of people against Apple.

[@luciferwhispers3173]

"For whatever reason, people have this hate for apple" is like saying "For whatever reason, people have this hate for EA" Thank you for the very informative video though! I'm STOKED to learn - through you - that Android has added Facial tracking also!

[@thefourthdymensionmusic]

exactly what i thought lmao.

[@Skycube100]

Could you explain? Sorry I'm not very aware of this whole android apple thing😅

[@luciferwhispers3173]

@@Skycube100 Apple is a very greedy and unethical company as a whole. You can look up videos of how they scam their consumers with "unrepairable" Macs, whose costs to repair means you should just buy a new Mac instead. Meanwhile, indie repair guys shows HOW to repair them for cheap, and they're reporting that Apple is making it harder and harder to repair their stuff! They are ALSO grossly over-pricing their stuff since... Since basically forever! You are NOT getting your money's worth. And it is old news that the factories where Apple employs people to make their hardware, there are "safety" nets attached outside of windows, catching people who want to throw themselves out, which speaks volumes about the bad working conditions. Apple has never ever been an ethical company. They're the EA of the tech industry.

[@Gray-Today]

The reason is in plain sight.

[@neferiusnexus]

@@Skycube100 Apple used their monopoly on hardware to pull off some pretty sleezy anti-repair moves, including nerfing older models through updates once the new version was released. And them banning suppliers from selling parts, or outright id-locking parts so they can't be swapped-out not only compounds the e-waste problem, but also makes it really hard for people to recover precious data when the devices break down. And they are DESIGNED to break down. Look up Louis Rossmann on youtube for a more in-depth explanation (he owns an Apple device repair shop but also does free tutorials on device repair)

[@FeralBlade]

There are actually a few android devices with "Depth Sensors" Some with 3D TOF, others with simple lazer. Always see this misconception where all android devices are treated the same. There are a few dozen android device manufacturers and no consistency. There is a huge saturation of bad devices and people just assume its "android" as a whole. Obviously Apple has control over all there products and operating systems.

[@Fofamit]

In the video I reference another video where I talk about this specifically.

[@FeralBlade]

@@Fofamit ahh alrighty. Sorry for missing that. Goes to show i cant multitask as well as i assume.

[@ScarberryVT]

This is amazing!! I agree there's room for improvement but Suvi is so talented and I'm excited to see where Meowface goes! Thanks so much for doing a comparison Fofa

[@potatlebacon7941]

In the past I did try alot of easy things from you, especially this one which is very accessible to me, yet here I am, no longer doing these stuff but love learning and enjoying hearing it from you, keep up the good work and thanks Fofa~Fofa

[@ProtoPropski]

Who knows perhaps it's a passion waiting to be explored in, one, or more ways? I know I use to love creating videos, and exploring different ways to do so, even with low end hardware, and software (at times just using a PS3, to make vids/thumbnails, ya IK, and it was HARD) I took the drive I learned from those days, and what I gathered since, and now I'm a Live Streamer makng engaging streams taking advantage of EVERYTHING my pea brain equipment, and funds can posible afford. It isn't easy, but it's definitely fun, and my audience makes it all worth it. So maybe, you too can do something with this inner passion you have, I'm certain you have what it takes, all you need to do is find a set of goals you can agree on, and walk towards them.

[@lampdevil]

Thanks bunches for the comparison footage! It's got room for improvement but I'm excited to update my model and give it a go!

[@JorgeEscobarMX]

Hey it's better than nothing.

[@chimeramanticore]

i noticed in the comparison that the android version seems to be fairly capable of producing the same expressions, just less exaggerated than the iphone version. do you think this could be worked around when setting the 52 blenshapes by making them more exaggerated than you would for iPhone tracking?

[@Nol_lName_]

"Android phones doesn't have depth sensor" 💀 All the new Android have depth sensor 🎉

[@Pez-San]

This is exactly comparison I was looking for. Thank you for great video

[@SethWuffer]

Good ol' Suvi! They do really helpful tutorials as well

[@alicemontero7028]

Hate for Apple?... girl im broke

[@pvpperdogs]

I'd be interested in seeing the android tracking vs a webcam. I've been using webcam, have been considering getting a busted iPhone just to use the facial tracking but even that is so expensive

[@echolovely]

WOOOT!!! IVE BEEN WAITING FOR THIS

[@barongaming4915]

so in the video you say you are using a galaxy s8 why are you using an old galaxy? would it not be better on say a galaxy s21?

[@PunkBune]

I have been waiting for this!!!

[@kiton_vtuber]

Wow this is such great content I'm in love with this channel already hahahha

[@ByakkoJ]

Sweet, I had no idea. I so need this thanks for the info.

[@paolaanimator]

Thank you so much for this video! I followed her socials and got her app as well. I have an Android phone for a while, and while I do wish I can get an iPhone, I can't at the moment thanks to college. College has been busy so I'll test out the app later and see if it works out for me (I have a higher end Android phone). Your video is helpful

[@Dude_Slick]

I don't think it's a hatred for Apple, as much as a price issue.

[@thefourthdymensionmusic]

they have a slight history of dishonest business practices

[@MintLilaq]

I don't hate Apple products, I'm just broke. 😭 But yooo I'm so excited to use this!

[@teenflamingo]

Same tho

[@sydneybeaa]

I have an older iPhone, perfect blend of no ARKit and no alternatives haha. It’s ok. I’ve been using my phone as a webcam to my pc and tracking there. VUP works surprisingly ok, but like you said, it’s a lot of computing power. Still wishing I could afford a new iPhone to get my blendshapes buttery smooth

[@jessicamapes2717]

I have just bought an Iphone X like not even two weeks ago just for vtubing T-T but hey this just means more accessibility for SO many people now to get started!

[@LyndsayLionHead]

THANK YOU

[@FlameSoulis]

In case anyone is curious, I got OpenSeeFace running on a Raspberry Pi 4. Even with a slight boost, it struggled to get 3 FPS for just detection. Just in case someone else thought about it, I'm saving you the few hours of struggling to get it to work.

[@NokoChan]

Woah! Nice vid, thanks (:

[@GabrielNunezcix444]

I LOVE UR MODEL

[@lilyai4062]

Oh my god i love your model

[@strangeworld33]

KZbin i need more videos like this

[@angiethequeen2871]

This is it, this is the secret sauce. Thank you so much!

[@SaltedMallows]

Why doesn't the iPad ever come up in this discussion what's the difference do they not have the same camera on an iPad otherwise why drop 1.5 grand on an iPad tbh

[@delimiter2886]

Fofa back on doing god's work

[@AA-zm6hp]

Wait, what exactly are you using? What is your set up like? Your video is absolutely nailing expressions and its the best Vtube video I saw in my life.

[@colecozy4398]

I really feel like you should be bigger than you are because you make very great video to be used as tutorials for how to be a vtuber.

[@rawbeef1321]

it mainly depends on what android u using because I'm running the program on my Samsung j7 , s22 ultra and galaxy a04 - galaxy a03

[@godned74]

The hate for apple is real.

[@AIM4USA]

Haters…

[@Linkmindz]

Thoughts on the vive face tracker?

[@AnothaplanetZizzer]

I am trying to figure out if I can use this app as a workaround for iclone's facial expression addon that requires an iPhone. I only have an android so I'm ok with it not being prefect as long as I can make this work. Is this possible?

[@daisyfairy42]

The app is awesome so far! It actually works even better than you demo'd here if you sit down and configure the blendshape multipliers to your needs (which suvi recommends in their tutorial). While I have issues picking up certain things, I'm not sure if that's my subpar lighting or glasses or weird shaped face though xD I'm using a pixel 5a if anyone's curious, so I have a pretty powerful camera ^^ it IS a hell of a battery hog rn though!

[@enderremieenderman]

Umm may I ask what is the app name ?

[@daisyfairy42]

@@enderremieenderman It's...the one fofa talked about in the video??? And linked in the description??? It's called meowfac but like, watch the video friend, that's in it. I don't mean to sound rude, I'm just baffled because your question is exactly what the video is about and demoing???

[@enderremieenderman]

@@daisyfairy42 oh okay I just haved no time to watch it now I got ! Dont worry your not rude:)

[@artbyalii]

Please I need help !! I have exported the 3D Rigged Character from Relisuon Character Creator 4, Completely facial Rigged. And that rigging working perfectly in every software even in UNITY. But when I export that character from UNITY in VRM FORMAT for use in V-FACE or ANIMAZE. But my Facial Expression Rigg is not working in V-FACE with tracking. I followed many tutorials but still not get any solution for that. Even i hvae turned my Character to Humonied but still !!! Please Please help me with this. That why my Facial Rigging is not working in V-FACE? And the rigging is working in other software perfectly

[@neferiusnexus]

does it support depth sensing tho? There are excellent Android phones like the G8 series from LG which also have stereoscopic 3D selfie cameras, which can also be used to open the phone just like with Apple 10 (X) series and later models.

[@flipman8565]

Hi like your video can tel us how you did manage to to make your ouwn charachter and animate it live like in the video

[@pastelbunzy3152]

Ok nevermind the video the chairs lagging-

[@GloriaTheAnimator]

how is it posible to setup meow face into unreal engine?

[@luisahelena4226]

hello, can we connect those face tracking app with Blender? I will use a vroid in blender

[@einscha3754]

whats the appss ?? I tried to search i didnt seee itt

[@Morganaplays]

How to get the Android version of this app

[@SunShine-nt5ky]

Can someone please tell me how she dose this effect on her avatar and make it more realistic looking ?!

[@Sfhjgds]

Kay I founded a bit of info She said she used unreal engine. Idk which one and how but with unreal engine ╮(╯_╰)╭

[@samizo5842]

What if I have an android phone that have depth tracking camera? Is there an app for android ?

[@Tremori_A]

I wonder if a face tracker can be made with Google's soli chip

[@ene_ai]

I mean, really the issue and upside of android are the same. There is just a lot more variety and freedom of choice for your individual needs, whereas an iphone is just an iphone, so if you don't like it you're just out of luck. But that also means applications like this can't be tested nearly as easily because there is little to no standardization between android phones. Like, some android phones have all the necessary tech to have something just as good as the iphone with enough development time. Many android phones do not.

[@CoisasdaLuisaHelena]

Is it possible to have Iphone apps on Android?

[@currentlyvibing9916]

Hey would an iPad work? For tracking I mean

[@karlpj1]

Could be this used for Unreal Live link?

[@Monarch_04]

I kinda need a video of how yo set this up XD idk what I'm doing

[@indigoiozzi]

Does Meowface support connecting to pc through USB? Because my Wi-fi doesn't reach my stream setup...

[@danieldragonlord]

omg ur model......... its crazy diffrent from when i started watching u i love it juist wow!!!!!

[@Striker_Kunwar]

i want to create character like yours so please make tutirial video

[@635574]

It makes a huge difference when a sensor is standard vs when its a rare optional thing. There are still android racing games these days that dont have tilt steering. I cant find a modern phone that doesnt have gyro but it used to be a problem. Now its the devs fault.

[@blenderknowledge8652]

Can it work for blender

[@dissonanceparadiddle]

LET'S FECKIN' GO!!!!!!

[@Death.Malakaiah]

Not everyone hates apple, most of the people can't afford an iPhone 14

[@jauxro]

idk if anyone wants to answer really Beginner Questions but it can't hurt to write 'em -- How can I reduce jitters in the model? And if I wear mocap face dots + add animations to the model, can I create expressions that squash and stretch?

[@Fofamit]

Not sure what you mean by model jitter, but I don't believe adding dots will help.

[@jauxro]

@@Fofamit Thank you! The model jitter seems to be part of the typical range of motion for a vtuber, but I watched someone get full-body trackers (Tundra trackers) and it really steadied their limbs!

[@XunluT]

I wonder if this Android trekking is better than webcam + Nvidia Broadcast addon for Vtube Studio.

[@smilesgrin9382]

whenever i use my android i somehow come up with the same face tracking the iphone she is showing does.

[@InkblotRaven]

Would have been nice if you put the name on screen a couple times. Especially for people like me with adhd who zone in and out... Felt like I heard the words "android facial tracking" multiple times but never text. Putting words on the screen is a lot more engaging. I kept checking the description as well but my tiny adhd reptilian brain didn't know who suvidriel was even after hearing the name.

[@happy-iw2yn]

How you created this video where you are looking as a 3d model (real) one instead of some sketch

[@Fofamit]

wha?

[@Dedbeat777]

What is the app

[@asnitube9907]

can you name the android app I don't have iPhone

[@friendlysealstudio4447]

why not use meowface to have perfect iphone blend shapes? from android camera?

[@shemziller1]

We're the app link

[@thecomputerhead64]

what's the application for android ?

[@draft_fusion]

Super confused with all the links, which one is the android?

[@CrocoDylianVT]

If I knew this I would've saved myself those $30 I spent on a webcam lol and spent that in the face tracking addon

[@AndrewSouzaGames555]

Awesome

[@zippyholland3001]

Thanks 👍

[@GlitchedVixen]

I'm really curious, how did you get such an amazing room and blur affect? What applications did you use 😊

[@kal8965]

I wouldn't bother asking he doesn't respond

[@GlitchedVixen]

@@kal8965 Aw, its a shame. Their setup is so pretty

[@ellalatheellie]

Yiii android finallyyyy

[@Semruk_The2headed]

where is the ANDROID link

[@babacartoure8120]

What is the name of application for android ?

[@arminarlert5626]

What the Android one called

[@vtnwesley]

How beefy an Android device would this require?

[@Fofamit]

Suvi tested it on a S21 and I tested it in this video on a S8+. Honestly I would say it's worth a shot if you already have an android phone~

[@PRODUCT5]

What's the name of apps for Android ?

[@FamilyFirstFighter]

I don't hate apple, I love my oneplus 8t+ with 12gb of ram that has never failed a thing I needed it to do. Including a built in 2mp depth sensor.

[@maymeisama]

I have iPhone 11(my old phone) and samsung s22(my new phone), which phone best for Mocap and Vtubers?

[@Fofamit]

You can see the direct comparison in the video

[@MyDigitalHub]

How about for higher model of Android such as Samsung?

[@Fofamit]

Won't make a difference.

[@Vampykinz]

i tried to contact you on twitter about being in the que for full face tracking for my vtuber

[@hagozillazockt7448]

Cool thx! Liebe geht raus! 💪❤

[@OFFICIAL_ERROR999]

Is it safe?

[@TaeruAlethea]

** Cries in Intel Realsense not getting turned into a vtuber solution **

[@Aelethil]

OH MY FU-... FINALLY!!!

[@ZeroNineFilms]

Would a note 22 work better cause s8+ is old an I don't wanna keep stealing my girl I phone for face tracking lmao

[@lesleyhaan116]

If you wanna do face tracking the iPhone is the only option or you need to look at some android that also have 3D scanning cameras but they are hard to find

[@Fofamit]

Yep!

[@rask8121]

I don't get it. Why are the iPhone and Android samples so awfully bad when compared to your avatar in the rest of the video? Is it animated in Blender manually (or with motion tracking) and rendered, or something?

[@MultiLegocrazy]

Seems like iPhones tracking is more exaggerated and that the visemes/blendshapes need to be altered for Android to look more appealing

[@IR_Mediaa]

Nice Video

[@Fofamit]

Thanks

[@zervo]

Well just because it's an android phone doesn't mean it doesn't have depth sensing cameras. That is not related to the operating system, it's up to the phone manufacturer. Android is overall more flexible, more powerful and more developer friendly than iOS. The limitations are not set by android itself, but the hardware. Some modern android phones are most likely a lot more capable of facial tracking than iphones are. iOS is a lot more limited than android is, just saying. With that random fact out of the way, great video! I was using a weird port of some crappy software for facial tracking before discovering this video, and to be honest that port was absolute garbage. This software is working great, especially if you have good lighting and a plain background, like a greenscreen.

[@Fofamit]

kzbin.info/www/bejne/a4unqaWJbdiBnq8

[@rakhiyadav3104]

Iam thinking of buying an I phone for face tracking 😅😅

[@Gray-Today]

Visible light is absorbed by dark colors or shadows, IR is not. That gives IR a certain advantage. It does not make IR the only game in town. It is Marketing that does that. Personally, I've never seen an iPhone face wink one eye. I've tried it with two programs.

[@kris220b]

well i already got a samsung a7 and i cant afford an iphone soooo