The Face & Eye Tracking Rabbit Hole [VRChat]

[@TheVirtualRealityShow]

Make sure to checkout SideQuest's Discord for Banter to get info on that Quest 3 giveaway :) discord.gg/bantaverse

[@petraexe]

discrod lol

[@ojbeez5260]

Hi Phia I contacted you a while back via Email about a potential research into VR etc. I have not heard back, maybe went to your spam folder. Let me know - you may find it very interesting.

[@Majicca]

Can i have some trackers so i can make a video on them

[@JayBayLaco]

Hey I just wanted to know since you know a lot about vr's and stuff, if i just got an htc vive headset with valve knuckles would i need anything else?

[@lkakh1694]

Is pico 4 pro ok for face tracking?

[@ThrillSeekerVR]

f a c e t r a c k i n g 👁️👄👁️

[@CentriVR]

Cidil

[@Hyprst]

trilseker

[@goteer10]

facetracking, but your eyes are covered by a magic brick that sends to you to cyberland Psst, check out project babble to do diy lip tracking. Goes great with eyetrackVR

[@summerbreeze2167]

BABBLE BABBLE BABBLE

[@forgottenfox3857]

@@summerbreeze2167Where my headset mr programmer socks?

[@RedCyberLizzie]

"If they don't do it, we will. That's the beauty of the open source community." Truer words have never been spoken.

[@prohurtz]

Thank you so much for featuring EyeTrackVR! The future of facial tracking deserves to be open, and I'm incredibly honoured to be a part of such a talented community working towards it!

[@RidgeXR_2]

Ello!

[@prohurtz]

@@RidgeXR_2hey ridge, lol

[@RidgeXR_2]

@@prohurtz *rigde

[@SuperSonic68]

Thrill + PHIA is the ultimate VR power couple. They both cover very different but very significant sides of the industry and work off of each other really well.

[@sleepysnowpoff]

Wishing you well with surgery, take as much time as need to recover ❤

[@EchoTheNeko]

EEE THANK YOU SO MUCH FOR INVITING ME TO BE A PART OF THIS!! Im really passionate about face tracking and I’m so happy a whole new audience is being exposed to it!

[@Midnight_7520]

imagine you were still mute

[@LilyanaVR]

yooo Echo :D

[@foxriver9156]

Are you currently taking any new commissions?

[@SuperFAILMC]

Are you commissions on gumroad or some type of free lance Platform?

[@Mayposquad]

Echoooo

[@Sam-gf1eb]

We really need more headsets to be released with face and eye tracking, or at least eye tracking. It's annoying that headsets being released still don't all have eye tracking. Sure, we can do it open source by ourselves, at least for eye tracking, but we shouldn't *need* to do it ourselves.

[@iAmBleuMajor]

Looking at you, Meta...

[@craigbeare9770]

Being able to look someone in the eye while having a conversation is why I always thought VR is the first true social media application. Everything else feels para or anti social in comparison. Great vid phia !

[@RyzawaCh]

I love the face track conversation having the virtual and actual perspectives, you've truly outdone yourself with this video!

[@nerdynanny]

Disabled VRChatter here! I'm blind in an eye and one day I'll get my hands on face tracking and folks will be able to TELL because of how it just-- kinda follows my good eye but doesn't move on its own. It's hilarious. ALSO Echo's face tracking is FANTASTIC. She's worked so hard on all the tracking and has spent YEARS in Blender [someone please bring my daughter back from the Blender wars]. I tell anyone who wants face tracking to hit her up because of how EXPRESSIVE it is.

[@T-MAX_X-H]

So does that mean you have similar eye movement to actor Tiny Lister aka Zeus in No Holds Barred/WWF aka Deebo in the Friday movies? He was also blind in one eye, so he had what you described in his eye movements as well.

[@saenxty]

Wait could I hit her up? If you don't mind giving out that info. I'd love to learn from her!

[@Drischa]

Thank you for making this video. the VRCFT community has inspired me so much and is so special to me. Huge thanks to Ben, Jerry, Azmidi and all the module and avatar developers who made it happen. Happy to see my friends Hash and Kombi featured here, they are awesome!

[@whoosh316]

HOW DID I JUST FIND OUT U AND THRILL ARE TOGETHER?!?!

[@Arukkao]

That's not Thrill?

[@parknich081]

@@Arukkao it is..

[@mrsweetkandy7673]

Yeah is that really him?? Like i recognize the voice but i find it so wild???

[@thelaughingmanofficial]

@@Arukkao Yeah it is.

[@TheBrokenEclipse]

Honestly, it is just mind boggling how much VR is being advanced these years, makes you wonder how far we will advance in just 1 or 2 years from now.

[@VirtuallyShay]

I love face tracking, hope to get it eventually. it's so neat to see how completely transforming interactions can be when seeing peoples expressions.

[@DaisyBunnyVR]

Seeing someone with eye tracking look at you in VRChat hits different, especially if they are flirting with you.

[@WindFireAllThatKindOfThing]

As long as my QPro doesn't make my eyebrows start doing jumping jacks for an hour straight just because I raised one slightly

[@PixelPusherGames]

Maybe you guys should do a VR chat podcast

[@ACMEJackStudio]

This is wild. That EyeTrackVR tech really opens doors to majority of us without native tracking headsets. I might explore this.

[@flameknight7]

hard to go without it once you got Facetracking! Just the subtle amount of movements you do in a conversation is crazy.

[@quipsilvervr]

I'd say you guy's relationship are goals for any VR community member! I like these videos that feature the both of you! EDIT - You just made countless thousands of people want face and eye tracking!

[@Dai_VR]

Epic video. I've been loving Face Tracking on the Quest Pro. For my purposes, I only wish the microphone was far better for Content Creation and Streaming on that headset, to be honest. It's also been fun learning about how it all works via OSC + VRCFT Software and more. Using it with my own Karin and Rusk avatars that I've spent 100s of hours on in Unity, to perfect to my likings, has been really neat too. Also I hope your surgery goes well and that you make a swift recovery. And once you're back to being able to make videos, I wish for you to be able to easily keep up the good work on the entertaining and informative content you produce. Take care. :)

[@vocaltune6504]

Most informative. I'm Trying to decide between quest pro and quest 3. I want to game but I also want to be very expressive. I'm still trying to find something somewhere in between or choose one or the other.

[@Retr0_]

The face tracking is SO impressive, and so was the way this video was edited. I was hooked 🙌

[@kingsqeerel]

Wishing you well in your health and please take the time you need for yourself! Love your videos.

[@bunnysabop]

i ordered a quest pro a few days ago and have been so excited looking for new videos on it and HERE YOU GO 8 HRS AGO!

[@bunnysabop]

wooooo!

[@HashEdits]

really glad to have been able to provide some footage. wasn't expecting my memes avis to be in there (I skibidied)

[@vtubernastia]

I have struggled with applying facetracking on my VTuber Avatar myself, but in the end i am really happy, that i have accomplished it :3

[@Pedro2figueiredo]

Fact that this channel didn't go viral yet, is for me simply mind-blowing

[@johan8676]

i cant afford to use any vr stuff since im poor. but i like watching your vids and keeping up with the tech evolving and what people are doing with it. i hope by the time i can afford to get into vr, things like eye tracking will already be standard and accessible!

[@kizashikaze9066]

Haven't been on your channel in three or four years...I'm so glad I found it again! Amazing how much improvement in technology has come in the past few years!

[@DuranMediaSolutions]

I love seeing you both in a video together! My wife and I met in second life almost 20 years ago and we love being together all the time in real and VR! ❤

[@ser-render-ser-vivor-ser-junn]

This is a pretty interesting aspect, I recently came across your channel recently and now I'm going to subscribe because I enjoy the editing and the ways your manage to get your content across. While I'm not too involved with this as all my characters are... knights/mass effect aliens/sci fi soldiers. This is good to stay in the know.

[@DuskstarShine]

My favourite thing to do when I music mirror dwell, is to gesture sync to the song, basically acting along and syncing my face gestures to the song, acting asif my avatar is singing it in a play or something. It's amazing and fun to see others that KNOW their gestures and actorly naturally use them when talking or acting; I love doing it so people can really see how I'm reacting, and feel better immersed when talking to me ^-^ My main avatars are not picked by how they look and toggles, but the most thing I look for a diverse gestures with a wide range of emotions. ones with diffrent gestures on each hand, or ones that have perfect mixed gestures. I always do different gestures per hand on my avatars, but am soon going to be adding a 3rd set for double gestures! :D I cant wait to use it ^-^ (and of course adding a toggle so people can turn off 2 of the gesture sets for when they just wanna chill or play games hehe.

[@kazeryu4834]

Eye tracking is a fascinating technology. When i get a new headset I will certainly try to get eye tracking with it. Also i hope your surgery went well

[@dawne2780]

Wow!! All the guest stars made this possibly the best video yet. I love the virtual reality show! Get feeling better, surgery is the worst 😭

[@honeybeechaos]

as a kid who got into vtubing first, then VRC, imagine the goldmine this was lol I was surprised to learn that hey, the rigging knowledge I might have in 2D rigs might be useful in 3D rigs! and the way that people make a living off it too!! I didn’t know it was like that in the 3D community- I thought it was limited to 2D models haha. I’m excited to see where all of this goes in a few years and maybe even get into it myself :0

[@we1rdo_man]

it's great to see the community grow like this, face/eye tracking allows so much

[@ReavoEnd]

I think you've had that surgery by now... Wishing you the best on your recovery! Hoping everything goes as smooth as possible. Health stuff getting in the way is definitely frustrating. Thanks for the video. Very informative and fun, as always!

[@HTCVive]

What a great video! We always learn so much from you PHIA! Thank you.

[@ReiTsukinoVT]

I'm pretty excited on where VR is heading with tech like this, though I share the very based concerns about corporate abuse (IE Excessive advertising and data collection). That said, I'm glad i'm recognizing some of the folks featured, makes me feel connected ^_^

[@shiftednrifted]

13:17 good to see vyde from New Eden making an appearance

[@itsmadi9725]

I'm planning on getting a meta pro soon and i cant wait! i have been playing vrc for 5+ years and i got fbt last year, i think I'm finally ready for the next step

[@HeyHebi]

I got mine along with a lot of other people for free at RDC, and while I never would've purchased it on my own its a fantastic device. My only issue with it is that because I'm on university wifi, it's on a separate connection so it HAS to be wired in order to access pcvr

[@skyfirefly76]

Ever since I started using face tracking in vrchat with my quest pro, I can't live without it. I have a quest 3 that sits on my shelf because face tracking is just that important to me. People are always blown away. And people comment that they can really tell that I'm listening to them when they talk to me. It's very important. It also keeps people from talking over me with the voice latency because they see my face expression forecast that I'm about to talk. It's great

[@Xario1]

As someone who has never had enough time to use vrchat often enough to memorize which hand gestures made which faces, face and eye tracking is a godsend. I can finally express myself in vrchat with literally no effort, as opposed to having to memorize hand gestures. Also, having started with vive wands never helped, since the circle trackpad was split into 5 sections for hand gestures. Because of the split like that, it's always been very un-intuitive and hard to memorize. It wasn't until 2 years ago when i upgraded to index controllers that i started to be able to more easily memorize hand gestures finally.

[@Bloopaz]

I really want a new headset to come out that has the same speakers + microphone quality as the index, But with the option to go wireless, better resolution, FOV and face + eye tracking. And of course it still needs the option for base station tracking as well, so I can use base-station tracked full body trackers.

[@BaileyGarman]

The vive focus vision is looking promising despite the fact everyone is upset with the fresnel lenses.

[@TheKAGORA]

I love the long form content! This was so fascinating and I am hellbent on trying face and eye tracking out now! Hope you have a good recovery from your surgery! ❤ To fellow commenters, where do I get a really solid option for a custom avatar?! I have been to so many avatar worlds for about two years of searching and I am struggling to find my fit. I think I need to have one made just for me.

[@threedkilla2579]

Hope a speedy recovery

[@CreemSmile]

Sidenote but I absolutely love the thumbnail! Really great composition and colours.

[@adicsbtw]

Another big opensource project in this field is "Project Babble" It's being worked on by a small but passionate development team, working on making it so that if you can get a camera onto your headset, you can use an opensource AI model made by the community to get face tracking up and running. Right now, it's somewhat picky, but they're working on getting more and more data, and using different techniques to train models. They've also been getting more and more people in to help render out new faces, and they recently announced that they'll even need to get help from the community just to train the next model, not even just making the images

[@liquidwater2]

As someone with the jankiest headset ever, the pimax crystal, setting up eye tracking and face tracking took me so deep down every reddit thread and discord server. After five third party programs and a lot of lost brain cells, I finally figured it out tho and its so cool :D

[@AllSquirrel]

Jerry's face tracking is amazing! Please check them out on gum road. Love my quest pro face and eye tracking. Only had it a week, and I do not think I could go back.

[@RibbityYT]

Take as long as you need to recover. Love your vids

[@hiimenglishyouknowwhoiam5798]

FacetrackVR is so cool! I made slimes this winter! Now I can build eye tracking too!

[@GoodMorningGabba]

Thanks for this great and nuanced video! I find the last part about larger corporations exploiting this information to be so interesting and important. ALSO hope the surgery went well and wishing you a speedy recovery Phia!!!

[@mesharkyvibes]

I’m literally on the edge with pulling the trigger on a Vive pro eye, I know I could get a quest pro easier but the vive does appeal to me more. The new focus vision is still an option too. This video may have pushed me over that edge LOL, GOOD JOB

[@redstefan6515]

amazing video, absolutely loving it, I just wish that foveated rendering would have been talked about more. I think it is an amazing way to enhance vr performance with literally no downgrade in visual fidelity

[@SeanBannister]

1:26 "...pulled my partner Thrill[seeker] into a..." so it's official? 😉

[@Thee_Titanis]

Exactly what I was thinking! Always had my suspicions but didn't want to jump to any conclusions or anything

[@metrotyler]

yes, they even mention the fact they live together and see each other every day

[@roldy6483]

3:18 exactly just got facetracking 4 days ago it changed everything never going back its just too good!

[@arkadyborys5545]

We're getting there! Can't wait for a future where face and eye tracking is available in most mainstream headsets

[@Lumiverse.]

YOUR EYES. So expressive!! 🤩

[@XietheZebra]

I have gotten so good at puppeting my models with hand gestures and locking them on and off, people have thought I had face tracking multiple times now.

[@SaturnSeraphim]

I'm really excited to see where this goes; while I really want facial expression tracking I wouldn't really be able to afford it currently - and hoping that headsets eventually all upgrade to track expression. I can only imagine that soon there will be programs run to block out tracking/streaming data to the companies themselves, but I am hopeful.

[@geoffreyprecht2410]

Face tracking is daunting, but once the technology is more available, I think it'll become commonplace. In his review of the AVP, Zuck said he's planning on bringing eye tracking back for future Meta headsets, and I expect that to influence the industry. I can't wait to get that extra layer of immersion myself!

[@mbridewell1010]

I have FBT and face tracking and alot of my friends do also. It absolutely gives a since of presence. I exclusively look for face tracked avatars now. Being able to smile and make eye contact is huge. I have a quest pro and with a few mods is an amazing and comfortable head set. Pitty they droped it.

[@proficientplumbus]

I personally bought a quest pro purely for the added built in Face and Eye (and now tongue yippee) tracking. It just adds an extra level of expression that feels way more natural to me than using controllers to determine how my avatar will look

[@Greggatr0n]

so this is cool 1 minute ago it got posted thats cool that i got here this quick :)

[@Dominik-K]

Thank you so much for the video. Im super interested in what VR headset tracking is recommended by you and others or if DIY is still the way to go. Hope your surgery went well, best wishes

[@Alexorsumidk]

Thrill, Boneless and PHIA are carrying the VR Industry's web side single-handedly🗣️🔥

[@RayRift]

I love the expressiveness Face and eye tracking give me with the Quest Pro. It's basically the only reason to visit Horizon. I had it set up in VRChat a few times but I need to relearn it and find or buy a good avatar.

[@JamieVatarga]

This is a great video. I had a Quest Pro for a bit for the eye tracking, but other factors had me go back to my Index, and Quest 3, but I really do miss the eye and face tracking. I have been interested in the DIY eye tracking, but I keep putting it off

[@BlackMew13]

This is certainly interesting and I'll happily see where this technology goes in the future! With that being said: as someone with ADHD I'm not sure this has much more value to me than just visual flare And now that I actually think about it, it's kinda scary to imagine that people could see my eye movements I already don't really read people's faces irl (those microexpressions your partner mentioned are something I've never picked up on before ???) And I know that neurodivergent people often get judged for not doing eye contact correctly The thing I like about digital interaction is, that I CAN abstract non verbal communication because I know I'm bad at doing it irl Having access to predetermined and labled expressions honestly seems easier than just hoping I happen to make the right face while talking to convey my emotion :'D Like if I'm angry and I can just trigger a mad expression on my avatar I KNOW that I'm gonna look angry and that others will be able to read the expression as angry If I'm just using my face naturally, I have no idea what expression I'm doing or how well others can read it But to be fair I'm also really new to vr chat, so I'm still learning how to communicate in that space in general xD This would be neat to play around with at least and having more direct controll over ones avatar always sound appealing!

[@n3e7]

For a while I had EyeTrackVR working on my valve index. To find out it was annoying. I use the portable mirror a lot for phantom touch/sense. And every time I peek at the mirror, the person(s) near me got confused where I am lookin at, even looking over their own shoulder to see if maybe someone else sneaked up next to them. But I was just checking the mirror for touch feelings, spacemove positioning (clipping in chairs) and if eyetrackvr wasn't glitching out and needed recalibration. In the end, I only found it usefull for blinking detection, and not for eyemovements itself. I just prefer to have phantom touch over eye movements, without other people getting confused.

[@Okmeis]

I wish you best of success with your surgery!

[@MildManNerd]

Glad facetracking is getting more awareness, PHIA your tracking looks so good. Can I ask your set up?

[@motthubris5122]

Hope your surgery went well, keep being awesome~

[@blackfoxOo]

finaly a new video. :)))

[@medris_OG]

I don't think a large company will invest any more money in face/eye tracking. meta only made one attempt with it. Last year I asked myself how I should improve my Vtubing. The meta quest pro was on the short list for a long time. But then I decided on an iPhone 14. Why is easily explained because I can continue to use my vriod model without any additional costs. The funny thing is, I'm still faced with the same question: where can it be better and more current? Is the question a quest 3 or a quest pro or wait for appel or a meta pro 2 So with many thanks for this video, it doesn't make the decision any easier, but it shows me alternatives, thanks for that❤

[@grantpodue39]

future president of vrchat, you do us proud

[@WunderWulfe]

I would love to see full face tracking natively added to newer and modern headsets, especially in the context of AR and MR, where if they really want XR to be able to substitute in things like office meetings and social experiences, the fidelity is absolutely required. this is especially important for things like eyebrow movement which not all eye tracking systems support, foveated rendering for high fidelity applications, and look-based control systems. I’ve been holding on before building any DIY systems simply because they are not capable of supporting all the features I would like, and some methods involve purchasing hardware which has since been discontinued.

[@Reveticate]

It's strange how many times I caught myself responding empathetically to you due to the micromovements you're eye-tracking was picking up.

[@ImaginaryX]

Loved this!! Great showcase of EYES!!! 👀 👀 👀 P.s. Hope the surgery went well! ❤️‍🩹

[@MajorNash-SC]

STARFARER! Great world, I wish more people would go to it, met a lot of my friends there.

[@Jestloo]

I think facial expressions are so important when I interact with people, that I refuse to use avatars without them. And I’ve memorized my gestures, so I thoughtlessly use them as I talk. Apparently so well, that people ask if I have facial tracking. I wish… If I had facial tracking, I’d be able to use hand tracking too. I won’t use the hand tracking feature, currently, because I can’t use gestures to make expressions while I use it. I’m so excited for the technology~

[@thelaughingmanofficial]

HTC has been really good at Discontinuing their useful products, which is why I left them for the Valve Index a long time ago and never looked back. At least I can mod a Valve Index somewhat to support face tracking.

[@kjjp5072]

Wishing you a speedy recovery .WingZero.

[@Suzuki_Hiakura]

I recall seeing a listing for plug and play HTC Vive eye trackers... was 1400 euros back in 2020. 1400 for two plastic clips that hold a single camera pointed at the eye, and patched into the HTC Vive Camera... the price for convince will remain high so long as most people want the simple solution that doesn't require soldering or setup, and if it isn't natively supported in the application, they likely will not blame the +$1k addon. Didn't know they had a solder solution, and will be looking forward to that, as last I checked there were no good solutions :D

[@DraterTTV]

I hope you get well soon... 2 months belated

[@m6ddiex]

hope the surgery went well!!

[@DvLRaphy]

i have been playing vrchat with face and eye tracking with my vive headset since 2021 and that has made it really difficult to adapt and move to another headset if I'm playing VRChat. it makes a hell of a difference to my experience in the game.

[@xKittyVR]

when me and my friend have a full face tracking convo we start wheezing its so funny 🤣🤣🤣

[@cryppi1510]

I am fairly sure the Quest 4 will have facetracking because of quest pro's appeal for facetracking and the apple vision pro's face tracking, so I'm waiting for the Quest 4 to see if they have face tracking with a nice headset at an affordable price!

[@FragrantPixelSociety]

I believe foveated rendering should be a feature in most if not all headsets in the future. It should be a reason to add eye tracking for people that want to use it for vrchat, or for people that want more performance. Or both!

[@lg.-]

I hope your surgery went well!

[@aliceisaliveVT]

Literally getting my Quest Pro next month so I can start recording music videos and skits with face tracking :3

[@Retromancer_Rackham]

Star citizen has had this tech for several years, it really is a game changer, it has head, face and eye tracking, and it will also get vr according to the guy dealing with the vulkan integration for it in his words "even if I have to code it at home in my own time, it will have vr" lol when you might ask? who can say they are not known for rushing things lol. but the eye, head and face tracking and proximity voip are all in already. If you want to see it in all its glory check out any of the multitudes of Star citizen machinima that exists because of it. :)

[@jean-baptistefox2819]

and once you do express yourself out of gestures, you tend to forget to use it on non facetrack avatar ^^ I'm deep deep into this rabbit-hole and I like it.

[@Lucid5h4d0w]

nyaw, hope u get well soon phia

[@chaosfire321]

I've always wanted more tracking available in VRChat. Initially, I thought VRC eye/facetracking seemed off. When anime characters are making these intricate and almost non-exaggerated expressions, it seemed to fall into an uncanny valley of sorts. It seems like it's a matter of how the original avatar was styled as, since IMO it looks a lot cleaner on you than Thrill. I think that's why a lot of the furry avatars seem better for it.

[@r0nium]

the skibidi toilet at the beginning caught me off guard 😭 i can't escape it

[@Deftempura]

Im getting my first headset Tomorrow, a Quest Pro. I'm sure you know why i chose that model ^^

[@trawll8659]

as someone that motions a bit with their hands while talking the hand gestures in VRchat make it really hard to convey the proper emotions, being on the fence about jumping into the VR enthusiast rabbit hole I hope a gaming focused standalone headset comes along with native face/eye tracking.