Unity3d AR Foundation - Adding Face Masks with AR Face Tracking And Turning Face Tracking On and Off

AR Foundation Face Tracking with Unity3d is used to create an augmented reality scene where I walk you through the implementation and all the components required to use AR to perform face mask swapping and also toggle of face tracking. We also go through the code and I explain to you how everything is configured and how the source code works.
New AR Foundation Components used in this Scene:
- AR Face Tracking
- AR Face Controller
- AR Face
ARKit 3 released for under iOS13 Beta recently and in this video we go through and use it to create a mesh of our face in real-time.
Recommended Unity Asset For Today "AR Foundation Editor Remote"
assetstore.unity.com/packages...
** Support me by Subscribing to the channel as it will help me in bringing a lot more content **
bit.ly/2NmnFMu
** Find the source code in Patreon and support me if you can even if it is with one of the lower tiers as it helps me in bringing you more content - THANK YOU **
/ dilmerv
** EQUIPMENT USED TO RECORD THIS VIDEO **
- Photography Photo Portrait Studio 600W Day Light Umbrella - amzn.to/2WwI0Tc
- 1350W Photography Lighting Softbox Lighting Kit - amzn.to/2MgvV0r
- LimoStudio 6 x 9 ft. Green Muslin Backdrop with Ring Metal Holding Clips - amzn.to/30NofpT
- AmazonBasics 60-Inch Lightweight Tripod with Bag - amzn.to/2Wd8utF
- Canon EOS Rebel T6 Digital SLR Camera Kit - amzn.to/2WcnRlV
Supporters in Patreon help me and show your appreciation for all the content and source code provided thanks!
github.com/dilmerv/UnityARFou...
Support me in Patreon so I can keep doing stuff like this for free!
/ dilmerv
Get realtime indie development tips from me in Twitter
/ dilmerv
Subscribe to my indie development videos & game releases newsletter
www.dilmergames.com
Blog about Indie development and startup
www.dilmergames.com/blog

[@keithkatane4854]

You're damn right about the depth. Especially with that first mask. Thanks a million.

[@dilmerv]

You are more than welcome my friend ! Thanks for your time and feedback.

[@sudahmalam1]

wait,.. what,...?? this is amazing playlist and comprehensive topic about AR,..

[@markbakker97]

Wauw you produce really great and interesting learning video’s on KZbin 👍👍👍

[@dilmerv]

Thanks Mark and welcome to this great community :)

[@jn4244]

thanks for a great tutorial!! using modulo operation makes swapCounter increment a bit easier to read, something like below. swapCounter = (++swapCounter) % materials.Length; anyway thanks for those great tutorial series!!

[@dilmerv]

Cool thanks I like that better :) also for your time !

[@arpanrajpurohit]

Great work @Dilmer Valecillos.However I wanted to point out that I started seeing your tutorial and thought that from ground up maybe you will explain with dependency installations. However, you reference old tutorials for this but I keep on searching for old tutorial[this is kind of 2 or 3rd video of yours]. Maybe I think it will be useful to you. and one more thing pretty little but I find it really different that I am overwhalmed by already gameobjects in scene so if you could make the scene and process from ground up I think your tutorial will be more helpful. I really appreciate your work.

[@SergeCao]

you are genius dude))

[@dilmerv]

Thanks dude I am glad you liked it !

[@MohanKumar-gj9th]

how to apply the camera texture from face tracker on top of face mesh ?

[@pulkitmidha5710]

Making a zoom like virtual background feature is possible with ARfoundation/ARkit?

[@xrdigitech]

Hi @Dilmer Valecillos When i am testing on android, the mouth and eyes meshes are filled with mesh but in your video there is whole in your mesh when opening mouth and opening eyes. so can you please help me to resolve this mesh related thing?

[@miguelomartorresneyra2937]

I also have the same problem, @Dilmer Valecillos could you please help us?

[@KoradiyaAbhay]

I have same issue.

[@virtualcity365]

Do you have any idea about face tracking by website.

[@jdgcol]

you can use this like make a coloring book for kids whit ar foundation? like some scan the texture and put in to the scene, i have a small proyect to put the drawing on a tv ramdomly if you can help me, tks

[@dilmerv]

Good luck on your project and currently no much time to help but best to you !

[@sabrango]

Amazing!

[@dilmerv]

Thank you Sab I appreciate your feedback !

[@anupammahajan7540]

Your Tutorials are really helpful. Do you have any solution for face tracking in standalone Desktop unity build ??

[@dilmerv]

Not at the moment but thanks for your question.

[@asifarefin1750]

Hello! Been watching your videos for some time now. Watching your tutorials I was trying to make an AR App. In my app, say when I press "Button A" a lipstick texture gets added to the lips, then when I press "Button B" a mascara texture gets added to the eyes. But I want both the buttons to be active to show both the textures simultaneously if pressed. Can you please make a video tutorial regarding this? That would be really helpful

[@seanleenew]

When I build on Mac, error in console ‘ The type or namespace name ‘PositsionTool’ could not be found (are you missing a using directive or an assembly reference?). Need some help. don’t know where is this ‘PositionTool’.

[@seanleenew]

In PositionMoveTool.cs

[@harshitgaur2147]

Hey @Dilmer Valecillos , I have been following your this face series. i did try the first face mask video of tricolor. but that video you made using arkit and i doing using arcore, there are two issues i am facing in that , the camera jitters a lot when i start the app and also the part of face like eyes and mouth ar covered by that tricolor texture unlike in your video can you help me making this using arcore. Wait you your response and Your vids are AWESOME!

[@dilmerv]

Thanks for your feedback, honestly I haven’t tried any of the AR Foundation videos with Android yet but I plan to do so very soon and do comparisons between the two. Thanks again for watching the videos !

[@jyc3282]

Awesome! I was wondering if you could let us know how to manipulate existing facial character such as sloth, later on?

[@dilmerv]

Thanks not too sure what you are looking for? Can you provide more details ?

[@jyc3282]

@@dilmervlike kzbin.info/www/bejne/ZpC0onmbj7mkrrs how to connect your awesome technic and existing character file such as .fbx or .obj

[@malik_chaabane]

Amazing! Can i use it for complete head tracking with a different backgound? (like animoji on iphone x)

[@dilmerv]

You could :) just with a few modifications !

[@malik_chaabane]

@@dilmerv Thank you 😊

[@dilmerv]

You are welcome !

[@almirbolduan]

Do you know an easier way to test AR Foundations Apps on iOS? It's boring to build every time to see the results on device... :/ Thanks for your awesome tutorials!

[@dilmerv]

Looks like there is some work been done here forum.unity.com/threads/ar-remote-tool-alpha-preview.573229/

[@danielrosales4029]

Hi, thanks for sharing! Do you have a version for windows or someone tried to make work it?

[@dilmerv]

For windows I haven’t done anything, do you mean running this on Android in windows ?

[@Fangh44]

ARFoundation is only for iOS and android.

[@harshpatel8748]

How to implement background image behind the user?

[@dilmerv]

Good question, I haven't mess with this in a while, but I will look into updating this video.

[@sabmory]

Hello. Thnx for your amazing tutorials. I have a very big problem, application crashes randomly and I have no clue what's the reason. Tested on iPhone X and iPhone 13. It happens with and without my face in the scene. Can u please help me? TY.

[@dilmerv]

There could be so many different thing causing this issue, I would suggest building with development mode and looking at the logs in XCode, more likely there is a setting missing from player settings iOS or you are missing the XR Plugin for iOS. Thanks let me know if that helps.

[@sabmory]

@@dilmerv Hello again. Just read my comment again and saw that I forgot a very important clue. Memory consumption ramps up in a second to about 2 gigs and then the app crashes with almost no memory left on device. Have you ever seen this?

[@lab450]

Thank you for good tutorial! It is really helpful. I have been followed your ARkit tutorial so I upgraded my phone to ios13 beta. But now I have problem because now I need AR remote.... Is there any way to use AR remote for unity on ARkit3??

[@dilmerv]

Currently I haven’t found a way to get the ARKit Remote working with AR Foundation, let me do some additional tests and see if can make it work.

[@Novecento]

The face tracking toggle button doesn't work, the face is always visible even when I turn it off. Can you help? thanks

[@dilmerv]

Did you implement your own code or use my repo ?

[@misterttr8821]

Hello Dilmer Can we get 3d obj from unity and write code to control in java android studio?

[@dilmerv]

Normally I would start with Blender3D or Maya3D or any other type of 3D Modeling tool for what you are asking.

[@Fangh44]

Hello. Does anyone know how the UV of the face mesh looks like so I can for example draw circles around the eyes ?

[@sabrango]

you need to try blender SPECIFIC uv MESH AREA MATERIAL method, AND THEN TRY AND FIX METHOD,

[@dilmerv]

I would say create a rough texture with a UV map (numbers from 1 - 20) something like that and then play it with your own face, that should give you an idea and template to create cooler textures.

[@Fangh44]

@@dilmerv This is exactly what I did : drive.google.com/open?id=1_2Fhc68fIMOv_onygG58JtgMeNp8awjw

[@chandhanamahesh673]

How to do with swapping of dresses in body

[@dilmerv]

You would need to look into body tracking tech

[@prabhakardeep5921]

I developed it for android 9.0, after opening the app the screen goes black and have two button at bottom(face material and face tracking).

[@dilmerv]

I don’t believe this works for Android since it uses ARKit specific features.

[@prabhakardeep5921]

Thank you, @@dilmerv. I tried your Human Body Tracking with Unity tutorial(kzbin.info/www/bejne/mZeuq4iPfZmcbaM), it doesn't work either. Does that also uses ARKit specific features? if yes, then how can I make it work using ARCore?

[@barreradev]

Hola Dilmer, Sabes si en un iPhone7Plus se pueda hacer? Gracias.

[@dilmerv]

No se puede, solo estos devices An iOS device with front-facing TrueDepth camera: iPhone X, iPhone XS, iPhone XS Max, or iPhone XR. iPad Pro (11-inch) or iPad Pro (12.9-inch, 3rd generation).

[@mrlevo27]

When I take a screenshot on android with the selfie camera it turns othere rendertextures black!!!! did someone had this issue?

[@dilmerv]

I haven’t had those issues before perhaps someone here has?

[@alvarotorreblancalozano6087]

great! Any idea How can I modify the mesh?

[@dilmerv]

How do you want to modify it ? What are you trying to do ?

[@alvarotorreblancalozano6087]

@@dilmerv like show only some glasses, for example

[@zyonneo]

With this can we add virtual try on like glasses and jewelry?

[@dilmerv]

Yes it could but it won’t be precise as this is only using face AI created through ARKit models.

[@zyonneo]

@@dilmerv from where do I start? OpenCV plugin for unity

[@vladkrutenyuk6715]

What packages shall I download to do that?

[@dilmerv]

You can look at my sample project in GitHub and it will show you the packages necessary in the manifest.json github.com/dilmerv/UnityARFoundationEssentials

[@jatinpawar8523]

Hey there man..... I would like to tell you that you are really doing great work and we are getting to learn a lot from your informative vids. Apart from this I have a doubt and it would be really great if you can help me out with it. Actually the the thing is I want to instantiate object in air and want them to be tracked by ARFoundation. I wanna do this without any trackable i.e. I don't want a plane and then hit it was a Ray and Instantiate a gameobject at the hit point. Instead I want to cast a ray from the camera then get a point on that ray by using ray.getPoint(distance); and then at that raypoint I want to instantiate a GameObject and want it to be tracked by ARFoundation. Is it possible to use ARSession.TryAddReferencePoint(myPose); In ARCore we can do so (without the need of any trackable)..... For example:- Session.CreateAnchor(myPose); And this will create an anchor at the given Pose. I want to do the same but in ARFoundation. Can you help me @Dilmer Valecilloss ??? Thankx in advance. PS: Keep up the good work.

[@Fangh44]

yes you can kzbin.info/www/bejne/g52VhpShrLeroM0

[@dilmerv]

I apologize for not responding sooner but I did a video on this and it is called ARReferencePointManager in AR Foundation so look into that docs.unity3d.com/Packages/com.unity.xr.arfoundation@1.0/api/UnityEngine.XR.ARFoundation.ARReferencePointManager.html

[@dilmerv]

Thanks for responding :)

[@saurabhshah559]

I have problem with code......eyes and mouth both are not visible when m apply this code

[@dilmerv]

Are you using an Android device or iOS?

[@joele.taverasmedina8657]

can i add earrings? My wife sells earrings and I want to create an app that customers can see how the earrings will look when they buy them.

[@raquetty]

Inside unity you can create different models. Maybe with a little reprogramming you can hide the mask and based on the type of face add the earrings close to the model's camera.

[@dilmerv]

Thanks for responding to this and I completely agree.

[@xunwuli6864]

Hello,how does it work on a PC?

[@dilmerv]

Hello thanks for your message but this doesn’t work on a PC, this is specific for Android and iOS.

[@abdoulraoufgambo]

cooll

[@dilmerv]

Thanks !

[@thx4playstudio973]

Hay you, github url can't download source code. Why?

[@dilmerv]

You should be able to, are you getting an error ?

[@msb_786]

@@dilmerv yess error link not found

[@hasnaat80]

@@msb_786 github.com/dilmerv/UnityARFoundationEssentials

[@StampedeGames]

Thanks for the great video! If someone is looking for face tracking in Unity Editor, I've recently added this feature to AR Foundation Remote plugin: forum.unity.com/threads/ar-foundation-editor-remote-test-and-debug-your-ar-project-in-the-editor.898433/#post-5933453

[@maherdhouibi3661]

i have a problem the script isn't working for me any ideas ?

[@dilmerv]

What’s the problem ? What errors are you getting ? More info thanks

[@maherdhouibi3661]

@@dilmerv thanks for replying, it says that the associated script can not be loaded. and all the scripts are in the scripts folder

[@dilmerv]

No problem I wonder if you used a newer version of Unity ? I haven’t upgraded the project for a while.

[@maherdhouibi3661]

@@dilmerv I'm using the last vision of unity

[@charlesferry9465]

Sir why facemesh is weird it full of mesh in eye and mouth how do I delete that? Please reply please 🥺

[@dilmerv]

You can’t delete that, that’s how the framework does it, ARKit and ARCore have their own way of constructing the face meshes. But great question thanks !

[@charlesferry9465]

@@dilmerv but its looks like weird in your iOS is awesome but in my Android eye and mouth are not invisible any way to visible? can you make a video for that? Thanks for reply sir

[@dilmerv]

Correct that’s because ARCore (Android) implements their face construction in a weird way, not much we can do about it unless you create your own implementation.

[@charlesferry9465]

@@dilmerv how can I create any advices please thanks for reply sir

[@ponzgamer8303]

is it IOS only ? or android too ?

[@dilmerv]

This works with both ARKit and ARCore

[@spadebussines8353]

Hi sir is it applicable if your platform is android ?

[@dilmerv]

I haven’t tested it with Android but AR Foundation works for both iOS and Android so I would say that it should work as long as you add the ARCore packages.

[@spadebussines8353]

@@dilmerv Hi sir i am just a beginner i've started to make apk on unity last week thank you for respose sir it already works but i cant change the material thing into 3D model is it posible to change it to become like snapchat application? thank you for your attention

[@jenpopovich2805]

exmafiascapegoat

[@ahmedserag7072]

hi this code can be used in android ? or iphone only... thanks in advance

[@dilmerv]

Hello there and yes until ARCore supports face mesh tracking...

[@dilmerv]

You will be able to when ARCore adds face tracking.

[@onuryldrm6845]

@@dilmerv did it come

[@jenpopovich2805]

jamie wyle blew up the tiguan

[@KoradiyaAbhay]

Hello in android tracking not work.

[@dilmerv]

What android device are you using ?

[@antivenomclan7255]

does he have a discord?

[@dilmerv]

I do but currently only through Patreon.

[@ravivishwakarma2728]

Sir is it possible android???

[@dilmerv]

It is possible, face tracking is supported for android just no Blendshapes

[@ravivishwakarma2728]

@@dilmerv and sir what about body tracking??

[@dilmerv]

Body tracking is specific to ARKit as well, face tracking you can use face meshing but unfortunately body tracking won’t work at all on android.

[@ravivishwakarma2728]

@@dilmerv sir any other way we use body detection on Android??? Sir is it possible that we use ML code of python of body detection can be imported in unity ??

[@najehsassi3821]

code source 404!!!!

[@dilmerv]

Try again I fixed the link and thanks for watching :)

[@najehsassi3821]

@@dilmerv Thanks Babe

[@chearner]

For the love of god, please take the word "basically" out of your vocabulary! Basically, you don't need to use it to basically say what you want to say, basically speaking of course. Basically, thank you, basically.

[@dilmerv]

Thanks for the feedback and I agree ! I have been trying ;) and will make sure to remember not to say it too many times.

[@chearner]

@@dilmerv Was trying to be more funny than rude ;) Otherwise I was able to get things working, your hard work was very helpful. Thanks.

[@laptoplifestyle5886]

You didn't make it clear... dislike

[@dilmerv]

Which part wasn’t clear and I can do a new video to ensure it makes more sense ?