AR Foundation With Unity3d - How To Track A Human Body in 3D With AR Foundation?

AR Foundation with Unity brings amazing new technologies available and provided with the effort of Apple ARKit Team and Unity3d Team. In this video I walk you through an example scene the Unity3d Team provided as a way to teach the community on how to use Human Body Tracking 3D Technology with Unity3d.
This video also provides you with requirements to build the demo scene shown in the beginning of this video.
Unity3d Example of Body Tracking can be cloned or downloaded from:
github.com/Unity-Technologies...
** Support me by Subscribing to the channel as it will help me in bringing a lot more content **
bit.ly/2NmnFMu
Support me in Patreon so I can keep doing stuff like this for free!
/ dilmerv
Get realtime indie development tips from me in Twitter
/ dilmerv
Subscribe to my indie development videos & game releases newsletter
www.dilmergames.com
Blog about Indie development and startup
www.dilmergames.com/blog

[@FrancisChenSF]

Holy crap how doesn't this have enough views. This is one of the most important videos ever for Body Tracking

[@dilmerv]

It doesn’t man and most of my videos have low views but they are increasing with time :)

[@izharahmad5467]

@@dilmerv Definitely sir it will increase with time since AR/VR are the emerging technologies and most the students have also taken this as career. You are doing great job sir well done keep it up.

[@christerf2002]

Thanks for sharing this! If anyone can figure out how to do mocap and hide their actual body it'll be a game changer!

[@dwassortedmedia]

It really would be! For now I’m wondering how well having a TCP client in the app and sending joint data to a server with the same model would work. Could see this working for live performances on projectors and such.

[@AnusruthyP]

Thank you so much for sharing such valuable content on AR Foundation With Unity3d! Your video was incredibly helpful in learning how to utilize ARKit effectively . Keep up the great work!

[@developer_shyam]

Awesome tutorial man. Thanks a lot. Will try building the sample in my iPad.

[@dilmerv]

Very cool thanks for the feedback !

[@hermanator123]

Just the tutorial I have been looking for! Thx man! so close to buy iphone wich support arkit 3 just because of body traking! Gonna do it very soon!

[@dilmerv]

That’s cool and thanks for the feedback, I agree with your comment and getting an iPhone that supports body tracking is a great investment :)

[@thatOne873]

Another amazing AR Foundation video! Imagine if we get the poses of the body and animate it into standalone character as an gameobject.. no need to use all the mocap equipment. This would be cool! Im so impressed how technology goes on!

[@dilmerv]

Yes I am going to be attempting adding a regular character and apply animations based on the movements in real-time :) tech is getting crazy I agree ! Thanks for watching !

[@thatOne873]

@@dilmerv It's a pleasure, can't wait to see it! : )

[@hermanator123]

@@dilmerv looking forward aswell (y)

[@tonirojas]

Thanks for this dude! Good job. Do you have, or are you planning to do, another video like this but on windows or android platforms? Will be sooo apretiated. Thanks again.

[@nwekarmahdi9211]

Thanks for your tutorial much appreciate it, I have a question could I replace this skelton to dress and work good ? could you answer me ASAP please

[@mubeenchandna2902]

Thanks for this dude! Sir can you upload like virtual try on Clothes etc please i was waiting for a along time

[@csuparna]

Hi Dilmer @Dilmer Valecillos , Thanks for doing this video. My problem is that the robot has a slightly backward bent knee even when the person is standing straight inside the AR frame. Do you know any solution to this. Would appreciate your help. Thanks again!

[@natalieieyoub5756]

Great video!

[@dilmerv]

Thanks a lot I appreciate it !

[@hnsalexchandriyaa4900]

Thanks a lot for the tutorial.. will Iphone XR support this feature or not??

[@kiranightray4970]

Thanks! Will wait for your next videos on body tracking. Could you make some tutorials on virtual try on of clothes, watches etc. Using these features? It could be of great business use. Thanks again!

[@dilmerv]

Sounds great thanks for the feedback and yes I will look into adding videos for business uses as you mentioned.

[@parthpatadiya9197]

@@dilmerv yes please i think 90% audience of this video landed here by finding some sources or help to create virtual try on of cloths in ar. so pls sir can you make a separate video tutorial for it ? it will help alot

[@mubeenchandna2902]

Dilmer sir One year ago i have not any scene your Video try on clothes

[@devanshgupta169]

Hello, can I save the tracked human model as a prefab to be used later.....Is it possible?

[@lostintheheatofitall]

Awesome tutorial! How would you use a model(prefab) that is not following the exact structure of the AR robot? For example my model only has 3 spine joints and 1 neck, instead of 7 and 4.

[@dilmerv]

I would say you would need to look into the joints implementation that AR Foundation provides and map the transforms to what you model requires.

[@gowthamn7004]

Hi Dilmer , i am working on a project where i need to adjust the clothes on a human body when a body is been tracked can you help me with this ....

[@rushikore]

As it doesn't support android phones. Is it possible to make web application which will run on webcam. As webcam supports body tracking. can we make web app from above asset

[@mohammedghabyen721]

Thank you so much

[@dilmerv]

You are very welcome !

[@user-fy4iq6if4z]

Thanks bro..

[@thornebrandt]

Has anyone found a solution for importing their own skeleton, like from mixamo for example? I've only found very crude hacks, and it doesn't seem to be possible to export the skeleton from the robot included in the project. The fbx has been removed, and it's insanely distorted if using unity fbx exporter.

[@m909PXLFRONT]

Excelent content

[@dilmerv]

Thank you

[@ajipus]

Halo Dilmer, terima kasih atas tutorialnya, sangat dihargai. Saya bertanya-tanya apakah mungkin untuk: - Apakah di Unity bisa menggabungkan baju digital ke arah human body target, apakah Anda tahu ada perusahaan/studio yang sudah menerapkan fitur itu, terima kasih!

[@muhammadtalhaali2541]

is ARcore version released for android or not ? been eagerly waiting for months for android :D btw thanks for teaching us

[@FerdiRizaldi]

in the bone controller, is the Index order of the indices is fixed? because as I read the code, it seems like the boneController maps through the Arhumanbody's joint with index number. Thank you for you vids btw!

[@dilmerv]

I believe the indices are fixed, I took a look through Unity's code also the BoneController and they also assume the indices to always have a specific integer value: github.com/Unity-Technologies/arfoundation-samples/blob/6cdf415fc78bc665fc6478fe5228a1ed08d4d8c8/Assets/Scripts/Runtime/BoneController.cs#L12C14-L12C26 also I recommend looking over Apple's docs: developer.apple.com/documentation/arkit/arskeleton Apple also states in their ARKit implementation the following: "A skeleton consists of a set of labeled joints that are defined in a certain hierarchy, i.e. joints are parented to other joints. One may use the parentIndices property to identify the hierarchy for a given skeleton definition." perhaps that can help. Thanks for your feedback.

[@wizzkidmaster]

Hi, great video like all the others! Did you ever get around to adding the pose recorder to your sample files? Where you can record the poses?

[@dilmerv]

I haven’t had a chance but I know someone in twitter did something similar. If I find the tweet I will post it here.

[@wizzkidmaster]

Ok that would be good thanks!

[@neilianaryan7939]

Hi there, I'm being bothered with a little problem. If I want to use other model which has its own skeleton instead of the robot model in the sample, should I remake its skeleton and make it exactly as same as the robot's skeleton(include the joints' positions&rotations)? Cuz the model always be twisted when I correspond the ARHumanBody.joints' pose to my model's joints' pose.HELP! T^T

[@yashvardhansingh8231]

did you find a solution for this ? please help me out

[@mohamadeibielsardjanto2906]

Cool tutorial man! Btw, is it possible to exclusively track some part of the body i.e. only the upper body parts (from head to arms)?

[@dilmerv]

It is possible and there is a Enum you can use to tell the system what bones to track, I will be sure to do a video similar to this very soon.

[@mohamadeibielsardjanto2906]

@@dilmerv woah ill be waiting for this video then haha keep it up man

[@alinazhuang904]

@@dilmerv please to do the video😢 I wanna track just hand but I failed

[@z3ven46]

Cool video man, Thanks. But, can i recognize only my hand and use it as command gesture?

[@dilmerv]

Hello there and thanks for watching, this is mainly body tracking, hand tracking for gestures would need to get more detailed into each finger, there is a IOS ML model that has been trained which can help with that, also Magic Leap provides hand tracking technology.

[@rawbbtc]

Hi there... thank you so much for the video, such exciting applications as a choreographer... Im having real trouble loading the demo. Tried everything. Ive installed all the dependent packages with package manager and downloaded the zip - when I try to drag the ARFoundation samples into assets it comes back with multiple errors - please select a valid package.json file in a package folder and you are trying to import an asset which contains a global game changer. In turn when I load the demo scene (3d body tracking) there are multiple missing mono scripts and compile errors. Tried everything including different installs. I am new to this in every sense trying to find my way, am I missing something really simple? I feel like im trying to install the ARFoundation samples in the wrong place (assets) but have no clue where else I can get the demo scene to load up. Any help would be so appreciated, this work is really important to me. Thanks in advance if anyone can help!

[@DDHH04]

Hey did you figure out that problem because im a beginner to and I have the same Problem

[@marieltellezleon5711]

@@DDHH04 It is happening the same to me! Did you figure out this problem? It is especifically with a Srcript

[@marieltellezleon5711]

@@DDHH04 In my case! I already Figured out... I went into my Manifest.json and checked packages. All was good! I added this line only "com.unity.xr.legacyinputhelpers": "1.3.11", and with this, the repositoory from github worked out too good!!! I hope this could help you. Save the manifest, close your Hub and your project warning windows that appear. Open the Hub. Select the project open it and you´ll see everything will be loading and then it will be opened. :D

[@KaiwenTeh]

I'm watching this video and it is December 2023. Is the information in this video still relevant today? If not, is there a similar video somewhere that is up to date?

[@theoriginalanimeker]

Hi do you think this project is translatable from AR Kit to ARcore and an android device that supports depth api? (It's a galaxy S22 Ultra). I've never made an AR app but I want this to be my first dabble into it, I want to follow along but currently only have an android device.

[@dilmerv]

Currently not supported on Android. Look into media pipes as they support body tracking.

[@siddiqaalhayki4962]

Amazing video !! Its really hard to find something about of body tracking Thank you !! Is there is a way that I can track the body with cloth dress, T-shirt ?

[@dilmerv]

Thank you, do you mean you want to put clothing to the robot? or are you asking if you can use body tracking with a dress or thirt? if so yes that is how it works.

[@siddiqaalhayki4962]

Yes I mean applying the body tracking with a shirt or dress can you guid me ?

[@robertoromano7625]

@@siddiqaalhayki4962 did you menage?

[@romethelife]

What’s a good way to record AR videos on the phone and save them? The demo video you had at the beginning looks clear how did you record that?

[@dilmerv]

Using the built in recording functionality that iOS provides :)

[@joshpj5581]

@@dilmerv Are you talking about screen record which is enabled from the settings?

[@siddharthsaxena2560]

Hey Dilmer, thanks a lot for the video! Do you know if there is a way to get this to work on WebAR using the webcam on a Mac or PC instead of on the phone?

[@dilmerv]

With this tech you can only do it with iOS ARKit so yes you need an iOS device that supports body tracking to be able use it.

[@siddharthsaxena2560]

Dilmer Valecillos Thanks very much for the reply, Dilmer. Do you have any sense of what would be the best body tracking solution for a browser-based Unity application using the webcam on a Mac/PC?

[@mohdgufranansari4835]

@@siddharthsaxena2560 mediapipe will do the job

[@ryutaro765]

Why don’t you post the related movie about how to implement body tracking system with AR Foundation? I’m looking forward to see the latest info!

[@dilmerv]

I will be re-making this video thanks for your suggestion !

[@yvnowww]

That's really amazing bro! Very helpful. But please let me know if it is updated to support android now. And do you get chance to work on Virtual dressing room application? And you said we can get info of few body parts in android can you please let me know to what parts we can apply this in android?

[@dilmerv]

I haven’t seen anything available yet for android but I will keep an eye on ARCore updates ! Thanks for the interest...

[@thespiralofpowah]

does it use VRM files?

[@OnlyTiramisu]

Hey kudos this is great! I can't seem to find something similar for ARCore- at least for upper body.. do you happen to know of anything you can suggest? Thanks again and keep it up!

[@dilmerv]

Not sure about android but let me look into it to see what I can find.

[@assaduzzamanashique2570]

@@dilmerv It is not working for Android. :(

[@dilmerv]

Ok thanks for the info

[@zippycat]

Is there another way? like i want to make android vr game with hand tracking, but i want it be with qr code tracked, so it be more responsible.

[@dilmerv]

What hardware do you want to use? It sounds like Android but currently that would apply to Oculus Quest or using few plugins available today such as www.manomotion.com/products/

[@chaitanyadeshmukh7049]

is there any video you uploaded that teaches how to place for example a Ring on a specific finger of the hand?

[@dilmerv]

Currently not supported with this technology, I will look into other options in the near future thanks for your interest.

[@chaitanyadeshmukh7049]

@@dilmerv do you have any idea which technology can help me to do this ? P.S. I like your video, I saw it on Instagram then after knowing about this channel yesterday I feel so grateful to you for these tutorials... :)

[@NONAME_G_R_I_D_]

Hey there, Happy Easter !!! Thank you for your videos!! they are great! I have got a little question, why does unity can not load the scripts when I load the scene? and there is also no script with ARHumanBodyManager name? What do I do wrong? Thank you anyway and wish you all the health and strength for these wired days

[@patrickmeehan6485]

Hi Idan, did you find a solution to the missing scripts?

[@unnos682]

@@patrickmeehan6485 in the ar foundation sample scene the scripts dont run properly. but if you copy them to a new project they work. but im still trying to manage body tracking to run on ARCore. the scripts are ok but nothing happen. are you trying to run it on android or ios?

[@NONAME_G_R_I_D_]

yes !! I did! thank you very much and hope that you are safe these days@@patrickmeehan6485

[@mohammedghabyen721]

Hi, want to ask, can we do that with the front camera?

[@dilmerv]

No currently but I am sure it will change :)

[@ashutoshkumar-mi4id]

Can we use AR clothes try-on using the above body tracking.

[@dilmerv]

Absolutely you can attach anything you like to the body.

[@alkacappellazzo4959]

@@dilmerv Hi Dilmer! Very useful tutorial! How can I attach a dress to the body?

[@anonymoussloth6687]

Any way to do this with real time video through a camera?

[@dilmerv]

Not currently with these tools, great question and I agree that would be super useful.

[@aditya-5924]

Hi, is this iOS only? or can we have another tutorial on android version?

[@dilmerv]

Currently iOS ARKit is the only SDK supporting AR body tracking, I will announce it if ARCore makes it available on Android, thanks for your question and interest.

[@keithkatane4854]

Hi, guys! Anyone else having trouble importing from the repo after cloning/downloading? I'm getting "Cannot perform upm operation: Unable to add package," error. Thanks.

[@dilmerv]

More likely is because you upgraded the project, I haven’t updated this repo in a long time so packages may be out of date. I recommend cloning Unity AR Foundation examples at github.com/Unity-Technologies/arfoundation-samples Thanks !

[@keithkatane4854]

@@dilmerv Thank you very much. It appears that the link in your description and the link in your reply are identical. In other words, this was the link I was already using anyway. But I'll keep trying. The issue might be something I'm doing wrong in Unity or VS.

[@kienle1215]

Hi Dilmer. Thanks for great tutorial. How can I replace that robot with my T-Pose model? I tried to download a T-Pose model on Mixamo and replace the robot but it didn't work. Do you have any advice? Thanks

[@jingga_8846]

me neither...

[@CtrlHelp]

Does the front camera work too? Android and iOS

[@dilmerv]

Great question, but unfortunately ARKit only supports the rear camera and it is only supported on iOS devices. For Android, I recommend looking into MediaPipe. Thanks.

[@kishorgujjar1193]

Hey Dilmer, I have already done Human body tracking in 3D with iOS device .But now I have to implement whole processes in Android. Do you have any idea by which I implement this all in Android devices. Just Acknowledge me.

[@dilmerv]

I haven’t seen this working in Android at least as of now but there may be an update from ARCore that perhaps includes it, I will look into it.

[@yvnowww]

@@dilmerv is there any update to support android bro???

[@dajoey2198]

is this only works for iOS? because in the feature support ARCore didn't support body tracking

[@dilmerv]

That’s correct this is only supported by ARKit in iOS.

[@dajoey2198]

@@dilmerv so it is not possible to implement body tracking in android then? or you actually can but with another sdk?

[@vincentlew1239]

The robot doesn't show...Help...I'm using unity 2019.3.0f1 version, iOS 13.2.3 and Xcode 11.3 beta

[@dilmerv]

Vincent Lew I haven’t tested it this feature with 2019.3.0f1 but I will and let you know my findings.

[@vincentlew1239]

@@dilmerv Is the hardware device must have A12 bionic chips?

[@yimingzhong4098]

Thank you for your video! It might be a stupid question, but why sometimes it recognizes an object like a water bottle as a human?

[@dilmerv]

Interesting I haven’t experienced that issue, what phone are you using and iOS version ?

[@yimingzhong4098]

@@dilmerv iphone 13 pro and version 16.1.1

[@ayaashraf674]

which file i must run in unity after i clone the project

[@dilmerv]

Hey thanks for your comment, but I am not sure I understand this question. You would need to deploy it to your iOS device to test it out, or use the AR Remote for AR Foundation found in the Unity asset store.

[@ayaashraf674]

I was want to run the project in my Android phone so I clone the project from GitHub then I connect my phone to unity throw unity remote app then I run many files I was want the camera be open and the avatar display like the demo in the video but this doesn't happen and I don't know what is the problem😢🚶🏻‍♀️​@@dilmerv

[@kishorgujjar2382]

Hello Sir, Do you have plan to make a tutorial about Virtual Dressing Room/ Virtual Fitting Room - Augmented Reality Unity app demo

[@dilmerv]

Not at the moment but I may do a video in the near future with that type of idea.

[@kishorgujjar2382]

Hey,@@dilmerv Please, if I want to learn this process, then you help me. Be grateful

[@yvnowww]

@@dilmerv did you work on that virtual dressing room project by any chance bro? 😄😁

[@catrinazou764]

@@dilmerv Could you upload a virtual try-on tutorial video? Thank you so much!

[@shashankkamath2171]

Sir can we do this using AR Core? Or is it Ar kit exclusive?

[@dilmerv]

ARKit exclusive unless AR Core added the feature which I haven't seen yet.

[@maheshvicky6078]

@@dilmerv can u please inform here once arcore provide full body tracking......plz

[@kucingfires]

cooooolsss

[@dilmerv]

Thank you !

[@hoomibz6219]

Thanks Thanks Thanks a lotttttttttttttttttttttttttttttttttttttttttt

[@dilmerv]

Hehe thank you my friend ! Have a wonderful day !

[@usmanarif8291]

How can I detect only feet in Unity 3d? Guide me please.

[@dilmerv]

Body tracking is probably not a good solution since you would have to be far from the camera to detect your own body. I would say look into machine learning solutions to accomplish this type of task. Look at lobe.ai as they provide a free and robust solution for AI detection and training.

[@nashlabs]

Can i just track my leg and foot movement ?

[@dilmerv]

You can track the skeleton as a whole but not specifically parts of your body.

[@asurstark]

Can we make this for android instead of ios?

[@dilmerv]

Currently this implementation is only available for iOS, I recommend looking into Google media pipes for an android alternative, great question !

[@nickname8668]

Can I do it on the front facing camera

[@dilmerv]

Unfortunately is only supported with the rear-facing camera as described here developer.apple.com/documentation/arkit/arbodytrackingconfiguration

[@nickname8668]

@@dilmerv is there anyway to achieve on android phone?

[@dilmerv]

@@nickname8668 you can but with a completely different system, check out developers.google.com/mediapipe

[@nithyashreeraja5241]

when i use the apk, just camera opens and nothing is tracked, can anyone spot what mistake am I doing, ps: I am using android

[@dilmerv]

Yes Android doesn’t support these features, take a look a media pipes with Android.

[@nithyashreeraja5241]

@@dilmerv oops my bad

[@dilmerv]

No problem that was a great question, thanks for your time :)

[@ichrafkhalfaoui646]

Can I do this on Android?

[@dilmerv]

Unfortunately, not possible with ARCore but yes with Google Media Pipe 😉

[@daniyalsaleem4763]

Will it work with android

[@dilmerv]

Unfortunately this specific tech only works on iOS ARKit, however you can use Media Pipe by Google which also provides body tracking, take a look at: google.github.io/mediapipe/

[@alishasingh9183]

Can I implement it in unity 2020 ?

[@dilmerv]

Yes you should be able to ;)

[@krishnakantsharma1053]

Please mention in the thumbnail that it is supported in android or not.

[@dilmerv]

I will in future videos thanks for your feedback.

[@gabrielpintohernandez4586]

this project is avaliable for android?

[@dilmerv]

Some parts of this project are but full body tracking is not currently supported.

[@yvnowww]

@@dilmerv what parts can be detected in android bro??

[@zyonwong]

does this work for android ?

[@dilmerv]

It doesn’t unfortunately, it is currently just supported in iOS ARKit. Great question.

[@Sureshchary1221]

Is this support for Android ??

[@dilmerv]

This is only supported on ARkit (iOS) I will let you know if that changes.

[@Sureshchary1221]

@@dilmerv any other way to track body in android device with unity, if there is any possible please let me know sir, little bit urgent

[@m909PXLFRONT]

you are from Venezuela ?

[@dilmerv]

Yes soy de Venezuela 🇻🇪 saludos !!

[@m909PXLFRONT]

@@dilmerv que fino pana

[@sahilrane2128]

Can I Add My Own Skeleton in this Project

[@dilmerv]

Yes you can, the skeleton prefab can be changed, you just need to follow the same bone naming conventions and structure when you create your own rig.

[@thornebrandt]

@@dilmerv How do we export the robot skeleton from the project to get the bone rotations?

[@shbarz6303]

1k like

[@dilmerv]

Thank you

[@mrpotato8196]

as a teenager... WTH AM I GETTING MYSELF INTO

[@dilmerv]

You are in the right path 😉

[@lukis6408]

please develop for android

[@dilmerv]

Sure when this feature becomes available for Android I will :) thanks for your suggestion.