Advanced Tutorial-OpenCV in Unity

[@Samurook]

A quick tutorial for a simple yet awesome application using only free tools. And you even took your time for some reflections at the end. Good stuff!

[@shubhamprakash9201]

@charulata vidhani you can do this by simply changing the target build to android platform and for this you need android sdk,jdk,ndk installed into your pc. Hope this helps you

[@burhanuddinlimdiwala6754]

@charulata vidhani if you want to deploy In android I can help you

[@charlytaylor1267]

Where did you get the objects/materials he used? When i searched on the asset store for flappy bird nothing came up.

[@turktube6929]

hi very amazing thid toturial is very useful........................... pleas continue learn open cv from basic in windows and android plarform

[@Sohneg]

Also if you build the game/program: place the haarcascade_frontalface_default.xml directly in the first folder in unity (assets) and change the Application.dataPath to System.IO.Directory.GetCurrentDirectory + @"/path where the xml file is in the build folder" The build cant find dataPath and also not when its in a under directory.

[@mhdrxshid]

Bro i Have an null reference from "Mat frame = OpenCvSharp.Unity.TextureToMat(webcamTexture)" how to fix this

[@itszezwa5962]

thank you so match 😘

[@c_42_pallavipawale77]

I have the same problem. Did you fix it?@@mhdrxshid

[@thegrey448]

very good tutorial . anyway is that possible for test it without webcam on pc? just wanna try directly build on the smartphone? thanks in advance.

[@jack750304]

Has anyone successfully run this face recognition feature on Android? I installed it directly on the phone and found that even the camera function does not work properly 😓

[@Sohneg]

If somebody has problem with the haarcascade_frontalface_default.xml. In my case it wouldnt found this file till i added a / infront of the path: cascade = new CascadeClassifier(Application.dataPath + @"/haarcascade_frontalface_default.xml");

[@Sohneg]

Also if you build the game/program: place the haarcascade_frontalface_default.xml directly in the first folder in unity (assets) and change the Application.dataPath to System.IO.Directory.GetCurrentDirectory + @"/path where the xml file is in the build folder" The build cant find dataPath and also not when its in a under directory.

[@aarontse183]

bro thank you so much

[@erfanroghani]

Thank you for saving my time!

[@Sohneg]

@@aarontse183 your welcome

[@Sohneg]

@@erfanroghani your welcome :)

[@aniketcoder1469]

Amazing tutorial but this detecting other things of my room sometimes my face how i fix this any suggestions???

[@sebastianhagen2500]

It tells me: FileNotFoundException: "/Users/***/***/***/MotionController3/Assets/haarcascade_frontalface_default.xml"not found OpenCvSharp.CascadeClassifier..ctor (System.String fileName) (at Assets/OpenCV+Unity/Assets/Scripts/OpenCvSharp/modules/objdetect/CascadeClassifier.cs:40) FaceDector.Start () (at Assets/Scripts/FaceDector.cs:18) It's exactly as in the video.

[@betaga2261]

I have the same problem.

[@muzmuzz]

is there any solution with this ?

[@OsirisStBrown-pz8qz]

I get a NullReferenceException error in the findNewFace function, how could I resolve this

[@arty7171]

I too...

[@sarahgraf591]

If it is still relevant, I got the same error. Check the dataPath to 'haarcascade_frontalface_default.xml', after I had corrected it, it worked

[@nicknytz2491]

Yeah, I am getting the same error and I made sure that that the dataPath + "haarcascade_frontalface_default.xml" was correct in the script. Any clue what is causing the error? Thanks for your help!

[@tonclaryinet8823]

@@nicknytz2491 I found that it would resolve the error once I directly pasted the path of the XML file into the cascade classifier: cascade = new CascadeClassifier("Assets/haarcascade_frontalface_default.xml");

[@theqihuasyazwan]

@@tonclaryinet8823 This works! thank you! You have to also paste the xml file in the projects assets folder too!

[@agi-news]

a easy way to fix the Null error with the cascade is to manually download the Haarcascade file (its on geeks for geeks) and simply drop it on the assets folder in your project. Thats where his path is looking for it. Everything else seems to work fine

[@edberthansteddyatmaja3481]

Hello, do you know any sort of way to detect if an object in real life is colliding with your mouth part? would love to know about it and would listen to your advice. Thanks for the great video as well, appreciate it... Keep it up buddy

[@Yumenoki99]

Amazing, thank you! I could use some help with landmarks and don't know where else to ask. Maybe someone has an idea? OpenCV plus made me download a shape predictor, of which I changed the file name from .dat to .bytes, I've seen it work this way in the demo scene. The idea is to track the mouth using point 63 out of 68 so I can later check if the mouth has opened and then do stuff. Code: private ShapePredictor predictor; public TextAsset shapes; //the downloaded LandmarkLib private Point mouth; private void Start() { byte[] shapeDat = shapes.bytes; predictor = new ShapePredictor(); predictor.LoadData(shapeDat); //Telling the shape predictor to use the downloaded library } void Update() { myFace = //find the face like in this Tutorial, all working findLandmarks(frame, myFace) } void findLandmarks(Mat frame, Rect face) //this makes Unity crash { if (predictor.DetectLandmarks(frame, face)[63] != null) { mouth.X = predictor.DetectLandmarks(frame, face)[63].X; mouth.Y = predictor.DetectLandmarks(frame, face)[63].Y; } } How else am I supposed to track a point on the face without Unity crashing & saying "This application has requested the runtime to terminate it in an unusual way(...)"? Is it crashing because it is running slowly? Have I referenced or imported the shape_predictor_68_face_landmarks.bytes in the wrong way? There surely must be a way to use ShapePredictor.DetectLandmarks, right? Any help is appreciated

[@queriesask1765]

Hi, Thank you so much for the great tutorial. I am going to use OpenCV for my Unity project. In that project I need to implement Jump, Run, SwipeLeft, SwipeRight and etc gestures with OpenCV using a Web camera. Can you please confirm that the Jump, Run, SwipeLeft, SwipeRight, and etc gestures are possible with OpenCV pose tracking? If possible can you please help me or share some of the references for the above gestures implementation using Unity. Thanks in advance.

[@nguyenhuynhhaidang93]

thank you so much! Hope you make more tutorial opencv

[@CarolainAnto]

Thank you for the video! I have tried this and works fine on desktop but the CascadeClassifier returns null on Android. Could :c

[@zoyakhanam4215]

I have a problem the video through the camera is upside down. Please help me to rectify this

[@z1d2g3c4]

I love you, you make my life easier 🥰

[@humamrakha6092]

I have a problem in outputArrayOfStockList.cs. How to solve it?

[@thanhthai18]

can not detect on android device

[@HuyNguyen-vp7eb]

thats great tutorial, but it really slow, only 10fps when run (My pc i7 6700 + Nvidia 1060 ). How to improve performance?

[@bnok5237]

Hi , Great tutorial , but is there a way to get the rotation of the face ???

[@santanaaybar7466]

Just a suggestion if you're getting features such as eyes and mouth and you can tell which one has a higher Y component, you can tell whether the face has rotated

[@viney50567]

Absolutely great video! Thanks very much

[@fcandann]

Hi, very good tutorial. Can Open CV works on integrated graphics like intel hd4000?

[@gameplayhubindia]

Awesome tutorial

[@mariasanchezmarques4727]

Thank you for your video! it was very interested, and I have a question, Is possible to deploy a project in WebGL instead of android or ios?

[@m.a.ygames87]

You can build any of the demo scenes by opening them on the unity android platform

[@hakimduparcq]

NO does not work for android

[@queriesask1765]

Hi, Can you please share a few references for the skeleton tracking detection using OpenCV in Unity.

[@sashathemaster]

Can u do the same with hand detection instead of face detection using OpenCV+Unity?

[@juliangabrielalvarez7642]

Hi! I´m having a problem, could anyone help me? I followed the steps, but I can´t use the faceY variable, the way of calling it doesn´t seem to work for me.

[@AnilKumar-ws8rr]

How to get last frame value??

[@Cadknowledge]

Awesome can you create more tutorial

[@024_arindamsarkar3]

I am having this error " NullReferenceException: Object reference not set to an instance of an object FaceDetector.findNewFace (OpenCvSharp.Mat frame) (at Assets/FaceDetector.cs:32) FaceDetector.Update () (at Assets/FaceDetector.cs:26)" ... my plane material is not switching to instance ..please help

[@TechnikFreakYOUTUBE]

i have the problem to

[@ranliu6307]

I have this problem. I searched for so many ways to tackle it but always failed. DllNotFoundException: OpenCvSharpExtern assembly: type: member:(null) OpenCvSharp.NativeMethods.TryPInvoke () (at Assets/OpenCV+Unity/Assets/Scripts/OpenCvSharp/PInvoke/NativeMethods.cs:115) Rethrow as OpenCvSharpException: OpenCvSharpExtern assembly: type: member:(null) *** An exception has occurred because of P/Invoke. *** Please check the following: (1) OpenCV's DLL files exist in the same directory as the executable file. (2) Visual C++ Redistributable Package has been installed. (3) The target platform(x86/x64) of OpenCV's DLL files and OpenCvSharp is the same as your project's. System.DllNotFoundException: OpenCvSharpExtern assembly: type: member:(null) OpenCvSharp.NativeMethods.TryPInvoke () (at Assets/OpenCV+Unity/Assets/Scripts/OpenCvSharp/PInvoke/NativeMethods.cs:115) OpenCvSharp.NativeMethods.TryPInvoke () (at Assets/OpenCV+Unity/Assets/Scripts/OpenCvSharp/PInvoke/NativeMethods.cs:124) OpenCvSharp.NativeMethods..cctor () (at Assets/OpenCV+Unity/Assets/Scripts/OpenCvSharp/PInvoke/NativeMethods.cs:59) Rethrow as TypeInitializationException: The type initializer for 'OpenCvSharp.NativeMethods' threw an exception. OpenCvSharp.CascadeClassifier..ctor (System.String fileName) (at Assets/OpenCV+Unity/Assets/Scripts/OpenCvSharp/modules/objdetect/CascadeClassifier.cs:39) NewBehaviourScript.Start () (at Assets/Scripts/NewBehaviourScript.cs:16)

[@ranliu6307]

Mine is mac OS.

[@withtheglassesandthe]

Thanks for a great tutorial - I had to add the "lastY=faceDector.faceY;" line suggested by someone below, unsure if it's supposed to work without this, but it did the trick for me. However, the face tracking is really jittery and imprecise for me, it jumps between different values even when my face is completely still. Does anyone know a good way to calibrate the tracking, or alternatively, to smooth out the transition between values?

[@videogames7]

Hello, did you solve the jittering of the face tracking?, I have the same issue

[@withtheglassesandthe]

@@videogames7 Not really, but I noticed slightly better results from trying another webcam. The second one that got better results was 720p and the first was 1080, the 1080 one also caused my scene to lag more, so if you can find a lower res cam or scale down the input before sending it to unity, it might be worth trying.

[@MixxusStudio]

Great job! Thanks!

[@leandrocruz6621]

Espetacular, Seria muito bom também com o jogo tetris usando detecção de mão

[@thanhthai18]

Nice! Thank you!

[@EliteCubingAlliance]

Anyone know how to release the camera like in normal opencv with cap.release?

[@permentohi]

Can I move it with markers?

[@mchughm16a4]

Thank you fro the awesome tutorial! Do you know where to find any documentation on this? the domain in the asset store seems to not exist anymore. I was trying to build off this tutorial to detect facial landmarks by making another cascade with the shape_predictor_68_face_landmarks.bytes file, but the file seems to be empty so it doesn't work the same way. Do you know how to find facial landmarks using this asset? I can't seem to find anything about it anywhere.

[@AnkitSingh-wq2rk]

thank you bookmarking this video

[@zainabaslam1044]

Everything is working fine, but my bird in game is still flying upwards despite the updating the code as shown in the video, can anyone help me on this? what could possibly the issue? Below is the code for the bird: using System.Collections; using System.Collections.Generic; using UnityEngine; public class BirdController : MonoBehaviour { FaceDector faceDector; float speed = 1f; float lastY = 0; public void Start() { faceDector = (FaceDector)FindObjectOfType(typeof(FaceDector)); } public void Update() { float step = speed * Time.deltaTime; float norm = Mathf.Clamp(faceDector.faceY - lastY, -1, 1); transform.position = Vector3.MoveTowards(transform.position, new Vector3(transform.position.x, transform.position.y + norm, transform.position.z), step); } }

[@mustafaozcicek3090]

I have the same issue, has anyone figured it out?

[@zainabaslam1044]

@@mustafaozcicek3090 Yes, the issue is the last y value in the birdcontroller script being not updated, I added the following line of code in the script for the birdcontroller, see if it works for you: public void Update() { float step = speed * Time.deltaTime; float norm = Mathf.Clamp(faceDector.faceY - lastY, -1, 1); //below transform code is in one line transform.position = Vector3.MoveTowards(transform.position, new Vector3(transform.position.x, transform.position.y + norm, transform.position.z), step); //below line was additional added, then the character starts moving lastY=faceDector.faceY; }

[@stefanavintila8985]

Does anyone have the archive for the elements from Flappy Bird?

[@jeromedeng2372]

Excuse me, do you have github address? I want to refer to it for my class assignment

[@subwaye]

thank you very muchhhh🤩🤩🤩🤩🤩

[@scsky4084]

i have a problem. How to fix haarcascade_frontalface_default.xml"not found.

[@rustem1777]

Ты решил эту проблему?

[@burhanuddinlimdiwala6754]

Please provide object detection by open cv

[@alexlysov7883]

ok)

[@charlytaylor1267]

Where did you get the assets you used in this tutorial? I searched flappy bird on the unity asset store and these materials don't come up.

[@Samurook]

Flappy Bird is a 2D game, so assets are simply images. You could probably just make some yourself or browse the internet for Flappy Bird assets instead of limiting yourself to the asset store. You will immediately find someones GitHub page with all the images.

[@Didi-bu8xw]

笑死。看这个tutorial的时候刚好是海康威视在美被禁