Depth Estimation with OpenCV Python for 3D Object Detection

[@NicolaiAI]

Join My AI Career Program 👉 www.nicolai-nielsen.com/aicareer Enroll in My School and Technical Courses 👉 www.nicos-school.com

[@bhavikdudhrejiya852]

This channel should grow more. Amazing content on computer vision.

[@NicolaiAI]

Thanks a lot man! Really appreciate it and u definitely help a lot

[@shivammishra7306]

I had a question : In the code script you said that the value has to be multiplied by 205.8 to get the result in cm. What is the default unit of distance (the one seen in both frames) , and how do you find this multiplication factor?

[@georgesaavedra2624]

Hi sir, appreciate so much your videos about stereo vision, I was wondering if you can respond me just two questions: 1. What is your CPU power in this experience?, I mean processor, RAM, memory, etc. 2. What do you think is the minimum CPU power required to run this sketch and have a good performance?, my PC is more for ofimatic. Look forward to your reply sir, thank you for the videos.

[@shrinandaggarwal8564]

Is there a way to get the alpha value by software? I think after rectification, my alpha has changed since i am not getting proper distance values. (They are way off). Plus, what is the use of "f" as it is not used anywhere in the triangulation code.

[@강병선-d7g]

It's very helpful for me, thank you! Then, I have some questions about calculating depth 1. focal length Is it possible to use focal length from camera matrix or new camera matrix? 2. can you explain the difference between camera matrix and new camera matrix

[@razmandhamarasheed4325]

Hello there, Could you please make a vid about your setup and how to prepare everything so I can start working through your vids. Thanks in advance

[@grah2561]

Do you have a youtube clip where you discuss how to get a two-camera setup? Did you buy that or build it yourself? Do you have any recommendations? Thanks.

[@NicolaiAI]

Hi great idea, i should do a video about stereo cameras or how to setup on urself. In these videos i just use two webcams. Thanks a lot

[@grah2561]

@@NicolaiAI Cheers for the quick reply. Do you have good/accurate results with that setup or is that why you need good calibration? :) I am looking to do a project that includes depth-perception. I saw your clip about using transformations on a 2D image where you used a picture of a snooker table. I will definitely have a try at that and use the transformed 'bird-eye view' to see if I can estimate the position of objects such as the balls on the table. But I am also interested in stereo vision to see how that works and how it compares, accuracy-wise. I'm also thinking about picking up a cheap Kinect detector to see what I can do with that. Interesting subject!

[@NicolaiAI]

It is really interesting and u can create some really nice things with it. If u do a project where u need accuracy I'll definitely recommend to calibrate the cameras. Also because it's just run a script and that's it. A Kinect would also be a good idea and good accuracy but from my pov i think its cooler with stereo camera if it's just for hobby 😃

[@grah2561]

Are your webcams just bolted onto a frame next to each other? Just wondering what is the best way. Of course part of the fun will be me trying it out.

[@NicolaiAI]

I just kinda place the cameras and measure the distance between them. I could improve the accuracy from those videos if it was necessary

[@RiomHash]

When you are doing the calibration is the baseline 9 cm which what you set in 20:36? How would I go about finding the focal length?

[@lucasxas]

you can usually get the focal length from the camera manufactor. the baseline you measure in real life,is the distance between the two cameras

[@lorenzoaraiza5468]

Very useful video but I do have a question, and I'm sorry if you explained this and missed it, but would this work if there are multiple faces in the images?

[@NicolaiAI]

Thanks a lot! If you have multiple faces in the image u might have to apply tracking on top of it so have the same mappings. You could maybe also just sort all the bounding boxes in the x axis and then match those from left to right directly

[@jaeyounglee574]

D435i show us the depth, pixel x, y. Why explain about two camera? It must to be considering post-processing getting datas from D435i? Or user just using the depth, pixel x, y? Have to considering the relative values and applied to code?

[@ishaquealidad8361]

Hi, I am facing error while calibrating imge File "C:/pythoncode/StereoVisionDepthEstimation/stereo_calibration.py", line 75, in retL, cameraMatrixL, distL, rvecsL, tvecsL = cv.calibrateCamera(objpoints, imgpointsL, frameSize, None, None) cv2.error: OpenCV(4.5.5) D:\a\opencv-python\opencv-python\opencv\modules\calib3d\src\calibration.cpp:3694: error: (-215:Assertion failed) nimages > 0 in function 'cv::calibrateCameraRO i have debug the code line by line and the image path is correct debuged line: retL, cornersL = cv.findChessboardCorners(grayL, chessboardSize, None) it is not getting any corner and the retL is going false.

[@-ShangsitNath]

I am also having the same error. Did you got any solution to this error?

[@miguelribeiro8771]

what is the camera model you are using to do stereoscopy?

[@rameshtirumala9993]

Great tutorial pulling all aspects of CV together. You mentioned GitHub link for the code base, is that only if you register for the course?

[@NicolaiAI]

Thanks a lot! Most of my code is available on GitHub. Only the course code is not in there

[@rameshtirumala9993]

@NicolaiAI Thanks for the response, where is the link for the code and am I permitted to use it for my project with changes to fit my application

[@rameshtirumala9993]

@@NicolaiAI Also, are you available for consulting, if so please let me know I can set up time to discuss my project and help I need

[@NicolaiAI]

Yeah i do. U Can reach out to me on mail [e-mail] Then I’ll find the code on GitHub aswell

[@ahmaddar2632]

Very informative video..i have a question what is the maximum baseline which can be set for a stereo camera setup??? I would really appreciate if u answer this 😊

[@NicolaiAI]

Thank you! The baseline pretty much depends on ur application, if the cameras are too far away from each other u can only see very little in both images. If they are too far away from each other, u might not be able to see any correspondences

[@ahmaddar2632]

@@NicolaiAI thank you so much for your reply. If the stereo setup is convergent axis setup( toed in camera setup) then what changes are necessary to calibrate the cameras to use them for depth estimation?

[@twiincentral8780]

Thank you for the great videos. At what point does the depth accuracy drop? Is the distance between the cameras a driver of the depth distance?

[@NicolaiAI]

Yeah the baseline distance plays an important role in the accuracy of the depth but also how far/close u can get accurate values for

[@twiincentral8780]

@@NicolaiAI got it. So the resolution/quality of the camera/sensor can also play a big role? Once again, thank you for great videos, been binge watching! Will start implementing some soon!

[@NicolaiAI]

@@twiincentral8780 the resolution of the camera is the most important factor since u want as accurate disparities as possible. Thanks for watching! Glad that it can help u

[@darkside3ng]

Amazing work

[@NicolaiAI]

Thank you very much! I really appreciate it

[@darkside3ng]

The videos you have posted are of excellent quality. It has helped me a lot!!! Thank you :)

[@NicolaiAI]

Glad that i can help, thank you! :)

[@mohamadfaresalibek7547]

It's a great lesson You are really great and amazing Thanks for this good tutorials 💙

[@NicolaiAI]

Thank you very much! I really appreciate those nice words! 😃

[@inheelee3779]

Truly, indeed. I found that this is a kinda breakthrough for my project.

[@NicolaiAI]

@@inheelee3779 thank you very much! Glad that u can use it

[@myyard9693]

I got problem on stereo calibration file; on the line retL, cameraMatrixL, distL, rvecsL, tvecsL = cv.calibrateCamera(objpoints, imgpointsL, frameSize, None, None) cv2.error: OpenCV(4.5.1) C:\Users\appveyor\AppData\Local\Temp\1\pip-req-build-5rb_9df3\opencv\modules\calib3d\src\calibration.cpp:3694: error: (-215:Assertion failed) nimages > 0 in function 'cv::calibrateCameraRO' Please can you tell me whats the problem? Thanks a lot.

[@NicolaiAI]

Yes that error is because it could not load in the images. Make sure the path is correct and try display the images when loading them in

[@myyard9693]

@@NicolaiAI Thanks. Now its works!

[@NicolaiAI]

Great!

[@solaimanazzan2222]

@@myyard9693 Hey, can you show us how did you solve the problem ?

[@solaimanazzan2222]

@@NicolaiAI Hey, can you show us how to solve the problem ?

[@elisazanni9210]

Hi! I love your videos. Just a question: how do I find the focal length and alpha at 20:50?

[@NicolaiAI]

Thanks a lot! Those are physical camera parameters, can either be found in camera specifications or calibrated. Focal length and FOV

[@RiomHash]

@@NicolaiAI When you say it can be found by calibration, does that mean I can calculate for the focal length in mm from the calibration matrix obtained? Which of the calibration values obtained do I use and what equation to get the focal length? Likewise, can I calculate for the exact FOV based on the calibration data obtained?

[@sumitgupta3477]

Which stereo camera module I can use with Raspberry Pi Zero 2 W ?

[@shrinandaggarwal8564]

Hey thanks for the tutorial!! I am experiencing some issues with it though i hope you can help. When i run the stereo_calib code, my images from both the cameras are rendered useless.. I tried calibrating my cameras one by one and got good results( 0.08 error). But when i try to do stereo calib using the same setup, it just doesnt work..

[@NicolaiAI]

Hi thank you very much! Did u use the latest code from GitHub? When u load in the images with blob for stereo calibration u will need to sort the images or i will give very odd results like u mention

[@granatapfel6661]

Which camera are you using? And what about the waveshare IMX219-83 Stereo Camera, the camera don`t create a pointcloud?

[@michaelturner5359]

Hey great vid. How does it fair with more than one person in the frame? I done a similar project for my final year project and the person detection model would detect the people in the left and right frames in different orders and it would throw off the distance measurement.

[@NicolaiAI]

It should work fine but u would need to keep track of the people with an id assigned to them when coming into the frame and then u will just have an id with a corresponding depth

[@kenrampingplay]

Can you use this for custom object recognition? Like for example the face? And can it distinguish between a real human face with just a photo of the human face?

[@sarrabm3099]

Negative values: I have trained an object recognition model (using tensorflow api) and would like to determine the depth of the recognized objects (using the centers of the bounding boxes of each object as shown in this video). But I am getting NEGATIVE values. This may be because the detection is not detecting the same objects on the left and right side at the same time! Any helpful suggestions??

[@granatapfel6661]

can you tell me the packages and algorithms to do that?

[@yonasun2987]

@@granatapfel6661 yolov8

[@quyennguyenvan8608]

It's a great video but I have some issues though i hope you can help. How can I estimate depth if there are multiple objection of an classification in the frame ? ( ex: I have 3 cups in the frame how can i know depth of each cup).

[@NicolaiAI]

Sorry for the late reply. U can find the depth to each individual object if u are able to detect it. Then u can just directly copy this code and use it. In this video i onyl do it for 1 object/face

[@masadaham628]

Does it also work with two sepatate cameras aligned

[@NicolaiAI]

Yes u can make it work. I actually use that in the video

[@gokhanevlek8531]

Hey man I am following this video series it is amazing work, I appreciate. I need to make a project which needs to recognaze the person and compare with the picture which we have already, after that the program should be say this person who shows his or her face to webcam is this person which we have the picture. Can I make it with opencv and this video series?

[@NicolaiAI]

Thank you for dropping by! Yes u can definitely create something like that by using the techniques throughout this tutorial with opencv and computer vision methods. U won't get as good results as if u had multiple images of that person and could use AI. But it is definitely possible to do something with OpenCV! :)

[@gokhanevlek8531]

Okay thank you!

[@muammertiryaki4746]

hello thanks for your videos, I am trying object detect with yolov4 and with ros/gazebo simulation. But I have some problems in the calibration. after the calibration images turn to [ 0 0 0 .... 0 0] matrixes and load black screen. I can not understand how to debug this. And of course I made stereo calibration and I have stereoMap.xml without 0 0 0 0 0 (any things looking like yours)

[@QuarktaschemitSenf]

Can i combine this with yolov7 and track the distance of multiple objects? If so, pls do a tutorial, would be a hot topic. Greetings from munich :-)

[@NicolaiAI]

Hi I have a course about yolov7 then u can just run that on the stereo camera and use the same code as in this one here

[@QuarktaschemitSenf]

@@NicolaiAI my Problem is the following. In Yolov3 and v5 it was easy to understand which line is the actual detector. In 7 not so much for me. So i just run it with the cmd detect.py command. Is there somewhere an indepth - video that explains what part of the code from v7 does what? So i can run it in Visual Studio Code or another editor for example? Edit: Also ty for all your helpful videos. And how did you teach yourself those topics? Are there books you can reccommend on computer-vision ?

[@NicolaiAI]

@@QuarktaschemitSenf in the course I just export it as onnx and also modify the detect file for live camera in python

[@QuarktaschemitSenf]

​@@NicolaiAI : So this is already part of a course you've uploaded? Or did i get you wrong ?

[@NicolaiAI]

@@QuarktaschemitSenf yeah everything is in there already

[@raihankhan5160]

How do I know the alpha value? Is it coming from my camera specifications? or you calculated it before?

[@NicolaiAI]

Yes camera specifications. Its the field of view (FOV)

[@raihankhan5160]

@@NicolaiAI Thanks! Just wanted to ask if my cameras are placed in orthogonal to each other can I use the same procedure to calculate the depth estimation? I guess then only parameter changing is the baseline b ?

[@NicolaiAI]

It's a requirement that the two image planes are parallel for disparities to work. If they are not parallel the calculations are a bit more complex and the x disparity can't be used since the epipolar lines won't be parallel too

[@raihankhan5160]

@@NicolaiAI Thanks. I got the idea now! I need to do rectification, two homographies (3*3 transform) and then epipolar line now horizontal :)

[@iiiceeeeee]

@@raihankhan5160 hello, I'm working about 2 cameras placed in orthogonal to each other too. May I have your contact email?

[@padha735]

I am working on the similar project where I need to find depth of the facial landmark points. Can you guide me how to find distance(depth information) with single camera? with single depth camera (using RGB image and depth image)

[@NicolaiAI]

U can just take the index in the depth image of the facial landmarks

[@boatpoowadon6565]

Hi Nicro. How to fix this error? INFO: Created TensorFlow Lite XNNPACK delegate for CPU.

[@NicolaiAI]

It’s not an error, it’s just info

[@boatpoowadon6565]

@@NicolaiAI but i run this code and the camera is not open ;-; can u help me

[@aranisdas7159]

Can I replace the face detection for a sign detector? I want to use this for measuring depth of signs. The car will be moving, so I'm not sure if this is going to work

[@granatapfel6661]

It depends on the depth sensor in the camera. Contact the manufacturer of the selected camera and tell us your insights. If the camera is behind the windshield it could be tricky to detect the reflected signal

[@shivammishra7306]

Hey a small question. How do I know the Field of View of my phone camera?

[@NicolaiAI]

That might be given some where in the specifications for ur phone on the internet what lens and so on they are using

[@shivammishra7306]

@@NicolaiAI Appreciate the reply. Three more things : (1) Is it necessary to have same focal length and FOV for both cameras? (2) Where exactly are we using the focal length value (variable f) ? (2) If I give recorded video as input, is it alright if I give same fov value, even if FOV is diff for both cameras?

[@NicolaiAI]

U will need to have the same fov on both cameras or u will get really weird values

[@shivammishra7306]

@@NicolaiAI Thanks. Really appreciate if you could answer 2nd and 3rd question too.

[@NicolaiAI]

Third question won't work when no in 1 and second question is to do the depth calculation

[@panudetpanumas7019]

what camera u used ?

[@NicolaiAI]

Just two budget webcams

[@Mohammedalhalabi16]

Can this project be done using a mobile camera and Arduino instead of a laptop?

[@Dawid30303]

Likely not as you need two camera and a mobile camera has one

[@Replacedhumans]

I was able to calculate depth, but how can we calculate the width and height of an object ? I wish to calculate the volume. Thanks.

[@pratikk4348]

how can you able to calculate depth . tell me , btw i am already done by calculate the width and height of an object ,now i only want to calculate distance between object and camera .

[@pratikk4348]

how can you able to calculate depth . tell me , btw i am already done by calculate the width and height of an object ,now i only want to calculate distance between object and camera .

[@syedsultan562]

i didnot find the code plz share this code

[@NicolaiAI]

github.com/niconielsen32/ComputerVision/tree/master/StereoVisionDepthEstimation

[@nivis.v1874]

ThankYou :)

[@NicolaiAI]

You're welcome!

[@donibrima3276]

how about relative speed?

[@jaiswal_stock]

can i get github link for it

[@zero2mastery]

INFO: Created TensorFlow Lite XNNPACK delegate for CPU. i get this and the program ends

[@chadkap.7697]

^ I am having the same issue

[@birendrakumar-gl5gq]

Why do I feel like he's rapping 😅

[@NicolaiAI]

Because I was a rapper before