AI on the Jetson Nano LESSON 46: Synchronizing Multiple Cameras in OpenCV

[@quaternion-pi]

For even more fun: If you have ip cameras running a rtsp stream in your home network (most ip security cams do), you can use the ip cam with your nano and cv2 to do these AI projects in addition to the USB and pi cams. It's super-simple : Let src= 'rtsp://ip_cam_user_name:ip_cam_password@ip_cam_ip_address' , then cam = cv2.VideoCapture(src) Not required to use threading and classes - but my ip cams run at 128 updates/sec on nano with threading. Resize the frame size as we learned. Works faster if you connect nano via ethernet to your router instead of slower WIFI to give nano access to ip camera stream.

[@TheRealFrankWizza]

Very cool

[@opalprestonshirley1700]

Every lesson gets more fun and learning a bit more, I do love learning. Have a great weekend.

[@benjaminlim1735]

for Line 24 of your code, instead of "cam1=vstream(1,dispH,dispW), you could use 'cam1=vstream('/dev/video1' ,dispH, dispW)'. You would notice that the video load a lot quicker and video lag is almost non-existent....

[@sanfinity_]

sir, you are really great at synchronizing our understandings with yours.🤟

[@paulmcwhorter]

Thanks a ton

[@quaternion-pi]

Brilliant! I'm using the nano, but I used the optimized camset for each camera as you described in your Xavier series. Lots of useful info for optimizing the nano on your Xavier series. I calculated fps = (Num_times_thru_loop)/(total_elapsed_time) which is ~64/sec on my nano. JTOP shows lots of GPU activity and all 4 CPU's almost max'd out. Thanks for a great series!

[@paulmcwhorter]

I am struggling on how to split things between the Xavier series and Nano series. Dont want to assume people are experts on the Xavier Series, but then again dont want it to just be a redo of the nano series. I do think the nano guys should tune into those lessons, as in this exampler, that camset string and how to use gstreamer are REAL important

[@somebody9033]

@@paulmcwhorter Please just assume that the xavier people aren't experts, but they are people who have watched the whole nano series and understand everything perfectly.

[@keithemerson9349]

Great lesson. Still struggling with classes but working on it. Different webcams give very different fps values. Also the assignment of the /dev/video( number) seems to change from time to time (using the same usb port each time). Sometimes need to use 0, 1 or 2 or various combinations.

[@paulmcwhorter]

not sure when things changed from '1' to '/dev/video1' but is important to do it the new way. Also, noticed the number changes. I think it works something like this. Plug in one web cam, it is '1' plug in 2nd one, it is '2'. unplug '1' and 2 is still '2' reboot with just that one in and it will become '1'. It is something like that but not for sure.

[@paulmeistrell1726]

Thanks for another great lesson it is a fun one. I took the swapfile out of the 2g and that helped. The reboot to clear problems is a bit tedious. Times on the 2g are very close to yours now.

[@gumball6804]

Best man alive on the planet. I hope you outlive us all!

[@wallacearaujo9469]

Hello! I was having problems with synchronization, but with these changes, I solved all of them. This software is parameterized to run with 3 cameras in perfect synchronization. from threading import Thread import cv2 import time import numpy as np class vStream: def __init__(self,scr,width,heigth): self.widht=width self.height=heigth self.capture=cv2.VideoCapture(scr,cv2.CAP_GSTREAMER) self.thread = Thread(target = self.update,args=()) self.thread.daemon=True self.thread.start() def update (self): while True: _,self.frame=self.capture.read() self.frame2=cv2.resize(self.frame,(self.widht,self.height)) def getFrame(self): return self.frame2 flip =4 dispW =320 dispH = 240 webcan1='v4l2src device=/dev/video1 ! video/x-raw,format=YUY2,width=640,height=480,framerate=30/1 ! videoconvert ! video/x-raw, format=BGR ! appsink drop=1 ' webcan2='v4l2src device=/dev/video2 ! video/x-raw,format=YUY2,width=640,height=480,framerate=30/1 ! videoconvert ! video/x-raw, format=BGR ! appsink drop=1 ' camset = 'nvarguscamerasrc ! video/x-raw(memory:NVMM), width=1280, height=720, format=NV12, framerate=21/1 ! nvvidconv flip-method='+str(flip)+' ! video/x-raw, width='+str(dispW)+', height='+str(dispH)+', format=BGRx ! videoconvert ! video/x-raw, format=BGR ! appsink' cam1 = vStream(camset,dispW,dispH) cam2 = vStream(webcan1,dispW,dispH) cam3 = vStream(webcan2,dispW,dispH) font =cv2.FONT_HERSHEY_SIMPLEX startTime=time.time() dtav=0 while True: try: myFrame1 = cam1.getFrame() myFrame2 = cam2.getFrame() myFrame3 = cam3.getFrame() myFrame4=np.hstack((myFrame1,myFrame2,myFrame3)) dt=time.time()-startTime startTime=time.time() dtav= 0.9*dtav+.1*dt fps=1/dtav cv2.rectangle(myFrame4,(0,0),(140,40),(0,0,255),-1) cv2.putText(myFrame4,str(round(fps,1))+' fps',(0,25),font,.75,(0,255,255),2) cv2.imshow("webCam", myFrame4) cv2.moveWindow("webCam",0,0) #print(fps) #cv2.imshow("nanoCam",myFrame2) except: pass #break #print("frame not avaible") if cv2.waitKey(1)==ord('q'): cam1.capture.release() cam2.capture.release() cv2.destroyAllWindows() exit(1) break

[@pralaymajumdar7822]

Very fun and cool stuff.....what a knowledge you have though you are an Electrical Engineer. Thank you.

[@royyanghozali6638]

Greeting from indonesia Mr paul, it would be great if you make tutorial about supermesh, norton and thevenin circuit analysis, id love to see that happen!

[@Mircea007]

Wow, what an improvement in out code, so clear now. Just wondering if we can add the fps to the image using a class as well? or adding the pi camSet to the vStream __init__?

[@ramseyerbeat9133]

Dear Paul Your tutorials are really great, I don't know anyone who can teach as entertaining as you can, congratulations from a 75 year old beginner! I have a problem: can I record the screens of the dual cameras overlapping by 50%? what should the line with the overlap be, now it's called while cv2.getWindowProperty ("CSI Cameras", 0)> = 0: Thank you very much for your help and kind regards Beat Ramseyer

[@OZtwo]

again here on this system if I use the direct link to the webcam (/dev/video1) I do not need to resize since both are the same and I was finally able to get MY app working with both in the Combo box each having their very own FPS. My first try at this was to try and get the class itself running as a thread not calling the thread within the class as you shown here. The only issue is picam doesn't shut down right and 4 out of 5 runs I need to reset the Jetson which I'm guessing you will address in future videos.

[@epixexplorations]

great lesson thanks Paul

[@itsmeintorrespain2714]

Homework is sort of done. 2 cameras are threaded and lag is next to nothing but after a while the videos freeze. And when I press "q" the program does not shut down cleanly. So, looking forward to seeing what I've not done correctly or just plain missed!

[@Qbanolokot]

I managed to run/exit it without crashing... In an ugly way using global variables. github.com/amartine/jetson-nano-learning

[@prakharpatidar6968]

does self.frame can be accessed in main program (from where the thread is started). since it's running in another thread how our main program is able to read variables from other thread.

[@OZtwo]

another item I have noticed is now that we are NOT waiting for the frame due to it being in a thread, our frames per second has jumped up. I'm at 180-200FPS with my webcam at 30.x and pi at 21.x fps.

[@vaughntaylor2855]

Great video Paul, but quick question it you have not answered before? When read a frame from the camera, there have been instances when you have used '_,frame' and other instances when you have used 'ret,frame'. i understand that 'ret' is return but am not clear on your usage selections? Thank You!

[@paulmcwhorter]

Does not matter, it is a throw away variable. The function returns two values, and the second one is the one you want, which is frame.

[@vaughntaylor2855]

@@paulmcwhorter Thank you, i appreciate your response and your time!

[@barsgezer8030]

How can i make a birds eye view look from 2 cameras with Jetson Nano, since I haven't found any device that can work flawlessly with Jetson Nano and run in real time (e.g. camera multiplexer from Arducam) ? Looking for help fellow learners and teachers.

[@shivaprasadgh4054]

Can we connect Leap sensor similar way ? If so Can you please tell me how one should do it ?

[@xacompany]

Very useful lesson! thanks again!

[@AmandeepKaurDhillon]

Hii Paul this is an amazing tutorial. It helped me alot. I just want to confirm one thing. My application can have multiple cameras (may be thousands) I want to monitor all those cameras concurrently and I want to detect objects and save video frames. Is this will be optimal solution for doing so?

[@theteenengineer7589]

Hello Mr.Paul, what version is the jetson nano?

[@IN3DMEX2.0]

And how can i do if i goint to use 2 Rasperry Camera??

[@SteJuMusic]

I tryed to use that example on the xavier nx, but here i get big problems with the speed. The frames counter sometimes drops down to less than 10 fps. I gues that something with the threads isn't working. I tryed to use '/dev/video1' doesnt change anything? Does anybody else have the same problem?

[@justdance9893]

Sir, i really admire your work, its really useful. (Oh, i start to drink strong black Vietnamese coffee on ice no sugar, like you, its taste good).

[@paulmcwhorter]

Vietnamese coffee if very good. Sometimes I order raw vietnamese beans and roast myself.

[@hixidom2274]

Can this multi-threading technique help me to connect 10 USB cameras to my laptop? I'm not sure. I'm only trying to capture still frames from each camera so bandwidth really shouldn't be an issue, but it turns out that the USB controller allocates the max possible amount of memory for each camera running at 30fps even though I'm effectively running them at 0fps. I've got a lot of ideas for how to get around this but am not really sure how viable they are. 1. Limit the bandwidth of each camera using something like V4L. Seems like my cheaper camera boards don't allow this. Actually it allows me to set the frame rate to 0fps but I still can't connect more than 2 at a time. 2. Write my own USB camera driver or firmware, or find source for one online and modify it. 3. Buy a PCIe expansion enclosure for additional USB controllers. 4. Buy PCIe-to-SATA boards for additional USB controllers and find a way to multiplex SATA to my laptop. might have to buy a desktop computer. 5. Buy expensive scientific cameras that allow bandwidth to be limited through API. 6. Buy expensive fireware/ethernet cameras. 7. USB to wifi adapter for each camera and connect via wifi I'm trying to make lenticular portraits with a linear camera array. I can do it currently but I basically have to connect each camera one at a time and it takes too long.

[@wayneswan3092]

Running 2 Logitech cams, same as yours, and an Intel realsense D455. With the old method, I was running the realsense at 60fps and the Logitech at 30fps, I could barely tell a difference in lag. Now with this method, both cameras are lagging, the Logitech considerably more so than the realsense. Running both Logitech cams, both lag real bad, one barely behind the other. Can't find any errors in the code between yours and mine. 🤷‍♂️ I'll let you know if I figure it out.

[@wayneswan3092]

So, I brought the program all the way back to the very basics, where we just got the camera turned on. and still it was lagging. I simply Cannot find where its causing lag as is. I did find a cure though. By using this "CamSet='v4l2src device=/dev/video3 ! video/x-raw,width=800,height=600,framerate=24/1 ! videoconvert ! appsink', I learned from your Xaviar video series lesson #2 or #3, if memory serves. However, the fps we put in the box on the window is still showing a very slow rate, like 0.7 or so. But the cameras are doing what they are supposed to be doing, so I can move on.

[@wayneswan3092]

I figured it out.

[@OZtwo]

close I was able to create a fully functional class but had an issue trying to figure out how to make that a thread object. (edit: found error, if app will not work and all seems ok, reboot..PI Camera my at times crash. :) )

[@jevylux]

I have a problem with the logitec camera, I have only 2 fps, anyone else having the same problem ?

[@paulmcwhorter]

use '/dev/video1' to launch it instead of just '1' try that

[@jevylux]

@@paulmcwhorter cool, much better, thanks a lot