Robot AI Demo - NVidia Deep Learning, ROS Navigation, Raspberry Pi

[@jamesbruton]

Patrons and Channel Members already have next week's video - it's about openDog V2 walking !

[@TheJeScast]

nvidea deep "learing". better luck next time James ;)

[@twobob]

"NVidia Deep Learing," “Many a true word hath been spoken in jest.”

[@afs7863]

James Bruton took ya long enough

[@twobob]

@@afs7863 30 some years, whut you wan? :P

[@pearcomputers]

@@TheJeScast so soon 1t will be us. ;) lears :P

[@christophermacier]

I wish you would publish a book for getting started in robotics. You truly are amazing.

[@junepark1003]

Agreed

[@yahyakhalid6599]

Make it happen! Maybe do a kickstarter. That would be awesome!

[@benjaminfindon5028]

agreed

[@cokeforever]

wtf are you talking about, this guy is at most mediocre: autonomous vacuum cleaners been doing this for over 10 yrs now, most of them run linux, so what is the point of "reinventing the wheel"?! also, the only true power of distributed computing is that many hosts send sample data to centralized server that replies with item identification and keeps teaching the gann at same time, why have a local server with partial data?! also, why would want a book or a tv show - both are things of the past...

[@_The_Phantom]

​@@cokeforever know anyone that explain in a way beginners could easily understand.

[@mipe3844]

That deep learning image recognition is impressive. The fact that the color indicator / overlay for the recognition of humans is RED triggers a slight feeling of unease, though - "Terminator" view activated ... xD

[@sethrd999]

You can get similar results from alot cheaper hardware and is more open source with better open source support like Yolo,TF,Keras among others.

[@Sifu-Myers]

The video says it was posted 2 days ago and yet your comment shows a week ago... what the hell did you do lmao

[@donnawander7710]

@@Sifu-Myers They're probably patrons, so they got early access.

[@Sifu-Myers]

@@donnawander7710 good point that makes more sense haha

[@wakjagner]

So glad to see the next logical step for your awesome projects! I could totally see the Performance Bots using the Nano's recognition of body poses to mimic people and dance with them.

[@DocMicha]

You are now stepping into a higher class of robotics. Very nice. I hope that you make a lot of tutorial on ROS and robots using it. The tutorial out on ROS are at the moment, if free, very scientific and hard to understand. So we all need you as knowledge transfer person. Thank you!

[@jamesbruton]

that is the plan!

[@finallyfunctional]

Great stuff! I'd love to try to use ROS to make a robot vacuum, lawnmower, and snowblower!

[@beni446]

Please make this a project James, like a thunderbirds 4 with swapable parts!

[@heimeier1970]

lawnmower robot would be so cool!

[@sethrd999]

You can without hardware as it comes with a decent simulator as you can run the roscore ( broker ) and the ros nodes on the same machine. Turtle bot has always been stock and you can customize it and control it or make it autonomous if you wish, Adding nodes ( camera's / other sensors ) is easy, ROS is not as hard as you think, you only have to get your head around brokers pub/sub like MQTT. Just one other thing starting out use Python but when working as needed port it over to C++ as that is where all the power is.

[@haidarsyriaismyheart5945]

Vacuum doesn't use laser, it uses IR as I think 😁

[@Skyentific]

Excellent video! Nice Jetson Nano setup! :) Big big thank you for the brief ROS introduction!!!

[@jamesbruton]

Thanks! I'm working on a more detailed ROS series to build a robot from scratch that can navigate.

[@Skyentific]

James Bruton this would be super interesting!

[@CrIMeFiBeR]

Ros 2 is also amazing, it is more focused around swarms and dynamic node allocation, because there is no longer a master required, and it has dds as middleware to improve security

[@ExStaticBass]

This really makes me wish this existed 20 years ago when I built my first Dalek. Speaking of which I reckon you'd do a fair job of making one of those. Mine had an old laptop for brains and loads of micro controllers for sensing and pariferial control. It had an airsoft canon for weapons, an old Kinect for mapping and visual data. For the body a rubbish bin turned upside down with styrofoam balls cut in half works nicely. I'm sure you can work out the rest. Anyway great content mate.

[@ovidiurosu6632]

In order to learn ROS, I've been trying to do exactly what you said: build a robot with ROS on it and try to implement everything from scratch. This was a couple of months back. I've got quite far, but not nearly enough to the smoothness and accuracy of TurtleBot. I'm really looking forward to what you'll come up with :D

[@jamesbruton]

I'm working on my own build from scratch right now, but I think there will be a V2 ;-)

[@PassionIsVita1]

I really look forward to seeing your journey of discovering ROS. The possible applications look really interesting. Great informative video! Can you get the deep learning to identify a specific face so that you could denote a particular person as not a target?

[@jamesbruton]

Yes you could train the model to do anything, but that takes a while and would require a lot of images of the person in different environments.

[@Vidicon31]

Nice to see you get started with ROS, it will bring your projects to the next level. I'm a professional robotics engineer and work daily with ROS so if you need any help, let me know.

[@easyfly99]

It did detect you as a toothbrush at 4:35 :-D

[@twobob]

it's all about certainty and context pretty sure it would have pegged him as human before applying the (Insert brand name of toothpaste here)

[@iluvtocnc]

@@daniel_4096 What about when at 4:50 he's detected as a toilet with 91.3% probability

[@hackhobbylab]

63.3% ToothBrush😂😂😂😂

[@neutralentr0py]

Definitely another promising avenue of robotics to delve into and definitely one I'd be interested in seeing you persue. Man this has to be folded back into some of the past projects for sure. R2 unit with these capabilities, open dog with these capabilities, BB8.

[@michaelgill5203]

Yeah true we need another bb8 robot

[@sujitvasanth2502]

loving this - lots of new ideas and hardware - havent seen you use linux or raspberry pi's much in the past...the ros lidar navigation node looksrealy useful!

[@jamesbruton]

thanks, I've got more coming up!

[@Super73VW]

Its crazy how many times I have been interested in a subject, and next thing I know, you are putting a video out on said subject!!! I am taking a Shark Ion RV750 robot vacuum, and turning it into a Snark ROS vacuum robot!

[@Martindooo]

I had a project during my stay at a University of Applied Sciences toying around with ROS. It was so damn impressive to see how quickly you can setup a working robot doing awesome stuff

[@AndrewDiGenova]

Fantastic video James! And your links for robotics courses and deep learning projects? Awesome! You've created a great resource here. Great way to get started. Inspiration my friend. Thank you!

[@AvivMakesRobots]

One advantage of the Nvidia card is it is not sold out the way Raspberry Pi 4s are these days!

[@natpim4730]

I have a solid year of work using ROS 1 for a research paper in college. I found the possibilities with ROS are truly endless where robotics are concerned... if you can dream it you can program it in ROS that said it also took me a solid 2 months of working on my own to get a good understanding of it before was able to really get going on my project ROS is a wonderful platform if you want to put in the work.

[@jouleio4525]

Can you give me any advice? I am a beginner in ROS and want to apply this to robotic manipulation in manufacturing

[@natpim4730]

@@jouleio4525 It has an extremely steep learning curve and I found that for ros 1 after you get past the basic tutorials support kinda evaporates. I would strongly reccmond you look at the construct they have free weekly webinars and other stuff However some of it is behind a pay wall They are really the only source of Info for the more advanced stuff Given it's open source nature the doccumation while not horrid is not particularly amazing particularly when it comes to the more advanced stuff I also strongly reccmond that you have a high end computer. Particularly if you are going to be using a VM to run linux to talk to the robot I was regularly using more than 20 gigs of ram running the vm and ros. The same goes for the processor. I would reccmond an i7 or equilivent However 16 gig and an i5 would suffice for most The other bit is that huh ou dont need to upload your code to the robot every time you run it Because of the de centralized nature of ros you are able to run the code for the host computer running the server As for starting platforms I would reccmond using the robotis turtlebot It is about 1300 usd and is plenty to learn the platform/explore it before investing in a more expensive platform There are also other platforms that are already made for indstural applications that you might want to look into Most are mobile platforms for moving materials but tbere as re also a coupple of as arms too I believe Baxter is one of them It has 2 arms and a screen with eyeballs Anyways hit me up if you want more info

[@jouleio4525]

@@natpim4730 Thankyou very much for replying hahah 😀. fortunately, I took the course at RobotIgniteAcademy and still learning for ROS with python. I often hear about the turtlebot. Currently I am interested in dobot magician, its a 3 axes desktop robotic arm. It is written that it is also support ROS. You can also check this things out. I am curious about the future career of ROS engineer especially in South East Asian countries since I am from these regions. Also what I noticed is that the high development of ROS still occur in developed countries such as USA, Europe, China.

[@natpim4730]

​@@jouleio4525 the turtle bot has a 6? axis arm that can be mounted ontop of it... as far as the jobs go yes there are ros development jobs for the usage of ROS in indstural environments however I don't know what the field looks on the other side of the pond where you are And why are you using python.... C++ is much much better.... manny more options with what you can do....

[@jouleio4525]

@@natpim4730 The thing is I want to master the concept of ROS as quick as possible, and at first I am not familiar with python and C++.Thus, I choose python to start with ROS since the language is easier than C++. (I know that C++ is much better for robotic software engineer careers )

[@defaltdj1493]

A ROS series?!! This is going to be quite the ride, can't wait to see the next episode

[@GNARGNARHEAD]

looking forward to the incorporation of this new layer, great stuff James, thanks!

[@LuisVasDeCamoes]

I used a Turtlebot 2 on the 1st semester of the bachelor in Robotics... No one explained us much, we worked in groups of 5 and 6, and they just assumed that it was likely that at least one in each group knew how to program, prior to entering the university (they were right for most groups). In my group I was the only one that knew how to program, but I had never heard about ROS, and it was very hard to understand it. But after you understand ROS, everything is very simple to program. Our goal was to develop the program for a "Mars exploration vehicle", we ended up with one that could autonomously explore take and send a picture every minute Analise color to detect possible mud vulcanos (if it contains water, they may contain live) Upon detecting a volcano it would send a picture with the coordinates if it bump into something it would stop and check the surroundings, sending pictures and change to manual driving mode (Manual mode was tricky because of the distance between Mars and Earth)

[@TylahKwin]

Building a slam robot from scratch with ROS is my current project also, excited to see how you go about it. Please go in depth about the code and ROS implementation

[@jamesbruton]

yep - it's coming up in a couple of weeks

[@user-st2ll5bq2p]

日本からのコメントです。 最高にクールだぜ！

[@shahbazfawbush]

Great review. First application is weaponization of the robot!

[@magnussorensen2565]

Hey James. Great video. I love ROS and I have a small tip for you. After a while, you will get many terminals running nodes, can be quite messy and confusing to keep track of every thin. Then if that happens you should try using Tmux. Its a terminal multiplexer allowing you to have multiple terminals in the same terminal. Also if you use Tmux over SSH the terminal int lost if you accidentally are disconnected. Can save a lot of headaches.

[@daanvandevoorde7270]

People who know nothing of AI: AI will take over the world!! AI: 4:45

[@Frrk]

I'll make sure I have toothbrush disguise ready!

[@devrim-oguz]

I'm 68.9% sure that it was an airplane though

[@TheRealFrankWizza]

To be honest, my face recognition/eye detection pan/tilt camera, is pretty impressive.It knows the faces I train it on, and can name/track them no problem. I got it working with lessons from paul mcwhorter here on youtube. I can even draw overlays over the eyes and stuff. I bought a laser to attach to it, but I haven't yet. His tutorials are currently on object detection, which I also have working a little bit, but not as well as the facial recognition. I have had people cover more than half their face and still be identified correctly. Yeah though, it's going to take over the world, and for most big data analysis it already has. Even cities around the world have already been applying facial, gait, color, recognition for years.

[@bardenegri21]

Amazing! Can't wait to watch the series, not a lot of great ROS content out there.

[@jamesbruton]

thanks, I'm working on it!

[@douglasmaclean5836]

Fantastic VIDEO !!! THANKS JAMES !!! ... love the AI bits, looking forward the next one ...

[@stormix_co]

Love to see more ROS stuff. It's pretty cool

[@travisbarton4288]

Sooo cool! First time I'm hearing about these Nvidia modules! I'm still WAY too amateur for that investment but it's definitely something worth looking forward to!

[@lawrence880]

Great video, as usual. Love your work, you always inspire me to get back into my projects. I’m working as well on a robot dog concept, based on MIT mini cheetah actuators. I would recommend trying to migrate to ROS2. ROS1 is still widely used, but it’s aimed really at research and development. So it still fine to use, but ROS2 is built to be more dev to production compatible and offers much more robustness to communications dropouts and natively supports higher reliability with the data distribution service middleware quality of service. Allows you to detect when nodes aren’t running as expected or messages are not arriving when you expect. Something to consider. TBH, ROS1 is still what I use mainly, old habits die hard, but I’m trying to move to only ROS2 to help handle what i spoke to above.

[@avejst]

Great update/project 👍😀 Thanks for sharing 👍😀

[@benGman69]

Thank you for the longer video 🙌

[@crckdns]

that is just perfect for a nerf-gun control :D now need to identify where the different body parts are to aim better ^^ great project, thanks for introducing! edit: oh wow, at 16:00 you're already showing a controlled nerf-gun O_O ok... nice!

[@michaelgill5203]

Really interesting video can’t wait for you to upgrade some of your other projects where you had an idea but the technology

[@jim2386]

4:30 “And hopefully we see that it detects a human.” *holds breath* *detects James as human* Welp one less possible android to worry for the robot uprising! If it detected James as a robot, I’d be very nervous indeed!

[@ColinMacKenzieRobots]

ROS2 ecosystem is growing. You should give it a try as well. I was turned off ROS1 due to protocol overhead but ROS2 is more efficient, realtime capable, and simpler node design. It also has microROS for arduino nodes. Community is not as far along as ROS1 but it's growing fast and we recently got MoveIt!2. I am using it on my humanoid robot.

[@KK-fh1ds]

Youre such an inspiration man

[@mikewashington4188]

This project is mental. Thanks!

[@cedrikkaurit3078]

Best of luck, gonna be a lot of work but you are surely up for the task!

[@mckenziekeith7434]

Pretty cool and also scary. One step closer to slaughter bots!

[@Paul-rs4gd]

I look forward to seeing how you get the Pi Camera + ROS working on Pi 4. I struggled with this for some time. I tried a few months back and found I could get Ubuntu 18.04 onto a Pi 4 the install ROS, but could not get raspicam_node to work. I then tried installing Raspbian Buster on the Pi 4, compiling ROS from source and then hacking Raspicam_node until it installed. That worked, but I was an emotional wreck by the end ! I would love to see an easy solution, installing from binaries. A Pi 4 + ROS + Camera is a great combination for robotics.

[@Frrk]

Fascinating stuff! If it can detect chickens, I could use it for a chicken coop door!

[@DerSolinski]

I look forward for what is to come out of this 👍

[@neoqueto]

okay, this is next-level.

[@nishantsharma4465]

Liked even before watching waiting for the next part

[@cap1sup]

You should definitely check out AlwaysAi. You can use all kinds of deep learning models and even train your own. I’ve been using it for a while and it supports Jetson Nano, GPUs, and RasPis. Very flexible, easy to use

[@jamesbruton]

thanks, I will check it out!

[@514_cricket]

James gettin into AI... looking forwad to see your microcomputer deepl projects

[@dr.benjaminbird7631]

For those not able to purchase the turtlebot 3 - the eManual for the TB3 contains a lot of simulation stuff, you can learn an awful lot from using a simulated robot in gazebo, all the ROS nodes function the same. It's also free. A caveat to that though, is that you won't get the sensor ambiguity that you get in real life, for example the map that the TB3 produced will look a lot better in sim than in real life, but then again it will look a lot better if you use a better LiDAR.

[@richardlincoln886]

Project idea - robot mower hack... DIY opensource approach without the need for a boundary cable?

[@BlueSerpent-7]

OMG I'm in love with that robot

[@9madness9]

Want a AI with security camera to keep shots of 6ft toothbrush in my backyard! off to research!

[@starblastershooter2444]

Amazing I'm going to be making robots using these

[@DamianReloaded]

I hope you got sponsored for a lot of videos on this product! Seriously! ^_^

[@DWiskow]

I would recommend that you put a good (sizeable) heatsink and active cooling with a fan on the Raspberry Pi 4. Without active thermal cooling, the performance of the Raspberry Pi will be significantly compromised (i.e. any significant compute load lasting more than a minute or two will result in the CPU throttling back to 600MHz) . . . with active cooling, you can overclock the RPi4 and consistently run at 2.147GHz ! The Argon One case is a good option in that it provides a very effective passive heatsink (the metal case) with a variable speed fan that kicks in under software control dependent upon CPU temperature. I am also very impressed with the Yaboom RGB Cooling HAT (does not use GPIO, but requires i2c for control), but this needs to also be paired with a large heatsink (e.g. aluminium heatsink case).

[@patrickwinterbourne8516]

Awesome video is always

[@nophead]

I just backed an OAK-D on kickstarter. It is a dual camera Open CV AI kit with stereo depth perception. Might make a good dog brain.

[@Matty.Hill_87]

I swear you are one good day away from becoming iron man.. Or one bad day away from becoming a super villain.. It could go either way 🤷‍♂️🤣

[@scifi_shop]

Now you can make your Ironman suit autonomous and summon it.

[@johntilghman]

What joysticks are you using there, they look pretty nice..

[@somebodyoncetoldme1704]

So a year has passed. Are you generally happy with ROS?

[@gregoryskaltsas1858]

One more great video. What is your background?

[@pranavpatil7312]

Sir , can we use Electronic Control Unit i.e.open ECU for controlling the 4 wheeler robots ?? How to start ?? What should be a roadmap??

[@seasnek7024]

Hey I’m curious did you say you could use just about any camera to identify an object with the first program? Would you able to use it with a computer’s inbuilt camera?

[@haidarsyriaismyheart5945]

That's great, thanks alot 😍 but how cleaning machines work without laser 😕🤔

[@TheRealFrankWizza]

Hey, I was just recommended your channel and subscribed, because what your're doing is relevant to my interests. Looking through your content, it doesn't seem like you go through your code when making these machines. I hope you do. Thanks!

[@jamesbruton]

Sometimes I do, but in the case of Robot dogs I've done it so many times in previous builds I don't cover it again unless it's new.

[@TheRealFrankWizza]

@@jamesbruton Alright, I will do some digging through your videos over the weekend.

[@StevenIngram]

Hmm. Considering the direction you're going, you may want to include a microphone on your new remote. I can imagine the need for issuing remote voice commands emerging in the near future. :)

[@chrispi314]

4:15 not done python from a long time but "for detection in detections" is useless if you use only the first one, an if should be more efficient

[@ethzero]

In all these vision-AI demos I've see, I'd never seen DL change/enhance it's understanding of another object based on the presents of another. I guess this goes on all the time with the context of a "face" being inexorably (at least usually!) attached to head, and by extension a body (again, usually).

[@theexo-engineer9603]

4:45 “Toothbrush”

[@idont3282]

4:50 "Toilet"

[@harsh9558]

@@idont3282 wow

[@HoRiOnS]

Would it be possible to use ROS for a smarter lawnmower that uses a precise mowing path that's the most efficient, this in contrast to how most mowers work now..? Maybe have easy anchor points that can be recognised for orientation?

[@pntra1220]

question, can I plan a path without interacting directly with the laptop or pc, like voice recognition or some other stuff?

[@jamesbruton]

Yes if you code it.

[@diceblue6817]

surfboard 61.8%, toothbrush 72.6%, airplane 68.9%, umbrella 64.3%, cat 61%, skateboard .... yeah

[@ShravanSuryanarayana]

Have you also checked the Maixduino from Sipeed or the Huskylens from DFRobot. They are Eur 20 and Eur 40 each and have lot of neural nw hw in them - its based on a Kendryrte K210.

[@joejia1410]

quality content ya got here

[@PSG231092]

if you're using ROS for messages and topics you could use MQTT instead does exactly that

[@eleanorhathaway927]

This is exactly what i was looking for for my next project. I want to build a robot with a tank base, and an arm on the top with i want it to have image recognition etc and be able to map out the area. can the laser module map in 3d or is it just 2d?

[@jamesbruton]

That one is just 2D, you'd need a depth camera or a 360 3D laser scanner, and a different mapping solution

[@TejasNarkhede-u6e]

How You have done obstacle detection part? I am also working on project

[@teja9676]

Excellent

[@taihatranduc8613]

it's so impressive

[@Sifu-Myers]

I'm trying to build a life sized robot with servo motors any advice for a beginner trying to build a robot?

[@JonathanKayne]

That doesn't look like Ubuntu 18.04 to me. Looks like 16.04 because of the unity launcher... (The one at the beginning on the Jetson nano, that is, the one at the end of the video is 18.04)

[@reggiep75]

5:25 - I'm so looking forward to aural AI (to add to visual AI) that will listen for surfaces that would hear the cup on a table and would say 'Beds are soft, this has to be a table!'.

[@pedroparamodelvalle6751]

Personally y prefer hector-slam over gmapping is it a more lightweight node yet it works as best if not better than gmapping. Finally goggle's cartographer is the top mapping library but it is way to complex for my liking a bit of an overkill really

[@koz]

The Nano has two camera connectors to be able to do VR 180 and 360.

[@matthijsvisser9087]

I would suggest to use ROS2 which has less overhead

[@MaxMustermann-sm4qu]

Funny that it has recognized your arm and hand with 86.8% accuracy as a toothbrush. 😁

[@paugpair]

also as an airplane with 68.9%

[@hellalsoundes4680]

Please answer me , why you pluged in an arduino uno into raspberry pi 4??

[@levon2371]

wow that was so good.

[@LukTzeChing]

Will you try out the Issac SDK from the Nvidia nano? I am thinking which platform should I use for my robot: ROS or Issac if I am using Jetson as my control unit.

[@LoyvanBeek]

You can run ROS on a Jetson board just as well. I've spent some time trying out NVidia Isaac and found it severely lacking documentation and not being developer friendly. The pre-built neural nets are nice, but not a killer feature. There is an Isaac-ROS bridge included with Isaac, but that is implemented poorly, eg. hiding exceptions etc. Also, ROS has a large community to ask questions etc.

[@LukTzeChing]

@@LoyvanBeek So, i would better use ROS to develop my robot instead... I have seen the video from Nvidia developer of using the Issac SDK, seem the functionality is quite similar, well if there is not much document, then ROS seem is a better approach. Thanks

[@LoyvanBeek]

@@LukTzeChing The similar functionality seems only in a first glance. When you start to develop anything more complex than they pre-cooked for you, it's really difficult or impossible.

[@ahmedwaly9073]

That's Amazing

[@Peavey_]

Is the Jetson image recognition have to be connected to the internet or once the code is written can it be mobile and offline?

[@augustopozzebon3188]

Can I use an arduino with NRF24L01 and my computer as base for de ROS navigation system?

[@ritinkornas5336]

What happens to the turtlebot if you put the box at the spot where it was supposed to stop, after the initial calculation of the path? does it stop before or after the box instead ?

[@jamesbruton]

It keeps trying to get to the target going back and forth and trying again. You can de-tune the accuracy for getting to the target though so it thinks it's near enough

[@twobob]

Yeah quality work. failed to get Ros running under windows. I'll retry on Linux then

[@jamesbruton]

I've not tried it, but it's better supported under Linux. You could always use VMware (free) Player to install Ubuntu

[@twobob]

@@jamesbruton it's cool. my machines a dual boot. Honestly just gutting out a few python versions could probably get it but meh, I'll try penguin. This was a genuinely exciting video for me. at last, some basic AI rammed in those mechanoid skulls. We can easily harness an extant OS to some cheap fast processor at some point to remove the PC link. Exciting times sir.

[@twobob]

@@jamesbruton Quick update, was fine on linux, so tried on WSL with a windows X server. seems fine. Hope that helps.

[@twobob]

@@jamesbruton Final update, building from source was easy to get a 32 bit version also. great code. nice project.

[@georgeaura]

Can you do a video on the adeept rasptank and see what can be done with it?

[@ryokohakubi2928]

imagine being able to do this with our Droids from Disneyland's Droid Depot

[@wouterheerwegh]

Nice video, I've been working with ROS for a year to build an autonomous car. What I've learned over the year is that if you're using ROS on only 1 system, it's best to make use of Nodelets instead of nodes, as it sends pointers to messages in memory instead of serializing and deserializing the messages. Also, rosbag is love (for debugging). ROS is switching to ROS2 though, which brings, apparently (haven't tried it yet), a lot of improvements, not sure if turtlebot will be switching to it aswell.

[@jamesbruton]

There are already ROS2 instructions for Turtlebot, but I'm not quite there yet