Really Useful Robot #3 - Head & Intel Realsense Camera

It's time to make the head and plan the robot arm mounting for the Really Useful Robot. The upper section of the robot can travel up and down the utility stick mounted on the robot's base. The head contains an Intel Realsense D435i depth camera that we can use to identify objects using NVIDIA's deep learning machine vision models, and then get the depth from the camera. This allows us to calculate kinematics for the robot arm so we can grasp the object.
The robot can already map and navigate it's environment using an RPLIDAR A2 and the ROS navigation stack, which is detailed in the previous videos, as well as the mechanical build.
Open source CAD and code: github.com/XRo...
You can support me on Patreon or buy my Merchandise:
***************************
Patreon: / xrobots
Merchandise: teespring.com/...
***************************
Affiliate links - I will get some money of you use them to sign up or buy something:
***************************
Matterhackers 3D printing supplies: www.matterhacke...?aff=7500
Music for your KZbin videos: share.epidemics...
***************************
Other socials:
***************************
Instagram: / xrobotsuk
Facebook: / xrobotsuk
Twitter: / xrobotsuk
***************************
CAD and Code for my projects: github.com/XRo...
Huge thanks to my Patrons, without whom my standard of living would drastically decline. Like, inside out-Farm Foods bag decline. Plus a very special shoutout to Lulzbot, Inc who keep me in LulzBot 3D printers and support me via Patreon.
HARDWARE/SOFTWARE
Below you can also find a lot of the typical tools, equipment and supplies used in my projects:
Filament from: www.3dfuel.com/
Lulzbot 3D Printers: bit.ly/2Sj6nil
Lincoln Electric Welder: bit.ly/2Rqhqos
CNC Router: bit.ly/2QdsNjt
Ryobi Tools: bit.ly/2RhArcD
Axminster Micro Lathe: bit.ly/2Sj6eeN
3D Printer Filament: bit.ly/2PdcdUu
Soldering Iron: bit.ly/2DrNWDR
Vectric CNC Software: bit.ly/2zxpZqv
Why not join my community, who are mostly made up of actual geniuses. There’s a Facebook group and everything: / 287089964833488
XROBOTS
Former toy designer, current KZbin maker and general robotics, electrical and mechanical engineer, I’m a fan of doing it yourself and innovation by trial and error. My channel is where I share some of my useful and not-so-useful inventions, designs and maker advice. Iron Man is my go-to cosplay, and 3D printing can solve most issues - broken bolts, missing parts, world hunger, you name it.
XRobots is the community around my content where you can get in touch, share tips and advice, and more build FAQs, schematics and designs are also available.

[@KylePapili]

This is absolutely the most amazing series I’m following on KZbin right now. Your projects never fail to amaze, keep it up!

[@pcgee12]

Man the quality of your stuff is just insane these days. The fact that you do so much work every week and distribute so much of your stuff for free (in video form and the open source stuff) is insane. I remember discovering you building the Hulkbuster suit years ago and have been watching ever since. Heck I'd even say you were one of my inspirations for going to school for Mechatronics Engineering. Thanks for the awesome work James.

[@jamesbruton]

thanks!

[@jstro-hobbytech]

@@jamesbruton the code is free but the hardware costing the price of a small tesla isn't hey. Lots of kids wonder why they can't do this at the robotics class in their local makerspace. A jetson Xavier NX is as fleeting to them as the existence of Santa Clause.

[@kwaaaa]

13:35 "I'm still a human" , quite presumptuous, James.

[@ThingsMadeOfOtherThings]

"I'm obviously a human. No problems there." Exactly what our robotic overlords will tell us as they prepare to enslave us.

[@GeorgiDimitrovX]

This was inspiring. Please continue working on such projects

[@twokool4skool129]

Very impressive. This is sleeker than some commercial home robot platforms. Well done.

[@TheGmodUser]

You can add the cameras point cloud to the costmap even with a 2d costmap, we use if for navigating around tables, even though we only have a 2D map. Also your basically recreating our semester project, which is kinda awesome :D

[@jamesbruton]

Thanks for that

[@davidbarrett7424]

I have a degree in electronics and work in IT and am humbled amazed by James' achievements! great work!!

[@ShravanSuryanarayana]

Nothing is difficult for you Sir. You seem to be able to churn out projects like a machine... Respect!!!. My projects take forever to finish.

[@alvaro4186]

I like getting home at the end of a Tuesday and watching your videos. Been doing it for year. Keep it up.

[@zonochromatica]

keep it up james, always looking forward to a new video

[@Innomen]

Please continue. This is what I've been waiting for.

[@Forcemaster2000]

No way I can do this sort of project myself, but I've subscribed to keep tabs on how you're doing with it! Amazing work, and I think it's great that you make everything you're doing free for others to follow along! This is how tech progress will become everyday technology!

[@jamesbruton]

thanks!

[@LoyvanBeek]

Super awesome, super jealous of having the time to do all of this. You can participate in RoboCup@Home with this once the arm is set up 👍

[@antonwinter630]

you are going from strength to strength. looking forward to seeing what you can get the d435i to do !

[@BuddyCrotty]

Love seeing the Raspberry Pi 400 in the background!

[@jamesbruton]

I think that's probably just the official Rasp Pi keyboard + mouse set

[@jonjon3829]

Mate, it would be awesome if you could setup a 3d scanner with the Intel Realsense, it would help everyone out there.

[@808GT]

A solid, exact hand scanner for cad. At a price that doesn't require a kidney

[@ks-yl9ed]

Isn't there a phone application that does this as well?

[@jonjon3829]

@@ks-yl9ed if you need to buy a $1000 phone to use it it defeats the whole purpose doesn't it?

[@psiewert83]

Phenomenal work James!

[@411Light]

Your work and results are just phenomenal !

[@Mr.Donahue]

These projects are progressing similar to child development. The dogs are learning to walk. Now you are training it to recognize objects. Soon it will respond to your commands and complete tasks. Eventually you'll have it doing chores around your house!

[@henrituhola]

The models seem really well thought out.

[@roostertechchan]

With such progress, at the end of the series the robot will be self aware :)

[@tom.oconnell]

Really curious to see what the arms will look like. I worry a bit that they might make it too top heavy.

[@UNSCPILOT]

He might use lessons learned from Open Dog V2 and keep the big motors close to center of mass

[@Redcactus5]

yeah i was worried that he might distort the belt or have issues with belt stretching and/or slip down the line. especial when it picks something up.

[@Forcemaster2000]

Impressive! I can't imagine 3D printing at that scale...I'm too used to my little 120mm cubed 3D printing space!

[@jedi_mario]

I am really enjoying this series!

[@jcktrck]

I am very excited about this project

[@deadrabbit9503]

Your videos are always amazing. Can’t wait to see the finished product. Can you explain the reason for moving the head with the arm carriage instead of it being static ?

[@jamesbruton]

So it can see things lower down

[@Max-kc2rc]

Very smart thought out masterpiece!!! Having the arm and ability to change heights of the head should make it capable of opening doors or using an elevator- that's MIT robotics level !!!!

[@adosaro]

"I'm obviously a human. No problems there." That sounds rehearsed

[@amelliamendel2227]

It's been a month getting anxious for the next one

[@Sturb100]

So much going in to this, that I might have to watch it again just to take it in 😂

[@turgaysenlet]

Great video James! Great progress, thank you for sharing! You make us feel unproductive when you achieve this much great work in a short time. I would like to add a few points that might not be touched in the video for who would like to integrate Intel RealSense cameras to their projects. 1. For D435, although a USB 2.0 connection would work, it would not give full set of features like depth and higher resolution, so USB 3.0 is almost mandatory. Otherwise it would be a very expensive webcam :) 2. Finding a single board computer with USB 3.0 is a consideration to be done early on in the project. Nvidia Jetson series (in the video) and LattePanda Alpha/Delta are good candidates. Raspberry PI3+ and below are not usable. I have tested with PI3 and can report that it is unusable. Raspberry PI 4 might work, but still could be underpowered for a large scale robot project. 3. Intel RealSense cameras are power-hungry and sensitive about USB supplied power. They do not have external power inputs, so usually they require a powered USB hub. A small factor USB 3.0 hub with good amount of ports is almost a must for a substantial robot project. Plugable and Anker have good and compact ones. Some hubs are more finicky than the others in Linux. Plugable needed few power cycles every now and then, Anker worked perfectly. 4. Besides the D4XX series (D435 and D415), Intel also has a T265 camera that can do visual odometry, tracking and localization. It is an amazing camera that compliments the D4XX series and makes the wheel odometry unnecessary and let's you build a complete vision based solution. This is particularly helpful in scenarios where wheel odometry is not feasible or not accurate (outdoor off-road, flying, human/animal mounted systems etc.). 5. The T265 camera is a pair of stereo fisheye cameras with onboard SLAM processing. Besides the very accurate localization output of the camera, the raw camera images can be used for other vision processing purposes, including a crude fisheye stereo depth estimation, but don't expect a detailed depth view like D435 provides. 6. Both D4XX series and T265 are mainly using near-infrared cameras. DXX series have an active near-IR illuminator (laser dot pattern that is not visible to the eye) just like Kinect, but it is also a passive stereo camera on the near-IR band (unlike Kinect). So it works both indoors in dark or light places with the help of illuminator and outdoors more relying on passive stereo. T265 does not have illuminator, but is pretty robust to lighting and I can report that it works accurately both indoors and outdoors, but most probably won't work in pitch black like D4XX series would. Please feel free to reply to this comment if you have any questions or issues about Intel RealSense integrations. Keep tinkering!

[@stardobas]

Amaizing work!!! Please add navigazion using Slam on depht camera!!

[@r.iyushofficial5318]

Amazing project 👏

[@DadeWheless]

So excited for this series! I have the real sense camera and a nano and have been tinkering with it for the past few months. I also have a platform for it made from an electric wheelchair base. My hope all along has been to build my own version of this robot. Now I can follow along with your project for tips and ideas! I'd like to use the mapping from the camera though since I don't have the laser scanner. Do you have any interest in showing that feature in a future video?

[@SergioGugliandolo]

Wow very interesting project!!! I think that you could add a rail protector (as cnc use) to make it looks more professional, without the sight of the belt, and to make it safer (it should be used in the future with kids in house). I'm talking about bellows. You can buy cheap but still professional ones from aliexpress (intended for cnc as I said) and make your very useful robot even cooler!!! Thanks for your idea-sharing of every video, hope the best for this project!!!

[@micultimy91]

For that LCD with SPI touchscreen you can try an Arduino Pro Micro with an SPI to HID mouse sketch. You can figure out how to do that because you're a smart guy! Good luck!

[@linsthankachan5506]

waiting for this episode...if you have time can you make a tutorial on ros for navigation.

[@patricklepoutre]

awesome stuff. Design is quite logic and simple, that's a sign a genius ! however, a bit top heavy and robot will have to manage the weight of object carried at the bottom of his arm... maybe he would need a kind of mobile weight at his back to avoid falling ?

[@thewatersavior]

Slight blind spot under the screen - might wonder where its willy is.. maybe a smaller PTZ wide angle camera under the screen for allowing dual vision?

[@70smaker]

Masterpieces !!!!!

[@jesterram7146]

would adding a second camera give the robot better depth persepstion like in animals?

[@fourspiked4667]

Hi can this robot use as temperature sensing and kick in data to storage maybe it useful at supermarket

[@nareshsharma206]

Very good

[@toniaguilar6992]

Very good job. Is amazing.

[@sharedinventions]

@16:42 This is the first time I heard someone used words pink-ball for a bear. :)

[@sleeplessengineer1450]

7:29 I made a UMPC using that exact same touchscreen!

[@niloinreverse]

Maybe you can get it to follow you around while carrying your tools using the depth sensor human identification feature~

[@hsmptg]

Hi Can you please share the link for the joysticks of your Smart Robot Remote? Thanks and congratulations for this project.

[@rdyer8764]

I don't like smooth idler gears on toothed belts. Getting the belt tight enough to eliminate slop causes the teeth on the idler gear to distort which can cause the very slop you're trying to eliminate. When you make the belt tighter to eliminate the slop, it doesn't help. It's caused me some problems in the past. It's a small thing, but I suggest in general that you use a toothed idler gears.

[@birdost8448]

Amazing!

[@MuhammadDaudkhanTV100]

Woo superbly

[@wombleofwimbledon5442]

Rossum's Universal Robot. RUR. I see what you did there...

[@mandar3567]

Really Cool!!!

[@MuhammadDaudkhanTV100]

Great robot

[@ovidiurosu6632]

Great job as always. One day I'll get back to building a new robot that can use ROS navigation, yours seem to work just fine for that purpose. I'm a bit intrigued about the current base version. You said you'll consider 2 caster wheels to make it more like a square (I'm guessing it will be closer to turtlebot waffle version). Could you please elaborate a little more on that?

[@jamesbruton]

It was just a thought really to make it more stable, but I won't be making any major changes for now. I guess the wheel layout would look more like an electric wheelchair.

[@FaithfulMC]

Hey robot, pass the butter

[@canofpulp]

How can i test the imu in my 435i? In realsense-viewer, the view frame does not move.

[@canofpulp]

nvm I thought it was like the t265.

[@selfawaredevices]

hi! is it possible to use RGB camera for the object detection and stereo for depth estimation on ROS or RealSense software at the same time? I'm gonna get one of these cameras but they cost a lot and haven't seen that info anywhere.

[@jameszhangwen7522]

What is the estimation of cost for all hardware?

[@georgeaura]

Amazing

[@Redcactus5]

he should probably add ultrasonic drop sensors, an ultrasonic low ledge sensor(ultrasonic sensor on the base looking forward), an ultrasonic backup sensor on the base, and a front facing floor camera on the base. he should also replace the y axis belt with a lead screw to prevent problems down the line like stretching and slipping. though for the arms it might be best and go the asimo route and use hydraulics in the base or head to control the griper and/or the arms.

[@mekabotteam6279]

Can you sell one to us. We are in India. We need some functions custom made.

[@dempa3]

I wish the iPhone LiDar was available as a module for DIY projects.

[@eamonglavin2532]

How does the Xavier handle running everything? Are you running it near 100% capacity or do you think you could use a Xavier NX instead?

[@judgeomega]

why not mount the camera on the very top? seems like having it move will only complicate things, especially if you want to do any machine learning using a video feed.

[@jamesbruton]

So that it can move down to the ground

[@phoenixaidensystems]

Exciting stuff, was wondering what 3d printers you are using. Mine print rubbish..

[@jamesbruton]

All Lulzbot printers, they support my channel.

[@phoenixaidensystems]

@@jamesbruton Let me check them out..thanx

[@jurjenbos228]

I see a physical version of SHRLDU coming up

[@wilgarcia1]

Damn that's awesome =D How do you feel about binary vision in robotics? there has to be a reason its not more widely used, but it may be a little better at getting a 3D perspective on an object.

[@ryanp0342]

It would be cool to program it so it would keep track of objects such as keys, phones, and wallets. Then when you can't find them just ask your robot

[@Mikesukes]

Did you get another Lutzbot printer? The Workhorse? Will your code work on the Jetson Nano Developer?

[@JohnDlugosz]

"get me the pink ball..." No, it should be "pass the butter".

[@marlonlacert8133]

But Sir the words, "Pass the Butter!", causes robots to have an existential crisis! lol...

[@ahbushnell1]

you should tell us about the new TAZ

[@ahbushnell1]

Do you have PEI on the bed?

[@jamesbruton]

yes

[@gooseguy967]

something he really needs to do is make it speak coz it has eyes, and later on, give it emotions

[@maithreyan.s4687]

I have one doubt. by creating robot what could you gain and after creating new robot what will you do with that.

[@captainTubes]

He's gaining invaluable systems integration experience in a public setting. He will apply the sophisticated sensor fusion, environment mapping, and navigation resolution solutions that he implements into his open source OpenDog robots.

[@noway5930]

James makes entertaining and educational content, the robots are cool byproducts!

[@TheBrickLot]

This seems like it'd be really useful in the medical feild

[@roseroserose588]

If it's getting a bit wobbly, you could maybe just add some weights into the bottom? Casting lead would be fun but perhaps a bit much, big bag of rice would do the same! I think as well if the base were slightly longer since it's wobbling back & forth, side-to-side seems less of an issue.

[@roseroserose588]

another thought - perhaps some vibration damping on the post itself? if you still have some flexible filament you could maybe make some bushings that isolate the stalk from the vibrations in the base

[@ernestassimutis6239]

Nice

[@the_multus]

you *are* amazing

[@DIYRobotGirl]

Its like robocup home. I hope we go for two arms

[@whosle]

Some guylines might help with the bouncing head. Like a mast on a ship.

[@BenDover-vh9rs]

YES!!!

[@KangJangkrik]

Would be better if it has edge detection and has ability to vectorize image

[@nareshsharma206]

Sir can you make iron man like personal ai assissent

[@memedude8303]

my dude really making droids a thing

[@KSAW00]

In todays episode of "how many times can James say ROS in the first minute"....

[@oliverdowning1543]

The music in this video really reminds me of Apex Legends

[@Nico-9138]

If this guy is not humanity's real Iron Man I don't know who is

[@kingmasterlord]

6:16 if you want to fix that hole in the head problem you could use one of these kzbin.info/www/bejne/oqCmc6V6acyFn5Y

[@kingmasterlord]

It's a 3D printed design and you got room in your robot's head to fit the coils

[@jstro-hobbytech]

I feel as if "open source" allows you to sleep at night....good Ole git hub

[@dr.benjaminbird7631]

Have a look at RTABmap_ros for 3D mapping. Happy to assist sorting out the config, i've used it extensively.

[@lucbettaieb]

Looks like a Fetch 👀

[@boom_headshot56]

I chalange you to make a fighting robot

[@Jkauppa]

likeness of a llama, alpaca

[@Jkauppa]

or a camel

[@Jkauppa]

partial item guess recognition

[@Jkauppa]

subset detection

[@Jkauppa]

that mapping system needs to improve in resolution, when you move the head and the bot around

[@Jkauppa]

now its only generates a snapshot from one point, not an integrated construction

[@rosstheprivate]

So the name of the robot is R.U.R. Maybe you can put that on the side of it one day haha

[@haidarsyriaismyheart5945]

🌹❤️👍

[@pearcomputers]

Just make the head a Dome James, you know you want to :P . #R2D2

[@sashakoshka]

this dude is building the allay from minecraft

[@MA-zs4bw]

N..nasa?

[@thewatersavior]

#TouchScreenROSWorkstation