How to Simulate a Self-Driving Car

We're going to use Udacity's car simulator app as an environment to create our own autonomous agent! We'll use Keras to train a convolutional neural network on images from the car's cameras as well as steering angles from human driving. Using just those 2 data points, it'll be able to drive itself on any road.
Code for this video:
github.com/llS...
More Learning resources:
hackernoon.com...
nicolovaligi.co...
/ teaching-car-how-drive...
selfdrivingcars...
blog.davidsingl...
www.coursera.o...
www.udacity.co...
Join us in the Wizards Slack channel:
wizards.herokua...
And please support me on Patreon:
www.patreon.co...
Follow me:
Twitter: / sirajraval
Facebook: / sirajology Instagram: / sirajraval Instagram: / sirajraval
Signup for my newsletter for exciting updates in the field of AI:
goo.gl/FZzJ5w
Hit the Join button above to sign up to become a member of my channel for access to exclusive content! Join my AI community: chatgptschool.io/ Sign up for my AI Sports betting Bot, WagerGPT! (500 spots available):
www.wagergpt.co

[@ablacklama]

Love the slower pace, really lets me understand what's going on!

[@SirajRaval]

noted

[@gamecraftczjaajenomja1057]

Meantime me at 1.5x :D

[@akshayrana9351]

Finally !! A video from Siraj that actually explains something instead of those useless memes. This is how you should make rest of your videos. This was one of your best videos so far.

[@SirajRaval]

thx love u

[@Stefan-mq7fd]

I love the way you are so passionate about teaching people machine learning. More people need to see these videos. Thanks for doing what you're doing.

[@SirajRaval]

thx!

[@jawedib]

As others said, LOVE the "live" style video. Took you total of 30-40 mins to make without having to edit and add memes. And we take more in without getting distracted by the edits and memes.

[@SirajRaval]

great feedback thx

[@ajinkyagawali6442]

Stopped at 4:30 right away. You have improved a lot. I love this new style where you toned down a bit and discuss and talk normally like a cool teacher. Really looking forward to what you will offer next with your talent and this new style. This channel is now perfect to learn machine learning from. Already recommended you to my entire classroom. Thank you very much, Siraj.

[@red-draken9880]

@Siraj I recently started following your channel, I am a student of Machine Learning and your videos have helped a lot, Thank you. Keep up the good work !

"it's a chain of whatever" - RAVAL, Siraj

[@stevenodonnell497]

love the "live" style video

[@SirajRaval]

thx noted

[@jackie.p6891]

I've been working with ML in Unity for about a month now, and i gotta say, it's stupidly simple. really, all you do is add some components and write how an agent can move, start up mlagents-learn in an anaconda environment with tensorflow, and it'll start training. If you know a little c#, i definitely recommend this.

[@ragnar_real]

huh, I found a hidden gem, you've been working, from your channel, I can see you have worked alot on AI before AI became a common thing to use

[@richardxu8865]

Your AI videos are a gift, Siraj. Thanks so much for making these cool, entertaining tutorials

[@VFXKitchen]

You did a good job but I would vote for @sentdex on his effort of creating self-driving car with GTA V series.

[@aidangomez4726]

Mahmoud Hisham he didn't succeed siraj always succeeds

[@colinkotar]

this comment is the equivalent of Kanye: "Yo, Taylor, I'm really happy for you, I'ma let you finish, but Beyoncé had one of the best videos of all time!" lol

[@artmario]

Mahmoud Hisham watcht him on twitch, Charles is training 24/7

[@rajathshetty325]

chill folks, this isn't a competition! :P

[@SirajRaval]

ok thx

[@vijayabhaskar-j]

This is the best explained video you have ever created. Continue to make videos like this,slow paced but well explained! :)

[@SirajRaval]

will do thx

[@peterssen6563]

My drive.py didn't connect to the simulator, how do i fix this ?

[@anukritisingh7204]

Why did't you run the test code show us the car driving in autonomous mode?

[@vishalmishra5335]

because training the model took him about 8 hours

[@plouismarie]

Ah ah :) Awesome Video ! Some people (..haters..) might be upset at you, since you are sharing and explaining what could make some car company dominating the car autonomous market.

[@someone5781]

Love this style of video! If you could make your videos more like this, that'd be awesome!

[@SirajRaval]

noted thx

[@gabriyalstephen8578]

bro you are an another level.Thanks for sharing this

[@abhiramajith26]

Hello . Can you please help . The python client says connection accepted but in the autonomous mode the car is not running. Please help.

[@pradeepkumar-qo8lu]

awesome video as usual and thanks for making this knowledge available to the masses

[@andymitchell2146]

sentdex did this in GTA last month.

[@artmario]

i love when charles do evasive manuvers.

[@artmario]

charles is training twitch . tv/ sentdex

[@artmario]

after this training , jugdment day by skynet!

[@sentdex]

Weird, I had no idea I failed

[@artmario]

sentdex can you make some transfer learning to this model?

[@MrNaveenmn]

Just to add: The reason why x/127.5-1.0 is there to fan in to 1st layer is to just normalize the pixel inputs between -1 & +1. Gradient descent converges quickly if inputs are scaled and normalized to same range. for RGB color space pixel varies 0-255 & x/12.7.5-1 will normalize them to -1,+1. Interestingly they have preprocessed the image from RGB to YUV color space where pixels value ranges would be further reduced because Y = 0.299R + 0.587G + 0.114B U = 0.492 (B-Y) V = 0.877 (R-Y) which helps to converge better.

[@vighneshsrinivasabalaji802]

I have a few things to clarify. When I run the drive.py file with the model.h5 file, though it says 'Recording this run' and starting wsgi, nothing happens after that. 1. Do we need Unity Engine and load the Udacity Simulator on that? if yes, then does Unity work on Ubuntu 16.04 2. The python files, the .exe of the simulator and also the image folder which contains the data is in the same folder. The driving_log.csv has the absolute paths of the images. Where should each component be present? Is there some directory problem here the reason for my problem? 3. What are the arguments needed to be provided by us while running the drive.py and train.py file? will the drive.py file work even when the model is not saved as a .h5 and saved as .ckpt?

[@edexnorth123]

Siraj! If you could increase the audio quality that would be awesome news for our ears.

[@SirajRaval]

i will from now on promise

[@doublegrey.broken]

thank you for your videos from Latvia)

[@hazlanmshabli9137]

keep it up and thanks for making me understand and interested in ML

[@manelaloui7069]

Thank you Siraj for this amazing video

[@zhengchunliu5839]

Thanks for the awesome tutorial.

[@Skythedragon]

Oh cool! I didn't expect a video today!

[@LinkSF1]

very cool video. Thanks for the upload!

[@PrinceGupta-jo8lo]

Just a question. Isn't end to end deep learning for a self driving car extremely unreliable? Like in realistic world we'll never be able to implement this because real world has so many variables and providng data that covers all of them satisfactorily is insane. Also we still can't be sure NN learnt everything correctly. Wouldn't approach where we do object and lane detection first then generate route be better?

[@junlinlim8524]

sorry if i missed, why 3 video inputs?

[@andrejmitreski3721]

You are actually self-driving car engineer and all that after this course?

[@larryteslaspacexboringlawr739]

thank you for self driving car simulator video, do you have any interest in talking about numenta htm or ibm watson in future video?

[@bharathbabuyp]

Hello Siraj, is there any possibility of you making a tutorial on Semantic segmentation using Tensor flow?

[@nikhilchhabra]

You are doing an awesome work. And you are very creative. :) I have few a questions siraj: 1)-Its seems so simple, Is this being used for real self driving car ? where is the catch ? 2)- If this is it then should all the companies stop working on semantic segmentation or object detection networks ? 3)- Do you think the network will learn which particular part of image lead to the output ?. (example : a stop sign lead to braking)

[@johnhammer8668]

This is so cool. Thanks for the video. Just curious how long did it take you to finish this project.

[@dimitardimitrov4979]

Upgrading the audio of the vidio will be definitely an amazing improvement.

[@wi8shad0w]

i am not sure if u will read this comment .. nevertheless i wanted to say this.. The approach here is only to see a single frame and take decision. but it DO NOT consider any sequential data, like no time series data at all .. This will critically lose a lot of info to understand the traffic and there is no tract of state variables like ego vehicle velocity, target velocity.. This is just single image analysis without knowing the past at all .. is this any good for real world problem. ? I wish you can do a video only on Deep learning architectures for Autonomous driving.. there is a lot more than just seeing a frame and inferring a steering angle.. i guess you will agree..

[@souravmohanrane2891]

good enough for a start i say !

[@mridulpareek4778]

How would we go on about implementing navigation/direction in it, for instance we want the car to drive between 2 points but we need to tell it when to make a turn

[@Snejk47]

[Q] How do you control such a car? I have trained in another game "self-driving car" with much smaller (naive) network and data just for curiosity if I'm able to output any values and it surprised me because it did actually learn to keep his lane (It did far more than I expected). But I have no control over him. I can't tell him to exit highway or go right on the next intersection. I had an idea to add to the images some data which will tell "go right" and train on that or connect another input layer after the CNN into Dense layers with data like "go straight/left/right" etc. but it didn't feel correct. For now I thought end-to-end for self-driving cars was rather toy experiment and not real use case but maybe it's not. Thank you.

[@nirajgaonkar7669]

ValueError: output of generator should be a tuple (x, y, sample_weight) or (x, y). Found: None how to solve it?

[@buggi666]

github.com/llSourcell/How_to_simulate_a_self_driving_car/issues/7 last answer...

[@amishakodape]

How did you install its dependencies please help me out. Thank you

[@Pragalbhkulshrestha]

You're awesome @Siraj.

[@JathavanSriram]

Thanks Siraj!

[@chrisdepalma6844]

Can't get it to run. Getting errorsTraceback (most recent call last): File "model.py", line 163, in main() File "model.py", line 159, in main train_model(model, args, *data) File "model.py", line 120, in train_model verbose=1) File "D:\Docs\Anaconda3\envs\car-behavioral-cloning\lib\site-packages\keras\models.py", line 924, in fit_generator pickle_safe=pickle_safe) File "D:\Docs\Anaconda3\envs\car-behavioral-cloning\lib\site-packages\keras\engine\training.py", line 1481, in fit_generator str(generator_output)) ValueError: output of generator should be a tuple (x, y, sample_weight) or (x, y). Found: None

[@SoundOfTheShutter]

Same problem here!

[@kaustabhsingha3880]

Same problem here, any resolution yet?

[@abdelrahmanyousef2237]

go to utils change the picture size to 160, 320

[@deepharia4209]

I did this whole project nothing ran as I wanted it to I setup the whole environment properly but nothing works help me please

[@iniyavanmanian2043]

when i execute the code, i get the following error.... sio = socketio.Server() AttributeError: 'module' object has no attribute 'Server'

[@theappliedcoder9824]

Bro i want to learn machine learning and self driving car projects

[@jawedib]

Hey Siraj. I am halfway through the Coursera (Andrew Ng) Machine Learning course and want to be able to do the cool stuff that you do. Where do I go next from there? I have a dream to be able to generate images, videos, sounds in the next 12 months. I am willing to dedicate hours per day to achieve this. I do realize you cover literally everything but I am not on that level yet and most things woosh over my head.

[@WreckingU]

You are on the right track I finished that course 2 months ago and it took me 3 months to get through to really understand. I would recommend taking the Udacity course after. Machine Learning is really surprising and I am not on Siraj level but I hope to get better and better the more I do it. Just make some fun projects after going though each courses and challenge yourself.

[@McMurchie]

yea I am working through the intro to machine learning course on Udacity... but I always find myself coming back to Siraj lol

[@cameronbehar7358]

New Name so Employers don't find me I'm neither an expert nor a video creator, but I created a Machine Learning Basics playlist on my account of other videos that helped me understand the high level principles. I think you can find it by looking on my "channel"?

[@alayshah1995]

New Name so Employers don't find me: Udacity Deep learning Foundation is your key to open all the locked doors. I'm currently enrolled in that course and just after few weeks, i already feel myself covering a lot of breadth that i would possibly not able to do somewhere else!

[@jeevankumar9594]

I am on the same track too! I have finished upto week 2.

[@SniperUSMC]

This is an awesome video, thanks! GREAT presentation. Wondering if I can record my own video and input it. For instance, currently building an autonomous mower, this could be useful if I could record video of RC crawler over yard, and use that to train the autonomous mower.

[@UsmanAhmed-sq9bl]

Superb Siraj

[@DanielWeikert]

Great Video again. One Question about the CNN. Does the CNN actually still need to "classify" the pictures? Since it only needs to recognize certain shapes which are stacked (basically a pattern) to learn the right steering direction. Thanks

[@suman2449]

sentdex inspires siraj I love them both

[@SirajRaval]

indeed love u

[@tjj405]

how to install udacity self driving car on macbook. I downloaded the file for mac but its not allowing me to run it..anyone tried on mac? please help

[@10monthspregnant]

Siraj, please cite the resources you use. images.nvidia.com/content/tegra/automotive/images/2016/solutions/pdf/end-to-end-dl-using-px.pdf

[@allaboutdata2050]

Thanx for the pdf file

[@rickyshi9735]

This is exactly what I'm looking for, thx!

[@musiclover470]

When i try to run the model.py file i have this error i am using python 3.5.4 IndexError: boolean index did not match indexed array along dimension 0; dimension is 160 but corresponding boolean dimension is 66

[@truegod1205]

github.com/llSourcell/How_to_simulate_a_self_driving_car/issues/16 they fixed!

[@fiosnetwork4314]

Can you explain the back propagation in the convolution layer in terms of updating the kernels involved in another video perhaps ? BTW great work !

[@MinecraftLetstime]

This tutorial is really good!

[@aladdindorado3608]

Sir i trained my self driving car and saved the csv file in the desktop but when i run the program in spyder it returns an error: FileNotFoundError: \\data\\driving_log.csv' does not exist. What did i do wrong? Please help me. Thank you.

[@fauge7]

It took me a while to get this to run, turns out, the csv file is not perfect, you have to add a header to the csv file that reads "center,left,right,steering,speed,throttle,brake".

[@jcxmej]

It was 'center,left,right,steering,throttle,brake,speed' for me.

[@fauge7]

Jaffin MK I never got this working so maybe that will help, I got it to compile but it wouldn't train

[@jcxmej]

Why? Whats the problem?

[@vighneshsrinivasabalaji802]

I have a few things to clarify. When I run the drive.py file with the model.h5 file, though it says 'Recording this run' and starting wsgi, nothing happens after that. 1. Do we need Unity Engine and load the Udacity Simulator on that? if yes, then does Unity work on Ubuntu 16.04 2. The python files, the .exe of the simulator and also the image folder which contains the data is in the same folder. The driving_log.csv has the absolute paths of the images. Where should each component be present? Is there some directory problem here the reason for my problem? 3. What are the arguments needed to be provided by us while running the drive.py and train.py file? will the drive.py file work even when the model is not saved as a .h5 and saved as .ckpt? 4. How to add the headers to the csv file?

[@christiejacob3541]

Could you shed some light on the image transformations happening before the images are fed into the CNN ( from the end to end learning paper from nvidia)?

[@reyariass]

Could you show and make a simple ai for a game using openAi and a custom environment?

[@udos19928]

Great video and fantastic content! thx

[@WreckingU]

You fine sir...deserve more subscribers.

[@SirajRaval]

thx getting there

[@coreyk123]

Great video! How do you integrate other cars, stop lights...?

[@AnkitBindal97]

Hey Siraj, Can you please tell me how does a self-driving car choose the route when there is multiple paths available? For example a T-point or a road intersection?

[@karangupta8999]

you are just outstanding just loved it

[@moebazzi8412]

Hey Siraj could you do a video illustrating how GAN's could be used to generate 3D models/pictures/videos using 3D Convolution networks. Thanks and keep it up!

[@versurio]

34:39 We are listening you carefully, Siraj

[@SirajRaval]

good

[@nitesh4146]

Siraj, do you know that you're awesome!? Keep up the good work.

[@JithinJose2]

Good video, but it will be great if you explain 'why' instead of 'what' a line of code does.

[@vibhanshusharma3150]

Do you have video on image segmentation

[@techtube2279]

@Siraj Raval I am downloading the Udacity simulator using below link , but where is the executable file in MAC system ?? I am not able to install in MAC Mac

[@jaydeepdas9326]

output of generator should be a tuple (x, y, sample_weight) or (x, y). Found: None plz somebody help me !!!

[@ganeshtowar8050]

Tesla developer: guys note it down its free

[@amithbk12man]

Nice one! Can u take up a session on Distributed Tensorflow????

[@gersonjordanvillugasavella8266]

can anybody help me with this steerin_angle=float(data["steering_angle"]) Value error :could not convert string to float : '0,00000'

[@themerkaahd1256]

print out data["steering_angle"], double check what you get

[@happydays3300]

how did you balance this data? if the outputs are all different sizes, wouldn't the data be more bias towards the highest output data? therefore the model would not generalise well on new data? interesting, great Vid

[@akshatagrawal1749]

So here's a doubt. If I train the car in one track and try autonomous mode on another track(but keeping the surroundings same, the track itself would be different) , will it work?

[@leonardolima1574]

Dude this is just awesome

[@saaho-m]

is there a tutorial on the same for cruise control

[@Calciuca]

Could someone help me? I get the following error when I try to train the model: TypeError: ('Keyword argument not understood:', 'subsample')

[@vijaykumawat256]

bohot badhiya

[@AnkitBindal97]

Dude.... you rock!

[@marcodomingo6763]

Hi Raj, does this mean the output of the last layer containing a single node is a range of numerical values representing the steering angle? I'm fairly new to neural networks and always thought the output is somehow a classification for example in steering angles - left, right, sharp left, sharp right, or with objects - dog, cat, human, vehicle, rather than an actual computed value that gives an absolute steering angle. Hope this makes sense and thanks

[@techyt8129]

hey bro i want small help regarding this project

[@raymondchua280]

Can I use this to train a reinforcement learning agent?

[@amishakodape]

How to download dependencies? I am unable to download it even with pip installed

[@victorsung3582]

Is the Udacity program specially programmed to make it easy to interact with an external agent? Or is this technically possible with most/all game applications? I would love to apply a similar approach to another game!

[@kingpopaul]

Quick question, pytorch docs are boasting about using dynamic graph/models compared to tf and theano which use static graphs. But are dynamic graphs really needed or useful? (currently I don't see any advantage to that, maybe in the future it'll be important but I have some doubts) What are your opinion(s) on that?

[@Micou1990]

Siraj, could u make up a tutorial how to use your readme data, i.e. how to rebuilt the model? I tried quite hard but i wasn't successful.... Would be awesome, thanks.

[@mashnoonislam1126]

Do you think enrolling in the Self-Driving Car Engineer Nano-degree Program is necessary to build a real self-driving car? If so, tell me more. If not, can you provide some additional concrete resources so that I can get started to build one of my own?

[@feyzanur8860]

Great! Thanks it helped a lot! :)

[@mohammedinnat8995]

But how you train your model , is it directly from jupyter or sublime .. ? sorry for silly question but i'm novice and confuse at this point :|

[@irfannayeem-cybertom1307]

can anybody help me how can i made this project runable on my system

[@vaibhavbhana2570]

Bro great video! How to add obstacles?

[@MorisonMs]

Siraj for the president gj

[@vijayakumarj8453]

Hey Siraj since the version of keras got updated the INPUT_SHAPE is deprecated so what is the Input_shape means here? I need to know the extract value of it can u explain me in detail or a reference will be great help Thanks for the awesome video!

[@suchandratungare7727]

dude ur the best :D !!

[@7olie]

Thanks Siraj for another great tutorial. Would you know if there's a way to change the max speed of the vehicle in the simulator? Also the gamepad doesn't seem to work for steering in the February release but it does in the November release. If anyone in the comments could help me out with this that would be great. Cheers

[@hooz84]

How can I run the Udacity simulator showing (or printing into a file) logs? Data sent by the "steer" socket event are not recognized by the simulator but I've no feedback on what's going wrong.

[@dexlee7277]

what does model.summary() do?