Andrej Karpathy - AI for Full-Self Driving at Tesla

For more talks and to view corresponding slides, go to scaledml.org, select [media archive].
Presented at the 5th Annual Scaled Machine Learning Conference 2020
Venue: Computer History Museum
scaledml.org | #scaledml2020

[@mathew.a]

I had to double check if my playback speed was set to 1.25

[@angelpico3236]

I usually watch videos at that speed but It's impossible with this Karpathy guy

[@MuscleTeamOfficial]

I used to watch all of this guys lectures and up to 1.5x speed, im so comfortable w/ this guys voice I thought he was actually speaking slowly so I cranked it up to 1.75x until the end i lowered it to 1.5x

[@Migus29]

I work with guys like him and I also speed up like him. It just means that not only you are smart, but you dominate the topic and you have it extremely well structured in your mind. So well that sometimes I can do fast improvised speech in front of many people and still have spare brain power to have separate thinking line.

[@salessiteboost7665]

Yup agreed. This guy is an absolute weapon in the field! WOW

[@switzerland]

Watch at 0.75 and see how ok it sounds, amazing

[@GregHassler]

His name is Karpathy... Kar-pathy CAR-PATHy Destiny.

[@bernardfinucane2061]

Ouch, And I thought it was because he comes from the Carpathian Mountains.

[@AdamFoldes]

I heard that he was actually hired only because of his name... the rest he learned from Elon...

[@shishirsks]

Good one! :-)

[@renjithravindran5018]

An Indian would read his name as the *King of Cars*!

[@kaisermatin2]

Kar-ma

[@leixun]

*My takeaways:* 1. What is Tesla Autopilot 1:20 2. Tesla's methods are heavily based on computer vision rather than lidar 5:25 3. Neural networks in production 6:55 4. Receive training images for tricky cases from the fleet 8:35 5. For testing, it is not enough to just rely on loss function and mean accuracy of test set 13:00 6. HydraNet contains 48 networks with shared backbone, 1,000 distinct predictions (Number of output tensors) and it takes 70,000 GPU hours to train 14:12 7. Neural networks for full self-driving 16:54 8. Get depth estimation from images directly by using self-supervised techniques 22:54, predict the depth, drive to it and measure the real distance 9. other uses of self-supervised learning 25:24 10. Q&A 26:50

[@debayandas1128]

Thank you

[@sumitmandloi9201]

Thank you bro.

[@carvalhoribeiro]

so handy thanks

[@fc.soccercard]

Thx!

[@canimbenim427]

What a job. Thanks a lot

[@sashapetrenko74]

1:06 The vehicle initials spell "Sexy Cars." I love it.

[@gilgalin8917]

"Model 3 was going to be called Model E, for obvious dumb humor reasons, but Ford sued to block it, so now it is S3X," tweeted Musk. "Totally different :)" What's more, Tesla has made noise about also producing a Model Y crossover vehicle - the Y in a thwarted S-E-X-Y line-up. Denied his E, it's clear now why Musk went with the number three, essentially a backwards E, in a line-up that otherwise sticks to letters. (c)copy+pasted from news article

[@ShadyRonin]

Andrej is a genius. The fact that this dude can speak that fast and touch on all these enormous abstract concepts while barely taking a breath just shows the speed this dude’s mind works at. Amazing presentation as always.

[@davidjkuchar]

Exactly!

[@MarcellNagy]

At first I thought that the playback speed was set to 1.5-2.0 on my KZbin hahaha

[@punkster36]

Andrej's hand/arm gestures can be used as a training set to try to predict his speech

[@Muhahahahaz]

Lmao

[@snoopysnoops007]

I'm an ML engineer and this is my first time actually seeing details for the Tesla AI under the hood. This idea of unit test curation and a hybrid 1.0 2.0 system is really clever. Put that together with building a fleet of essentially data gatherers and labellers, it's truly ingenious!

[@debayandas1128]

Isn't it!

[@sebbecht]

Completely agree! It makes me wonder.. how many AI vision applications are there were you can have the application curate data and label data to this extent? I at least find myself envious.

[@tanyouliang]

It's mindblowing when alot of robotics/autonomous companies are coming out algorithims to solve some crucial problem in self-driving, AI is the core of the whole autonomous driving pipeline. And the most impressive part is deploying it in scale.

[@henryallenlaudemilk5161]

5:40 LiDAR vs vision 6:55 What vision means 8:35 Why it’s difficult- a stop sign detector 10:25 why you can’t solve it without the fleet 13:00 how Tesla does TDD 14:00 the nitty gritty of the NNets 15:20 building the infrastructure 16:55 2D vs 3D 19:40 Bev Net 22:15 pseudo-LiDAR : self-supervised learning based depth prediction 24:05 driving policy

[@AmnesiaPhotography]

Absolutely phenomenal. Love the use of the fleet, TDD and CI in order to check for regressions and focus on what matters.

[@vinothborn2win]

Wow... Mind blown.. Great presentation.. Mind blown at how one could just request fleets to capture and send back data for training set..

[@realulli]

Especially, *GIANT* fleets. I mean, other manufacturers have a few hundred cars out there to collect that data. Tesla has 1.000.000!

[@debayandas1128]

The unit testing idea was fabulous. I'm gonna copy it shamelessly.

[@tanmayjaiswal5935]

I think as a programmer you stop feeling shame when it comes to copying code, architecture and techniques pretty early in your carrier 😂 Part of the reason the software industry grows so fast is because we don't reinvent every wheel.

[@gobl-analienabductedbyhuma5387]

Impressive guy, impressive technology

[@unexplainedmysteries9540]

160,000 views and 0 comments? I guess you left everyone speechless

[@gilgalin8917]

that's cause it's embedded in scaledml.org and no option to comment

[@ah64Dcoming4U]

Thanks Andrej!

[@marc_1285]

Ok lesson learned. Teslas are awesome and I will need 20 lifetimes to understand how they work.

[@TheKdcool]

Autopilot team is awesome

[@pixels_per_inch]

Very interesting, I did not expect Tesla to use NN for per pixel depth prediction. I was sure it was using NN only for object recognition with optical flow for 3D estimation, judging by how FSD beta draws tiny dots for objects the NN hasn't learnt. Or is it not implemented in the system yet?

[@luccahuguet]

Amazing. Looks very scalable.

[@jamesatkins7592]

20:11 That's cool. I haven't seen that logic/prediction analysis done by other innovators although I guess they could have done it. First real evidence I've seen that the computer vision approach is likely possible with machine learning/neural nets without the need for true AI.

[@alexeykononov5596]

Wow! Intense talk ;)

[@skaterope]

high quality stuff

[@editg121]

stop sign can be rectangular shape as well in some countries

[@latabai3533]

Which algorithm and software are you used??? Hydra net explaintation..

[@billykotsos4642]

The ML infastructure required is massive. Also a lot of this infrastructure requires a lot of soft 1.0 code.

[@maqboolfida786]

If he speaks so fast...imagine how fast he thinks.

[@davidjkuchar]

Exactly!

[@SameerKhan-ht4mx]

12:34 now that's what you call a flex

[@DefaultName-dp4cu]

What about unexpected problem solving? for example, I encountered a traffic signal that was turned side ways by wind. I cannot see whether it is green or Red and because it was low density traffic there were no cars moving on the road cutting across for me to see when they stop. What I did is I looked to the signals controlling the traffic flowing across and when that turned Red, deduced mine has to be green... can an FSD car do that? can FSD car cooperate with human drivers? like someone waving you to pass or flashing his head light for you to pass? what about hearing sounds. like fire truck, ambulance, car horns..etc.

[@archangelgaming4259]

Setting the playback speed to 0.75 helps a ton.

[@VDD0797]

Hi I want his work as I want to study autopilot vision so where can I find His research paper

[@Jedi2155]

I love how he goes "yes, yes, thank you very much when someone kept asking questions" - LOL great way to shut up continuous questions.

[@heltok]

It was a thank you for helping to label their maps

[@randomwalker332]

It was a great question but not a direct even relevant answer.

[@yangtianxi3980]

depth estimation is not explained at length, this part is core for 3D reconstrunction

[@salessiteboost7665]

Replicating LIDAR performance using Cameras and NNets...Thats insane!! These guys are gonna win the race. Trust me. I will never doubt Elon and Andrej again!

[@nalinbranden]

Awesome presentation. Question: when he says '"fleet", that means all the autopilot enabled Tesla cars currently owned by public, or a dedicated fleet (purely for testing) of Teslas driven all around the world by a group of Tesla employees or contractors?

[@juliahello6673]

He means all Tesla cars, regardless of whether the owner has paid for full self-driving.

[@johndehaan2764]

This man requires more security than Elon Musk.

[@jamc666]

very interresting

[@pd.dataframe2833]

how are they getting the 3d position then. from the point clound? how do they know the orientation of the vehicle to draw a cuboid around it?

[@sameerningoo6236]

I think they do SLAM .. simultaneous localization and mapping.

[@ivoriankoua3916]

My bad I thought that only George Hotz was kinda good at DL, The Autopilot team for sure have many of them.

[@AdamWood]

It's happening!

[@user-bl2iu5ob2n]

Install all network devices on the road that can pre-scan road conditions. Autonomous driving can be accelerated by 30 years.

[@johndehaan2764]

did no one notice that at 1.00min in you have a graphic of the tesla model line up and it spells sexy cars??

[@sovitrath4735]

Looks like neural nets are going to be present in the AI field for a long long time.

[@johndehaan2764]

I will say this once more simply compute a refraction indices co-efficient for observable light spectrums and you will solve depth perception.

[@johndehaan2764]

The spectrum of light across all of it's spectrums is inversely proportional to its wavelength. Hello.

[@dhruvg6437]

AlexNet in 2020? How come?

[@aimmaz]

Can't wait for the Tesla transformers car. Would totally use the robot form for work.

[@kenhiett5266]

I rarely feel intellectually dominated. Listening to Andrej give this presentation was humbling.

[@edwardolvera5280]

for a second, just for a second : I thought the TESLA autopilot would be already a system that was capable of learning on their own. But no, yet not

[@edwardolvera5280]

y

[@surfermx]

YES U ARE ALL RIGHT THESE WEIRDOS HAVE TO PREVENT AND FEED THE SYSTEM WITH ALL THE POSIBILITIES THAT THE CAR COULD MET, haha thats really STUPID thats why, i see, the TESLAS KEEP CRASHING

[@oxide9717]

@@surfermx lol

[@justnate-4810]

If I want to apply for this job, what programming languages would I need to learn?

[@conorlieu4635]

He did mention C++, but ML is more about math, linear algebra, and calculus. A lot of ML in the market uses Python. But yeah, language is the easy part, understanding the math and understanding how ML works is harder.

[@otub]

its here

[@oxide9717]

Men do much information

[@johndehaan2764]

What a mind!!!!!!!

[@NeuralEngin33r]

24:20 The driving policy is still fully hand crafted. They have so much farther to go before full sell driving.

[@Stranger-pz4fp]

can u explain much please ?

[@craq47]

In principle at least, you can have full self driving with hand crafted driving policy. There are controllers which deal with uncertainty in the inputs from sensors in deterministic ways. But given the sheer number of possible situations you might want to cover, it might end up being quicker (in terms of development time) to use AI end-to-end. I don't think it's actually clear yet what the first full self-driving vehicles will use.

[@NeuralEngin33r]

I doubt that a fully hand crafted diving policy will deliver full self driving. A fully hand crafted policy can surely work 99.9÷ of the time....but some level of ML is needed for those rare circumstances. And with such a large data set, it should be achievable to learn the driving policy.

[@dalerichardson7202]

No, he said that the driving heuristics that is hand crafted is doomed to failure and that they are implementing 2.0 to gradually subsume the driving policy. A specific example he gave immediately was running FSD in shadow mode and pinging back to Tesla when the driver performed an action that the shadow FSD definitely would have never done..... This is the way and I approve their strategy.

[@Phsoco]

Lol imagine your job title is "Stop Sign ML Engineer"

[@theSpicyHam]

since it's getting gamefied.. may as well try out some games, or more so as previously usually did before!

[@collwyr]

that poor guy who asked the question first, I don't think he understood the answer properly when Andrej gave it, and the way he just shuts him down with "yes, yes, thank you very much" was funny af.

[@akarmdit2267]

indeed kudos ماشاء الله

[@edwardolvera5280]

jesus its imposible to hear this guy

[@johndehaan2764]

This is the smartest person on earth, He must be protected.

[@KARTIKEYA007]

His channel is badmephisto

[@pratik245]

Andrej is a hero now for Tesla.. 😂😂😂

[@SDLordHUN]

I understand that Tesla HQ is in the US, but I believe they would have less problems in the EU where you have more sign types but less signs with text and whatnot.

[@AllBecomesGood]

Maybe, maybe not. Really hard to tell wtf is going on in many different places here as well, prob lots of funny special cases hardly anyone actually knows about. But that's why they build it the way they do, ie based on Vision, coz then you aren't completely forced into 1 place/country, but can much more easily make it work all over the world by just repeating the steps you took to make it work in the US, plus some knowledge is prob usefully transferred by the system and can be used in other countries

[@SDLordHUN]

@@AllBecomesGood pro'lly it has more to do with regulations and the US market being closer to their current main manufacturing output. Though I agree somewhat, the more complicated it can handle the less work it needs to be implemented in other markets.

[@patricksokoloski2622]

I hope that Tesla full self driving will be complete and hope that Tesla will have fully autonomous cars by the end of this year. I want a level 5 fully autonomous Tesla, so that I can sleep, text and talk on my phone, be at my destination and so that my friends and I can go to a bar and have a couple alcoholic drinks and get drunk together and not have to worry about drunk driving, since the self driving car will be the designated driver and drive us home safely.

[@StevenAkinyemi]

It won't happen.

[@patricksokoloski2622]

The Last Hacker It’s going to take a few more years right? Do you think you most people are going to trust a car to drive by itself or no, would most people rather be in control of the car?

[@gairabad]

I searched for this video because I was curious where the "hard part" was with full self-driving, since Tesla has been promising for years and hasn't yet delivered. Which of the challenges Andrej described is the real bottleneck to FSD?

[@dalerichardson7202]

It's the edge cases that is the real bottleneck. His example of the stop sign (except when turning right) can only be solved with NN output, ie Software 2.0 vs Heuristic coding, Software 1.0. Everytime the NN is updated however, to solve for the newly discovered edge case, the entire NN has to be retrained and revalidated to make sure it didn't break something else....

[@Shiffo]

I came here for similar reasons and this video made it clear for me. You need to understand that computers are incredibly stupid / childlike in recognizing objects, so you need a lot of data in order to train it and make it recognize objects. What became clear to me in this presentation is that for every task / every object data needs to be collected and trained on the system. The AI team will keep on creating new tasks for every disengagement they analyse, they will request the fleet to collect data and will train the neural net on this task. the real bottleneck is: collecting data on edge cases (which can be limited), adding edge cases as tasks to the 48 neural nets and training the system. Releasing FSD in Europe is going to be hell.

[@KARTIKEYA007]

Am I the only one who knows him as badmephisto who made Rubik cube videos?

[@ruriker7063]

FSD Beta it's here!

[@amoltandon2900]

AI Day 2021!

[@jahmbo]

It’s magical (when it works). that was funny.

[@benrayfield2153]

What if someone hung a stop sign over a bridge on a hiway?

[@charlesfeng3823]

This a very profound question. This is about safety. However, we need a law to guaranty this is not happening at this stage.

[@myskullisred]

22:01 RIP pedestrian

[@ktop4u]

Why not just use lidar instead of creating low def maps, camera feed?

[@1flash3571]

It is because with Lidar, You have to MAP out the U.S. roads and MAINTAIN the mapping. If the route change, they have to map out the changes. You have no idea how hard that is because when there is a ROADWORK, someone have to map out the changes EACH TIME. Stop being silly.

[@ktop4u]

@@1flash3571 Google feels differently

[@sameerningoo6236]

LIDAR makes the car very expensive.

[@FizZovic]

yeah, you slam on the brakes, but what happens when there is snow or ice or moisture on the road? slam on the brakes and you take a hit.. unless it doesnt react sooner than in the video

[@pratik245]

Why can't a human not annotate depth? They can based on an understanding of a measuring unit.. Same thing any sensor can do

[@amoltandon2900]

500,000 cars delivered in 2020!

[@editg121]

In some countries, you dont even have lines for parking spot. Go to India if you know what I mean.

[@dagma3437]

Tezzzla?

[@dislike__button]

Yez.

[@filipmichalsky1143]

When referring to "the fleet" sending over data - does that mean every consumer who drives Tesla sends their data without their knowledge back to Tesla data lake?

[@simonschneider9547]

Not without knowledge. The Tesla asks for permission

[@mouzurX]

Yes. Its all anonymized though.

[@craq47]

@@mouzurX how anonymized can it be when it has GPS labels? There wouldn't be many teslas that drive the same route between home and work as I do. Anyway, I think it's part of the bargain when you buy a Tesla, and most owners are aware of it.

[@oferkrupka]

I wish I had more money to buy more shares. As an investor It's important to see what can bring your investments down, I am trying hard to find a competition that might be able to beat Tesla... But all I get is *Error 404 - Competition Not Found* VW are the only ones outside of Tesla that finally gets it, but they only started to be 100% serious a year ago and they still need to beat the Model S of 2012 in range, and I didn't even mention self driving or battery plant. so I think they have a good chance to be in 2nd place (or maybe Rivian?), because there have to be someone, Tesla can't be the only car maker in the world, but they sure gonna be the largest one.

[@johndehaan2764]

This is a perfectly sharp mind. Elon Musk would love to be as sharp as he is. Elon draws competence on levels that ultimately make Elon look elementary, This man is so excellent. He is the ultimate weapon when Law meets Technology, He fuses intelligence with the art of the adversarial. Your cannot know when hardware that is internally manufactured equals software.

[@h2o2u29]

Not even a minute in: S E X Y C A R S Love the memes (If you read from top to bottom)

[@johndehaan2764]

refraction indices co-efficient, relative light will improve depth of field. Come on.

[@mementomori7207]

Watching this at 2x speed you need to be focused

[@Samusepicness]

Wait... This isn’t my game...

[@swait239]

“When we come to an intersection, we come at it for the first time” why? Humans remember places and small details about roads over time, maybe even seasonal risks over time, why shouldn’t the neural nets take place and time into consideration?

[@MagneVikjord]

Sometimes we arrive at an intersection we’ve seen before, but this is an easier problem than when we see it for the first time. We have to solve the harder problem for self driving to work, and then by default the simpler problem is already solved

[@Level6]

22년도에도 들으시는 분?

[@Level6]

kzbin.info/www/bejne/nqmac4ufnZytrpo

[@philoso377]

Page 1:36 depicts 3 billions vehicle-miles total learning is unbelievable ... I can believe we have accumulated 3 billions mile-vehicle of data but don’t believe we have enough human resource to train and mouth feed all that into the neural network, i can believe hand pick samples were used. We still have to manually “tweek” the AI that is why it take so long. The AI spit out one or multiple choice, the team have to respond which one is/isn’t the stop sign or stop light. Which curve situation for what amount of steering. Is the EV drift too left or right .... Anyone thinks that AI learning / training is made real time on each vehicle are coarsely oversold by those hypers.

[@haydenmacfarlane7194]

Andrej talks at 1.25 speed

[@RiCKTEEZ]

He’s not speaking fast, you’re just thinking too slow

[@e3chicago]

There's something Tesla seems hiding there. When one hacker looked at some internal Auto Pilot data, Tesla cars clearly use preloaded maps with stop signs and other road information. Real time vision data are used to enhance or override the maps and overlay what the cameras detect, like moving and stationary objects, traffic lights (colors), etc. If someone owns a Tesla, try covering a stop sign and drive by it. If your Tesla does not see a sign, it's purely vision-based. If not, there's something else...

[@craq47]

Andrej was pretty clear that Tesla uses low definition maps with important infrastructure like stop signs. They just don't have the high resolution 3D maps which other companies use to localise the vehicle.

[@innomind]

Is that a Russian accent he has?

[@Muonium1]

no

[@madcalm2024]

Ukrainian more probably. His lastname refers to the mountains in the west of Ukraine.

[@oferkrupka]

Mad Calm No, it’s coming for his work at Tesla: Car-Path-Y

[@valentinkrizan2754]

I belive he is Slovakian originaly.

[@imranq9241]

There are so many exceptions in self driving that solving all of self driving is pretty close to agi

[@hugodasilva6691]

And with the rain? nobody talks about the behavior in the rain or does it not rain in america?

[@Brendon471]

Hugo da Silva The current version of autopilot handles heavy rain no problem. Usually better than me at seeing the lane lines actually. I’d guess this more advanced version will be significantly better (in all areas actually).

[@rickmartony9566]

lidar struggles when it rains/snows.

[@fahadahmed8143]

1:08 "SEXY CARS"

[@dclpgh]

I didnt understand shit. that's why i know it will work

[@tomwup]

Now we have to fix the EU regulation from yesterday!

[@dixonpinfold2582]

I really hated the way he shut down that guy who was trying to do a bit of follow-up. Notice that it's the Most Replayed moment, because people didn't know what on earth they just saw, so they had to go back. No one acts like that. Maybe somebody was supposed to grab the mic from the guy as soon as he got his main question out, as is common practice. But if so, live with it-and don't treat anyone like that. Really came across as arrogant and pompous.

[@anand_dudi]

now we can say computers are smart and humans are stupid

[@user232349]

It sounds all impressive, but when you look at current FSD visualizations, they are full of errors.

[@ytxstream]

Recording the world around the car and sending it back is illegal for privacy reasons in lots of places though.

[@korawega6293]

you agree to the terms on purchase of product its a legally binding contract

[@juanok2775]

my problem with Tesla is that they lie on their current product, they dont have full self driving at best they have L3...

[@fields1]

He needs to slow down. Good god he talks so freaking fast.

[@loveanimals-0197]

Amazing how these "researchers" don't address the basic flaw in this type of ML. Drivers are overwhelmingly driving around their houses most of the time, so the data they provide is the same thing over and over again. Of the billions of miles they claim to have, an extremely miniscule amount would be rare cases like severe accidents. There's just no way to tell for sure if all outcomes have been accounted for. Also, most drivers don't typically get into accidents, but that doesn't mean the AI will also do that. So, the billions of hours metric is just BS. It can only serve marketing purposes. Let's take the case of dark night with a dark scooter and a rider wearing black. Autopilot is either going to ignore these objects and risk driving blind to small vehicles when visibility is poor or it can be programmed to be overly-cautious and will frequently hallucinate that moving shadows on foliage are also scooter riders (causing the car to erratically break or swerve in efforts to avoid collisions with the imaginary objects around it).

[@seankelly9870]

He often addresses this. Just saw a lengthy talk on it in the CVRP presentation. How to surface the edge cases you're talking about.

[@EngineeringNibbles]

There's a radar that serves to help for your kind of edge case collision prevention

[@craq47]

The key word he mentions in this, and the CVPR presentation, is "long tail". That means they explicitly mine their dataset for rare events and prioritise including them in their dataset. It's still a massive (possibly infinite) task, but it's definitely being addressed. Keep in mind that self-driving vehicles don't have to be perfect, they just have to be better than human drivers. How would the average human perform on your test with a dark scooter on a dark night?