[CVPR'22 WAD] Keynote - Ashok Elluswamy, Tesla

Talk given at the Workshop on Autonomous Driving at CVPR 2022.
2022-06-20

[@floxer]

Thats so epic. Tesla AI Day 2 can't come soon enough.

[@SirHargreeves]

I’m counting down to it like Christmas.

[@DirtyTesla]

Thank you for all of this hard work. I'm excited to get this software onto my car! Onto cashless cars!!

[@loonatic90]

*crashless 😉

[@qunzuo171]

That's impressive! Thanks Ashok and the autopilot team!

[@SaadAhmed3000]

Omg they're using NerFs. That's absolutely wild

[@CampTeslaFun]

The progress has been amazing to watch. Can’t wait to see the FSD beta 10.69 videos. Thanks for making us safer, Ashok. Congrats to you & the Tesla AI team! Exciting!

[@slavko321]

Chuck cook and dirty tesla (at least) already got the updates, videos are coming soon!

[@sabahoudini]

@@slavko321 Chucks videos are already here, so far it seems like a huge leap forward.

[@XPengMotors]

That I am one of the only 37k people worldwide who have seen this revolutionary new technology is so unbelievable. History in the making!

[@CensoredSheepChannel]

@SawyerMerritt send me here You guy's 🎸 rock 🙌

[@TeslaCylon]

Same

[@na1067]

Good luck to anyone trying to catch up with Tesla, This is what real innovation means :) Thank you Ashok and Tesla Team :)

[@mahendrareddy6]

@sawyermeritt thank you for directing me here. Thank you Ashok for amazing job your team is doing at Tesla

[@MrFoxRobert]

Thank you!

[@runeoveras3966]

Amazing. Thank you for this. 👍🏻

[@SiestaKeyJimmy]

As I watched, I was struck by the thought that the problem is very similar to the NC machining problem. NC Machining Problem: Compute a safe “tool path” through a valley of interfering 3D obstructions Autopilot Problem: Compute a safe “vehicle path” through a valley of interfering 3D obstructions (with the additional challenge of doing it in real time)! Back some 30+ years ago, I worked with a research team to explore different software approaches to solving the NC machining “interference problem”, (that problem being our computed “Tool Path” kept cutting through adjacent parts of the 3D model because the “tool path” was unaware of the many 3D obstructions surrounding it.) Part of the solution to that general NC problem, which we came up with, was to capture every possible “obstruction” in the adjacent area and build a kind-of 3D “protective bubble” around each item. Then compute the tool path and check it didn’t intersect any bubble, and correct the path if required. I want to work at Tesla in my next life!

[@jonathanreader228]

wow - amazing work!

[@jamesabraham5381]

Great job, Ashok! Great job, Tesla!

[@foodmaker5771]

Wow we need more such of this lec!

[@GaryCalcott]

Incredible.

[@vittoriobenedet2221]

Thanks a lot Ashok!!!! You are awesome!!!!

[@varunvelpula4516]

This so exciting!

[@gridcoregilry666]

Thank you for the upload! Can't wait for FSD to come true eventually!!

[@chandermeena1490]

such cool stuff

[@kyril3125]

Truly amazing

[@LouieGrind]

Very cool and informative!

[@RealRusty]

Very insightful, thanks!

[@ken830]

Super-impressive!

[@kenlund7004]

Great talk! So excited about the progress being done! Great work!

[@romanwowk4269]

Thank you Ashok for all of your hard work and for presenting this to the world.

[@kunletim8464]

History is being made here👏🏾

[@thepee-onpress]

Congratulations 🎉🍾🎊🎈 How exciting for the world 🫶🏼

[@sfkjbg]

There is nothing like watching the future unfold before our eyes.

[@roddlez]

👏👏👏Thank you, Ashok and CVPR for making these presentations public! This video makes it clear that Tesla's pace of innovation is incredibly high. Very excited to see the full suite of Collision Avoidance features in production and one day have vehicles that fully account for human error, avoiding accidents and crashes.

[@gailalfar9752]

Love this. The more information the better, it all leads to safer transportation situations.

[@AvinashKumaravinashasitis]

"Worth hearing about Tesla Autopilot software/AI progress" - Elon Musk

[@digitaldreamer8637]

Massive architecture changes! Awesome work guys! I bet some of this makes into Optimus! 😉

[@pavelt9391]

Simple. Genius. Amazing.

[@binkding]

fricken awesome

[@4puf]

Great presentation. It has been really nice to see how Tesla has changed the direction of their FSD architecture and every time getting a little closer to solving FSD.

[@Tommm73]

Thank you Ashok -. Great work, great presentation, and as one of those 100,000 FSD Beta drivers it’s exciting to see what’s under the hood.

[@officialyasir]

I totally understood everything in this presentation!! 😂

[@richardteychenne3950]

Brilliant presentation, you turned a complex final solution set into a clear illuminating logical route. The Tesla's teams progress gives one confidence in humanities future potential.

[@walkerrichardson]

This was fantastic. I'd like to think I understood most of it, might need a few re-watches. But it's possible to see where FSD is going and how close it is to be publicly available to anyone who wants it.

[@zilogfan]

Best presentation ever. You and your team have done an outstanding job. I am proud to be a beta tester on your project!!

[@op4394]

good presentation, can be used for example as a debunk of this mannequin/ children mowed down à la Dan O'Dowd. Was quite clear to me after rewatching AI day last year. Basically every obstacle on your FSD car trajectory path is just something to avoid in that time horizon from 10 ms starting evaluating the field to 2/4 sec on path, whatever it's moving or fix, a car, a human, a dog, a bird or a mannequin. One feedback given about 40 collisions A DAY avoided (vs 273 in total included in NHTSA current open inquiry) that has high value. Ashok, you are very good in explaining complex stuff, and even if there's still plenty of topics in work, you could be a tick more confident in your voice & communication that there is no way you are not going to solve level 2 to 5

[@brewedicedcode1341]

Great presentation Ashok! So good to get in-depth understanding of how FSD works. Give you sense of how complex collision avoidance problem is. Thanks for great work from you and your team.

[@explor794]

Brilliant, I hope you got a lot of stock options for your work.

[@sennlich]

Wow !

[@trip_to_moon]

Just a little treat before AI Day 2. Cant wait for it... and cant wait to know how we will resolve AGI from v0.1 to v1.0 and beyond

[@slavko321]

Great work! The only way to cram the intelligence into local compute is to differentiate low and high resolution necessities (even strategies, like the eyes seeing differently on the corners). What happens when a pillow is dropped in front of the car and a huge truck is behind it (highway speeds)? Or a child? Squirrel? Lots to be done, obviously going in a good direction.

[@Discoducky73]

The most advanced Autonomous AI in the world. Well done Ashok!

[@casvanmarcel]

This is just mind blowing how advanced the system is so far. Of course it's not perfect (it will never be 100% perfect) but with the rate of progress being so good, statistics will show how better FSD is than humans. I have so much respect for the Tesla team, and can't wait to see more updates in the future. Thank you for the presentation and good luck with everything you guys are doing at Tesla!

[@MsAjax409]

As I understand it, FSD Beta does not use HD maps, but it does depend on course street and intersection topology maps for drive path planning. I suspect the quality of these maps varies across the country which explains why the performance of FSD Beta, in terms of frequency of disengagements, can be so good in some places, and disappointing in others.

[@AhmedSaeed97]

Collision avoidance is insane 🤯

[@samuelcarbone9870]

Images in 9:52 are from North Terrace in Adelaide. Interesting to see they’re using images from aussie cars 👀

[@kalleballex]

Thanks Ahok! You are really good at making advanced concepts easy to understand. Not an easy feat. Not surprised Tesla put you up to this speach. Aplaude!

[@davids3905]

Going to need James douma to explain this

[@pw7225]

What question do you have?

[@op4394]

@@pw7225 in general clear after 1,5 view (rewind on some concepts), just the 21mn passage on diffracted light or rain, the one possible solution using high level descriptor is kind of fast treated. You just get a glimpse with the enhanced imaging for the street view what it's. But this enhanced street view image doesn't go back with a better view for the rainy situation picture from the minute before as the problem was described.

[@pw7225]

@@op4394 Puhhh, dude, GPT-1 had better language skills. I cannot parse your sentences.

[@op4394]

@@pw7225 go back to school

[@dwilliamsnetosnet]

recent accident on FSD involved a cloud burst and hydroplaning. Can FSD sense intense rain and slow down like it does for numerous other conditions like, slow car in front, exiting, speed limit changes etc. My sense is that FSD did not respond to loosing control due to hydroplaning, the car went off the road (very stable) and broke the LF lateral (angled) suspension arm.

[@vinayr.6015]

So much hard work goes behind invention, it’s very easy to dismiss and say FSD beta sucks when it’s make mistakes but things are going in right direction.

[@hornetutube]

Thanks for the insights - great stuff. But the clips being shown do not seem to run anywhere near 100Hz? Looks more like maybe 2-5 FPS? Also what I didnt really understand is how much of this technology is already deployed in current Tesla vehicles? And will this be part of FSD or part of *any* Tesla? You seemingly showed real-world examples of the car preventing an accident despite the accellerator being (fully and wrongly) pressed by the driver: Has autopilot to be explicitely activated in order to intervene in such situations or is that safety feature automatically always active? Also, in general, drivers inputs are supposed to overrule autopilot - so how in this case can autopilot overrule the driver? Finally, there are tons of these accidents documented on video and still continue to happen, where the driver accidently and wrongly accellerates into an obstacle... one infamous example as of late was the recorded test to show how FSD would run into a child-dummy... which might have been manipulated in a way that it was actually the driver, not FSD, that ran into the obstacle. *If* the car already has automatic obstacle awareness and obstacle prevention, how are such incidents even possible at all? The other way around: If that obstacle prevention is not rolled out to the fleet yet, how and why would the car intervene to prevent an accident in the real word examples in your presentation? Why the persons legs (24:00min) were saved, while the garage (25:00) was not? With great respect, best regards and thank you again.

[@simsonyee]

How does the network account for different calibrations on different cars? The relative poses of the 8 cameras will a little different from car to car and also drift in time and influenced by heat/cold between day and night.

[@switzerland]

I think we need these collision maps at 31:00 for each moving object and then apply these to the cars predicted path. It's mind blowing what our brains are doing. I'm certain we need another FSD hardware generation.

[@op4394]

sometimes our brains are not doing what they could do and be reliable at any time day or night. that's why a highly consistent automation system is needed to avoid all those crashes, injuries and death. new FSD HW4 could provide more crunching power, but given the presentation and the collected feedback from the beta testers, 99.95% is software related, according to James Douma's recent comment four days ago from 1:38:30 in Tesla Bot, AI Day, FSD, AI, AGI, and More with Farzad Mesbahi

[@jo-han]

Everything is a movable object :) Good that will eventually solve for containers falling of trucks, rock falling of a mountain slope, truck or car driving of an overpass, walls of buildings that collapse, etc, etc.

[@suryatamilan576]

தமிழன் ❤️🔥

[@agera12]

Anybody knows if Tesla is storing knowledge generated while driving? Or plans to do so? Like when a new builidng site with a traffic light is opened just behind a curve. We humans adjust our driving according to new situations we encountered in the past.

[@khkgkgkjgkjh6647]

Very interesting. The ego car should assume that every other car is trying to avoid collisions in a similar way, and also assume that every other car assumes that every other car is assuming this and everyone is changing their courses accordingly. It gets pretty complex! Would love to see simulations of hundreds of cars running this and see how they behave.

[@JRao]

This message is for Ashok Elluswamy. - Regarding the Tesla crash that is shown in this video starting at 24:38 and running to 25:14 - The driver was driving in slowly into the driveway and then are you saying the following? - The driver suddenly put their foot fully on the accelerator causing the forwards crash - Then the driver put the car into Reverse and drove it backward until it crashed into the garage?

[@thesteaktc]

Great video. I’m a bit perplexed by the bit at 25:00. The driver unintentionally pressed the accelerator to 100%, first in the forwards direction, and then in reverse. If you watch the video they are reversing at full speed quite a long way down the road before they crash in to the garage. That’s a long way to drive without reacting. I can’t really imagine being in that situation. If my car was going full speed backwards I wouldn’t just leave my foot fully pressed. Seems kind of odd. But then I know there are a lot of odd people out there so I guess it could make sense. Tesla is doing such an amazing job of working to prevent these sorts of things from happening. Incredible how it can detect you are doing something wrong and stop it.

[@TristanCunhasprofile]

Almost always when this happens the person thinks they're hitting the brakes, the car speeds off and so they hit the "brakes" even harder. It's a panic reaction and the driver doesn't realize that they're the one telling the car to accelerate That's why the recommendation from experts for the situation where you think the accelerator is stuck down is to always take your foot off all pedals first, and then make sure you really hit the brakes. But it seems like that's something that's really hard to remember to do when you're panicking

[@bennabulsi6756]

I really hope they have a better documentation method internally.

[@gailalfar9752]

Hello Ashok, Regarding 25:00 where the driver backs into a collision. Is there a way to warn the driver (other than w/ the visualizations) that a human is nearby to walking behind the Tesla? Happens a lot in suburban neighborhoods. I've had people "ask me to ask the Tesla Team" to fix this with a software update. (ie: yes, the human form is visualized on the backup screen, but ppl saying they also want option for an audible warning)

[@LeoAculaMiles]

Tweeting this at the CEO, Elon, might be your best bet. He's weirdly contactable.

[@bobojake6979]

Can't believe it only depends on image supervision. But does it work well on the white semi-truck case?

[@LetterRip1]

Are you still using YOLO after the BiFPN for feature extraction? The illustration of the architecture from the previous Tesla AI Day didn't show it anymore, but it wasn't clear if it was dropped from the architecture, or was present but just not shown on the slide.

[@_baumi_]

20:30 GOLD - right there - forget FSD L4/L5 for now - just licence this got Apple/Google and BOOOOM

[@digitaldreamer8637]

Can’t believe how much inference performance you keep extracting from HW3. Now processing 36fps with higher accuracy and more capability. Insane!!

[@agera12]

25:00 is a case where people will tell the newspapers the cars accelerated itself and brake pedal didnt work - and newspaper will print that BS because Tesla

[@zshn25]

How is the occupancy network trained? Where is the supervision coming from? As was mentioned, NeRF could be an additional way to supervise it but what is the primary supervision?

[@JRao]

This message is for Ashok Elluswamy. - How were you able to determine that it was in fact the driver’s fault? - Has Tesla contacted the driver of this vehicle? - Is there some data that was recorded by the Tesla when the crash(s) occurred?

[@CyberKyle]

Can someone please explain to me the difference between the instantaneous and 2 second horizon collision fields? Not sure what the time means. Thanks!

[@DontThinkSo11]

The 2 second field is saying, "given current velocity and heading, would I be colliding 2 seconds from now if I were positioned at that location?" The goal of the car then is to change velocity and/or heading until the pixel immediately under the car is green.

[@CyberKyle]

@@DontThinkSo11 ahh that makes sense, thank you!

[@geteng3324]

Hi, in the video, how the autolabelling works?Thanks

[@switzerland]

Looking forward when FSD will be able to use trains and ferrys. That will be one masterpiece of a neural network. Yes, there are trains cars can use, similar to ferrys.

[@johngraham3238]

So how does Occupancy Network with Collision Avoidance handle a plastic bag blown by the wind towards the vehicle in the lane in which the vehicle is travelling? Does it move around the occupied space to avoid the “collision”, notwithstanding that damage would be zero?

[@hornetutube]

Well- even *if* the car would auto-slow down to avoid such "soft obstacles", that would be a very small glitch in the grand theme of things. One day the neuronal net could very well be able to identify such typical "soft obstacles" to minimise these occassions (which are already rare in and of itself). But better to be on the safe side than the other way around.

[@IamtheDill]

Seems like a large departure from past approaches?

[@benjesus6571]

I was just wondering if it could recognize bodies of water, like will it stop from going into a lake or over a cliff?

[@t.w.3074]

Would love to hear how phantom breaking plays into this. I took a long trip recently and had 14 occurrences of phantom breaking on the interstate...very annoying and not safe if a person doesn't react appropriately to the false breaking. Does anyone have any insight?

[@Muskar2]

Was it near bridges and/or noisy shadows? Those can trip up the car in my experience - I've not had as many occurrences as you on long trips though - usually it's once per 1000km/600mi on the highway for me and barely noticeable for passengers and other drivers. It happens more often for me on rural roads where basic AP is not designed to be run on. Typically on sharp occluded right-turns and sometimes in forests on clear sunny days (with noisy shadows).

[@ryangerrity2851]

Reset and re-calibrating your cameras/sensors should help.

[@rustyfox81]

At 18:00 it looks like the repeater camera has a FOV of slightly more than 90 degrees, is this the case ?

[@rickkay9548]

fantastic! Meanwhile the amateurs still pre-scanning wonder why they are stuck in chander AZ

[@jiangnan7370]

Train NeRF in single shot? My toy training projects train LEGO model for 24hrs...🤣

[@mfpears]

This will end up being e2e neural networks. Fun visualizations though

[@nonietoomila8890]

30:36 🎉🎉😅😅😅😢😢😂😂😂😮😮🎉🎉😢😢😂😂😮😅😮😢🎉😂❤❤❤❤❤❤❤❤❤❤❤❤❤

[@jacksprat9350]

Does Tesla have collision avoidance while backing up?

[@HenryLoenwind]

General avoidance (like shown in the last clip) needs to take control of the vehicle away from the driver. But as the driver is legally responsible for everything the vehicle does, it cannot be rolled out until the vehicle is licensed to drive itself. Until then, there are only a couple overridable safety features like lane-keeping assist, pedal mixup braking, emergency braking assist and so on.

[@tslaEnglish]

A question: why are these pictures/video used in presentation always so brown colored, like the old movies? Are they processed or as-is-from-cameras?

[@archigoel]

Ya, they don't use full color images as it takes a lot more processing power.

[@kazioo2]

These cameras aren't standard RGB cameras, because they are not optimal for computer vision.

[@moonw0man]

They're 12-bit cameras (HDR is 10-bit) so when they normalize down to standard RGB it looks washed out. They do some processing in-car to make the backup camera look less awful, but don't do it for internal stuff like this. The neural nets see the raw data which has much more detail both in the highlights and shadows.

[@priancho]

Wow! Detecting the extruded leg of a crain car is so shocking ;-)

[@trent_carter]

You should consider using LiDAR or radar to see exactly how far behind the competition is.

[@digitaldreamer8637]

🤣👍🏼

[@Muskar2]

Hilarious 😅 but a bit mean and pointless, I think. And actually, I've seen they have been using LiDAR and RADAR on some in-house test vehicles to test specific things sometimes - in case you didn't know

[@tobiasmuller5435]

@@Muskar2 it is just used as a reference system but for nothing more

[@frangalarza]

An interesting exercise is to see how many views the videos on this channel get. They're all around ~1K except the Tesla ones :D

[@pw7225]

Surprised how low the resolution of the occupancy 3D map is...

[@switzerland]

Maybe it‘s occupied even with small occupation. Then just avoid it, no matter how small it is. You don't wanna be too close anyway.

[@pw7225]

@@switzerland That's a good hypothesis.

[@Factoryseconds123]

What did you expect from 8 cameras running at less than 720p?

[@pw7225]

@@Factoryseconds123 I expected a higher resolution. But I think @Fabian gave the correct answer.

[@JamesAwokeKnowing]

I was surprised how HIGH res it is. And i was right since he said it actually makes lower res voxels and is upscaled. What maybe isn't clear to u is this is all from scratch in 10ms over and over, wheras what you are used to from lidar is built up over many seconds by just remembering how you moved. This is instantly the whole scene. If it builds up over time, it can do that detailed nerf quality image.

[@roberts932]

how about collapsing bridges, mud slides, sudden holes in the road ?

[@peterfireflylund]

I don't think they are going to introduce FSD in China any time soon.

[@roberts932]

@@peterfireflylund yeah, but imagine a bridge collapses in front if you, and one fsd after the other drives into the abyss.

[@ogzombieblunt4626]

@@roberts932 It would do what he showed the car about to drive into the river did. Stop.

[@TheEvilmooseofdoom]

It see things as moving even it doesn't know what they are. It does not see the road surface.

[@MatthiasGotzke]

It would be interesting to find an intersection like kzbin.info/www/bejne/oIGmh2d9gL1setk where the left lane is the end of a ramp/bridge. So that looking flat and straight left would not actually indicate the correct road surface.

[@user-ry3zg5im3z]

is there paper name of this work? occupancy networks

[@olavkokovkin7009]

I remember Elon saying "LIDAR is a fool's errand", and this looks pretty much like "LIDAR from cameras" :)

[@zodiacfml]

nice direction like how an experienced human driver approaches it. however, I do think there is need for multiple solutions depending on driving environment/situations where FSD switches modes dynamically e.g. highway, parking lot, garage, animal/human presence, severe weather..

[@ScuroChiaro]

wow. elon musk hast 100.000.000+ followers on twitter ... unfortunately the TITLE and DESCRIPTION of this AMAZING CONTENT is so incredibly POOR that only 0.002% have found it in the internet so far

[@nonietoomila8890]

0:33 🎉🎉🎉😅😅😅😢😢😮😮🎉😂😂😂😂😂😂🎉❤❤❤❤❤❤❤❤❤❤❤❤❤

[@anonanon1604]

24:39 what kind of drugs do you have to be on to do something like this by mistake???

[@tenzinpassang4812]

#RemindMeIn2030: How much is TSLA worth? 🤑🤑🤑🤑🤑🤑