Tesla Engineer: Tesla's FSD Foundation Model will be WORLD CHANGING... Stay Tuned!!

In a recent CVPR talk, Phil Duan, Senior staff software engineer at Tesla, provides yet more clues about where Tesla is headed with their current Occupancy Networks and future Foundation World models. Along with Ashok Elluswamy's talk and other recent videos (see links below) this talk helps us understand where Tesla is headed in the next 6 months. Hint: it'll be the everything bagel that helps them build a true AI FSD empire!
Join this channel to get access to perks:
/ @drknowitallknows
**To become part of our Patreon team, help support the channel, and get awesome perks, check out our Patreon site here: / drknowitallknows . Thanks for your support!
Go to drinkag1.com/drknowitall to get started on your first purchase and receive a FREE 1-year supply of Vitamin D3+K2 and 5 travel packs.
Get 25% off Blinkist premium and enjoy 2 memberships for the price of 1! Start your 7-day free trial by clicking here: blinkist.com/drknowitallknows
Get The Elon Musk Mission (I've got two chapters in it) here:
Paperback: amzn.to/3TQXV9g
Kindle: amzn.to/3U7f7Hr!
**Want some awesome Dr. Know-it-all merch, including the AI STUDENT DRIVER Bumper Sticker? Check out our awesome Merch store: drknowitall.itemorder.com/sale
For a limited time, use the code "Knows2021" to get 20% off your entire order!
**Check out Artimatic: www.artimatic.io
**Want to get in on WeBull's awesome Crypto and stock fun? Check out this link, and get started trading stock and Crypto!
a.webull.com/i/DrKnow-it-allK...
**If you are looking to purchase a new Tesla CAR, Solar roof, Solar tiles or PowerWall, just click this link to get up to $500 off! www.tesla.com/referral/john11286. Thank you!
**You can help support this channel with one click! We have an Amazon Affiliate link in several countries. If you click the link for your country, anything you buy from Amazon in the next several hours gives us a small commission, and costs you nothing. Thank you!
* USA: amzn.to/39n5mPH
* Germany: amzn.to/2XbdxJi
* United Kingdom: amzn.to/3hGlzTR
* France: amzn.to/2KRAwXh
* Spain: amzn.to/3hJYYFV
**What do we use to shoot our videos?
-Sony alpha a7 III: amzn.to/3czV2XJ
--and lens: amzn.to/3aujOqE
-Feelworld portable field monitor: amzn.to/38yf2ah
-Neewer compact desk tripod: amzn.to/3l8yrUk
-Glidegear teleprompter: amzn.to/3rJeFkP
-Neewer dimmable LED lights: amzn.to/3qAg3oF
-Rode Wireless Go II Lavalier microphones: amzn.to/3eC9jUZ
-Rode NT USB+ Studio Microphone: amzn.to/3U65Q3w
-Focusrite Scarlette 2i2 audio interface: amzn.to/3l8vqDu
-Studio soundproofing tiles: amzn.to/3rFUtQU
-Sony MDR-7506 Professional Headphones: amzn.to/2OoDdBd
-Apple M1 Max Studio: amzn.to/3GfxPYY
-Apple M1 MacBook Pro: amzn.to/3wPYV1D
-Docking Station for MacBook: amzn.to/3yIhc1S
-Philips Brilliance 4K Docking Monitor: amzn.to/3xwSKAb
-Sabrent 8TB SSD drive: amzn.to/3rhSxQM
-DJI Mavic Mini Drone: amzn.to/2OnHCEw
-GoPro Hero 9 Black action camera: amzn.to/3vgVMrH
-GoPro Max 360 camera: amzn.to/3nORGYk
-Tesla phone mount: amzn.to/3U92fl9
-Suction car mount for camera: amzn.to/3tcUfRK
-Extender Rod for car mount camera: amzn.to/3wHQXsw
**Here are a few products we've found really fun and/or useful:
-NeoCharge Dryer/EV charger splitter: amzn.to/39UcKWx
-Lift pucks for your Tesla: amzn.to/3vJF3iB
-Emergency tire fill and repair kit: amzn.to/3vMkL8d
-CO2 Monitor: amzn.to/3PsQRh2
-Camping mattress for your Tesla model S/3/X/Y: amzn.to/3m7ffef
**Music by Zenlee. Check out his amazing music on instagram -@zenlee_music
or KZbin - / @zenlee_music
Tesla Stock: TSLA
**EVANNEX
Check out the Evannex web site: evannex.com/
If you use my discount code, KnowsEVs, you get $10 off any order over $100!
**For business inquiries, please email me here: [e-mail]
Twitter: / drknowitall16
Also on Twitter: @Tesla_UnPR: / tesla_un
Instagram: @drknowitallknows
**Want some outdoorsy videos? Check out Whole Nuts and Donuts: / @wholenutsanddonuts5741
Source video:
• CVPR23 E2EAD | Phil Du...
Other videos in this series:
• Elon: FSD Beta WON'T B...
• ONE MODEL to Rule Them...
• Did Ross's Tesla Blow ...

[@ehhhhhhhhhh]

3:10 The math, if anyone is curious: if you take (1 second)/36 * Speed, you'll find that this means that the car has a new snapshot of the road every 2.8 feet at 70mph. In pedestrian dense areas where you're only driving 15mph, it has a new snapshot every 7 inches that the car moves forward.

[@fluffyduckbutt24]

Damn! that is amazing

[@thesoundsmith]

At 70, that sounds fine. At 15, it's a bit squirmy. Two images to determine a suddenly-appearing little girl is too late, even one is tight..

[@Yahoo886]

@@thesoundsmithtrue but can a human do any better for reaction time 🤔…definitely not an 80 year old driver…not sure about other age groups.

[@89bazoly]

@@Yahoo886some exceptional people might be better. But for sure not in 360 degrees 😊 it is amazing to see some fsd saves when a car is coming from the back and fsd (or whatever the name is for collision avoidance) is accelerating to avoid getting hit from behind

[@kipling1957]

Input from a physiologist: The fastest human reaction times are about 150ms, reflexes half that. There’s a lot of variance beyond central brain processing, including the distance the nerve impulse has to travel in the peripheral nervous system, and then add in neuro-muscular activation times. I’m not a programmer or engineer, but I’m guessing that by comparison FSD reaction times at 36 fps would be vastly superior.

[@tommornini2470]

🎯 Human visual perception alone is generally understood to be about 30ms, this was an early goal of computer system to be able to take a key press and show the output on screen. If you could do it in 30ms, it would appear instantaneous to a human.

[@kipling1957]

@@tommornini2470 Excellent point!

[@davidhawkins7138]

Deconvolutions are exactly what AI chips are designed for. I was concerned that full-feature FSD wouldn't fit on hardware 3 a year ago, but this architecture makes the inference engine running on the car very workable. It's a brilliant and robust solution. Heuristic-based solutions being used by competitors can't scale the way this will.

[@jonbowes5999]

You appear to have a solid grasp of the technology! Yes almost any complex task that is moved from Heuristic code to Neural Network vastly reduces it's execution time (inference vs classic code execution) thus reducing load on the non--NN portion of Hardware 3 however it will also ADDs to the queue of NNs for inference thus increasing load on the NN portion of the hardware, this is a constant juggling game for Tesla. EDIT: I would add that this is further nuanced by the fact some inference can be performed (mush less efficiently) on the non-NN portion whilst no heuristic code can be processed in the NN specific hardware.

[@Supreme_Lobster]

@@jonbowes5999 on the other hand, heuristic code can often be approximated by NNs, and if done well it can perform better due to generalization (or interpolation at the very least)

[@thesoundsmith]

Elon's systems architecture has always been designed for scalability and adaptability. That's why he is so much further ahead than his competition realizes. Ford builds cars. Elon's not building cars. He's building a system to Terraform a world. Cars are one piece of that. I just hope he doesn't go irretrievably off the rails in the process...

[@bluetoad2668]

I think the ability to have something like full level 5 autonomy is directly related to how accurate the computer model in 4d represents the actual world. Humans will make mistakes at points where their internal model doesn't match reality too, often they can recover but sometimes an accident of some kind happens. Self driving cars will be the same, except those accidents should be less frequent and less severe by at least one order of magnitude. So infrequent, in fact, that they will probably make the news. One advantage will be that for any accidents we'll have a lot of data.

[@lewiswithrow1936]

Very useful, John. Thank you for helping me to understand a little better. You provide great value.

[@z4zuse]

They could gamify the scene data collection. Encourage you to drive over less travelled roads, or allowing a 10% longer drive by letting the car navigate through a detour.

[@tommornini2470]

And pay for robotaxi rides once we’re at level 5. They’ll want to “refresh” their world view on a regular basis to handle new roads, detours, pot holes, speed bumps, etc. - even on rarely traveled routes.

[@robertweekley5926]

On that note, what does it do, if only "One" car drives a certain area, once, and it is the First time, on a non paved road? 🤔😳🧐

[@tommornini2470]

@@robertweekley5926 Easy! It drives the same way it does today, slowly and carefully, though more experienced in driving than today.

[@bluetoad2668]

​@@robertweekley5926it relies on its generalised training - in a similar way you do when doing a similar drive in an area you've never seen.

[@FrunkensteinVonZipperneck]

Comments are thought-provoking. I’m quite impressed! ❤

[@EIYEI]

Great summary , with outstanding insights. Please continue your work. Kudos

[@gaussian3750]

This video is too impressive! Thank you Dr. Know-it-all

[@davidhawkins7138]

It looks like the vector space is many more than 3 or 4 dimensions. They must be encoding many other attributes in the space for the deconvolution to generate useful inputs for later steps in the process. This is a powerful idea because it would be relatively easy to add other dimensions incrementally as they need them to enrich the data to support different queries.

[@skierpage]

The latent space is a million or billion dimension vector. Tesla doesn't choose the representation, rather the neural net builds it during training.

[@davidhawkins7138]

@@skierpage What you say is true. However, each of the heads used to extract features from the latent space does this by filtering out unnecessary dimensions. These heads are also trained, so their internal representation isn't known, but the result is a deconvolution. It isn't necessary to understand how the information is represented in the network for this to work. Extracting the necessary features is part of the training.

[@jonbowes5999]

in place of dimensions I would prefer to use the term "input data" all dimensions are simply input data to a particular network... For example Tesla removed some input data from the perception network when they removed radar. To add radar back in would require complete re-training with verified radar data for every training clip. The reason radar was removed was simply because the efficacy of that particular input data was too low, if and when high definition radar is available it may then make sense to reintroduce high definition radar data. However ALL training clips would then have to also be accompanied by verified high definition radar data. To achieve this Tesla would either have to acquire new training clips from vehicles with high definition radar or somehow synthesize the high dominion radar accurately for ALL existing training clips, with both options full re-training would be required.

[@geoyoshinaka5251]

Thanks for making a complex topic accessible to us, feel better soon!

[@FrunkensteinVonZipperneck]

Can you imagine Jim Farley & Mary Barra watching - and comprehending - Phil’s explication? Me neither.

[@pauldobson9379]

That was awesome. Thanks!

[@oddvarsand5568]

Great Value John! We need you to translate the computer code lingo to something that we can relate to, please keep it coming 👍👍👍

[@thesoundsmith]

And ALL this data can be easily added to things like the Boring Machine and even the Starship conceptual space. Foundation and Empire indeed! 😎 And here I thought a Transformer would be a Model S Plaid that could become a Cybertruck! 😄

[@curtisyoung7107]

Beautiful presentation on rhe future evolution of process control that extends much further than FSD, which is the early start of air, water and space navigation systems. Looking forward to see how this tech will be used in Giga factory production to build the Tesla mass market target vehicles and CyberTruck releases. 😁

[@robinsoncrusoeonmars8594]

ON some of the visuals there are a lot of shadows draped over the curbs. Even for my eyes it is hard to see the physical curb. Without LIDAR wonder how they can determine the curb location? Might be missing something but in general wonder how shadows are handled? Well done presentation on this topic!

[@sparkysho-ze7nm]

How many 50 cents word are being used below by MOST commenters . I love it how everybody is got or ordered a Tesla literally Amazing

[@helmutshotthesheriff1942]

Good video ❤👍

[@shadylane99]

Constructive: Every gamer knows empirically that we see more than 30 FPS.

[@CYBRLFT]

That’s not what he meant. He was saying our baseline is lower. Think of it like our entry frame rate interacting in the real world, not our maximum.

[@XeonSX]

@@CYBRLFThe’s just wrong. When I drive I make a decision as soon as I see things… I see things way faster than 1/12th of a second. There isn’t a way around that.

[@Maserbeam]

@@XeonSX i believe everyone in this thread is technically correct, just not explicitly explained at all. Dr Know-it-all is loosely saying that at a certain point, we stop seeing the frames as "frames only" and moves into "motion visuals", meaning that instead of our brain interpreting what we see as separate images (we see image 1: it is a brown box, and then image 2: the brown box is now somewhere else), it now considers the visual input as "something is moving" (enough images that we consider the box moving across our screen instead of being different images). this does not relate to being able to see frame rates beyond 83 frames a second (1/12th of a second). this can differ from person to person. i know people who have convinced themselves they can't see a difference between 30 and 60 fps, let alone from movies' 24 fps. and then there's people like you and me who have monitors that have high refresh rates.

[@brunoheggli2888]

Wow this is amazing,the ioniq6 is realy the best,game over for Tesla!

[@FrunkensteinVonZipperneck]

28:30) “gamification” is the singularity. I get in the robotaxi at the airport & FSD chooses the looong route to jack up the fare.

[@tonyfromaus7855]

Hey Doc or anyone that knows…. have Tesla said anything about the perfectly smooth path floor in the occupancy network? In the real world a shovel falls off a truck, a pothole opens up, a piece of the mountain road is washed away. I assume that a higher semantic layer will eventually recognise those things but how close is it to happening? Rail tracks, speed bumps and potholes etc are poorly handled so far…. hopefully that will improve. Cardboard boxes etc seem to be completely ignored despite having significant damage possibilities. Anyone know how many versions till obstacles like that will be handled?

[@camielkotte]

Nice one. Isaac Asimov.

[@Marc_de_Car]

tHANKS

[@Supreme_Lobster]

At this point I strongly believe the main problem is no longer perception but control and planning. Generative models will probably play a big role in the coming months to handle this. Very interested to see how they make this fit in the available compute.

[@ArendJanV]

What is the npv of this per share?

[@damfadd]

Dan O'Dowd need to see this

[@tommornini2470]

Dan O’Dowd is not interested in the truth, his entire concern is based upon no human involvement. He should be equally concerned about traditional and adaptive cruise control.

[@FrunkensteinVonZipperneck]

@@tommornini2470The guy is driven by hate. Emotion blinds cognition.

[@knpstrr]

And Waymo/Cruise is just getting the same locations over and over again. No Waymo/Cruise in my small town but there are a few Teslas.

[@FrunkensteinVonZipperneck]

My town is 550 humans (and me), 200km from a city. Tesla installed chargers on Main St 3 years ago. Folks started buying EVs…

[@anonmouse956]

Are you saying "box cell" when discussing the AI graphics?

[@bluetoad2668]

After update 2023.20.6 I was asked to agree to a whole lot more data feedback that before, which I did. I am in Europe, have a model S with hw3 and have paid for the FSD capability (I don't have beta due to location of course) maybe this means they are ramping up the data collection from hw3 cars running fsd in shadow mode in Europe??? (I am in an LHD country, so not UK or Ireland)

[@Dr_PhilK]

Dr,need to show me and guide me how to be investor with tesla or others..which platform to use etc..which one do you recommend? Please please..thanks

[@rogerforsman5064]

Buy my beach bungalow in Arizona! It is a real bargain!

[@fluffyduckbutt24]

At 12:00 on the CVPR23 E2EAD video I found two things interesting. The first that people in SF understand the topagraphy is crazy there and is no where else. The second was "Remove the flat world option to improve our driving smoothness." How does that work?

[@tommornini2470]

Doctor describes the multi-pass scene construction as “high resolution.” I’ve been using the term “medium resolution” to differentiate this technique from LIDAR based solutions, because their resolution is between Google Maps and LIDAR, and very likely far richer (higher dimensional) than both.

[@robertweekley5926]

Obviously, in high Tesla Density areas, more Tesla's will traverse the same areas, more times, at both the same time of day, and at different times of the day, and in direction of travel! And, one would think Cities have more "Variables" than some Back Country Dirt Road, as to changes due to Vehicles on the Road, potential Construction & Detours, as well as Pedestrian Traffic, but a Back country road might have more "4 Legged Critters" to manage!

[@markschneider8187]

Would love to see FSD drive a road racing track like Elkhart Lake in Wisconsin, Mid-Ohio Sports Car Course, Sonoma Raceway etc. Curious how it would handle speeds and turns if given no limit on speed and understanding that it is not a street. Could it learn to drive the track like a human would learn it.

[@GrigoriZhukov]

I'd rather see it on the ring or IOM.

[@mnhsty]

I suspect it is too constrained to do anything terribly impressive time-wise. But then so am I.

[@mariusm62]

The perception side is good. The only thing that bothers me about Tesla's architecture is that so far very little has been said about the planning stage, as well as the method by which they plan to resolve edge cases. Because if going by Chuck's UPL, that is going to take a lot of resources and time to solve enough edge cases to achieve al least level 4 autonomy.

[@beilkster]

The subtle plan ive gathered is that they will have level 3 capable vehicles for a year before they assume responsibility for level 3 (call it level 2 even though it meets level 3+). Same for level 4 and 5 over time. This plan allows for public trust to build before moving to the next level

[@mariusm62]

@@beilkster Sure. But this doesn't answer how they will solve the remaining issues with FSD, they haven't presented a plan as detailed as they have regarding perception.

[@tommornini2470]

They’ve actually spoken at great length about their planning techniques in previous AI days. It’s a multi dimension decision limited first by plan achievability, and then sorted by a combination of safety (priority), overall traffic flow, passenger comfort, and efficiency. Those talks were presented when the planner was procedural code, but 11.4.x has begun moving planning and execution into neural networks. I suspect those networks are and will be trained from the procedural code providing ground truth on the backend until the auto-labeler is superior to the procedural code. What edge cases are you concerned with? In driving, edge cases often result in accidents, so I’d hazard a guess that your edge-case concerns are equivalent to their “long tail” data collection advantage of a fleet of several million (and growing quickly) cars on the road.

[@mariusm62]

@@tommornini2470 The edge cases I refer to, are handling of emergency vehicles and personnel, construction scenarios and asociated personnel, proper roundabout handling, low visibility signs and stop lights, railroad crossings, and hundreds more

[@tommornini2470]

@@mariusm62 Yes, all necessary for level 3 and higher certifications. So what? They’ll soon have MILLIONS of vehicles, increasing at 500k/quarter, collecting data on your hundreds of edge cases.

[@garycarson3128]

Hey John, the Tesla Bot doesn’t have to make decisions about the physical environment moving at 60, 70, or 80 mph, therefore maybe it doesn’t need Hardware 4 as much as a car could benefit from Hardware 4. If Tesla is planning to produce millions of Bots annually, a cheaper Hardware 3 that consumes less energy than Hardware 4 might be the most cost efficient approach.

[@chidude]

I guess the Tesla Bot probably won't be designed with the capability to run super fast like the 6 million dollar man!

[@jimcallahan448]

I am not sure I understand at what point (if any) they recognize objects? It seems to me that a major point of the occupancy network was how far one could get by just mapping occupied space cubes (or other 3-d shapes). How does one go from occupied shapes to vectors without some object recognition or at least image segmentation?

[@tp_123]

I think in the video where he talks about "Surface Outputs" at 19:30 where they train the NN with "supervised" learning from labeling of data. Like for driving, road signs, road markings, etc. For a robot in an warehouse, the training may be different and specific to the needs of a warehouse. Or at home it may have specific training for in home stuff.

[@cook6052]

Dr KIA, interestingly, in EAA astronomy (Electronicly Aided Astronomy) we use image stacking via software, to paint, reveal dynamically in realtime a clear image of say a galaxy 1,000,000 light years distant on a monitor. It is an amazing process that has been used by professional astronomers for over a hundred years. Its called stacking. Stacking is used due to the fact that peering through the atmosphere is like a swimmer looking up through the water at someone standing by the pool. The image is convoluted due to undulations, moisture, dust ect. Stacking allows for capturing clear components of each image to create a total image much more complete. Its all magic to me yet very intriguing.

[@companyman7128]

That's approaching "Computer - Enhance!" movie trope territory here.

[@tommornini2470]

Tesla FSD does stacking as described in two ways: 1) multi-frame, 4D 2) eventually, Tesla only maps built from the fleet.

[@ksrithan]

I like the Second Foundation more.

[@curtisquick1582]

So, would the Second Foundation also be at Hawthorn, only hidden?

[@Preciouspink]

Hardware 4 builds the suite of FSD parameters to make Full Self Driving to be able to run in hardware three technology…

[@craighermle7727]

Frame rate and ability to make decsions are 2 entirely differnt things aren't they? If the machine or the person doesn't understand a specific circumstance or how to resolve it, it's a problem, correct? Computerr are great, but sometimes, it takes a human to make sense out ok kaos. Neil Armstong being a pretty good example.

[@sparkysho-ze7nm]

Th complexity of reality ; when reality is not deterministic justa bunch of complex possibilities

[@FrunkensteinVonZipperneck]

High Probabilities are resolved. Low probabilities- “impossible” edge cases- are merely late.

[@bluetoad2668]

6:00 I want to see a comparison with Mobile Eye not Cruise! Mobile Eye claim to have a very large fleet collecting data.

[@DanaOredson]

About every ~2 years, Tesla changes its architecture. Maybe this will be The One.

[@craighermle7727]

with or without HW4?

[@DanaOredson]

@@craighermle7727 I have no idea, but it will of course be easier to run in HW4, as would anything.

[@Starckgazer]

What does high, high dimensional mean, Dr?

[@jonbowes5999]

We should also be aware that Tesla can and do collect data regardless of weather a particular vehicle is operating with autonomy be it FSD, AutoPilot etc... Tesla can harvest data from any Tesla that has the hardware installed. It is therefore likely a gross under estimate of the available data miles to restrict that only to FSD miles driven! If we think Autopilot is in say 1 in 10 vehicles then the available data capture could actually be 10x FSD miles driven...

[@WarrenLacefield]

A very interesting 4-video series. Foundational models remind me of Ender's Game. But in another direction, ChatGPT is text-based (or spoken). We read text. Readability analysis is a method to determine the difficulty of books and various texts. A colleague and mentor of mine, Dr. Earl Rankin, is known as one of the "fathers" of the "Cloze procedure" (i.e., predict the missing word) for determining human levels of literal understanding of reading materials. The College Board's "Degrees of Reading Power (DRP) test" is based on this. A research goal was calibration of reading difficulty and ability on the same measurement scale, so that text books and articles in college could be compared to students' ability (to read with at least literal understanding). If Optimus is to benefit children or ChatGPT can be of help in K-12+ education or elsewhere, they will need to recognize and use a level of language appropriate for the recipient. GPT is the foundational model for that. Maybe they will be able to detect that and adjust after a few minutes of texting or conversation.

[@ricksyap]

So 1 week's worth of collected data from Tesla is equivalent to 3 years on Cruise & 28.8 yrs on Waymo!?!

[@akira28shima32]

I put my money on Optimus handing out the first keys to Cybertruck delivery. What say ye?

[@BradAkersphotography]

Thank you for the enlightening video! I do feel the ARK slide with time/1M is sus! To compare Waymo’s “public rider only”miles! Just compare the autonomous miles? This skewed the data! Waymo’s doing 65 mph in a class 8 truck. No way Cruise collects more miles doing 45 mph max in a bolt getting 100 miles of range that charges at 50 kw. public driverless miles and supervised test miles is Not apples to apples. They limit Waymo to “public only” that excludes the trusted tester and all other test miles! Why not state Cruise’s public rider only miles? Trust me its not even close! Cruise is pivoting to delivery and ADAS systems. We can agree Tesla is the most innovative scalable system and has the unit economics to win💯

[@grahammewburn]

I'm not financially able to buy a new vehicle. I will use Robotaxi. I will avoid the need for insurance, government fines, parking fines, all the expenses associated with car ownership etc. Yes, Tesla FSD will radically alter the world. Cheers Gray Australia

[@user-qg5qb7cg4p]

Does Tesla FSD sense the road surface condition and reflect it in driving? "For example, if there is a hole in the road, is it possible to avoid it?

[@brunosmith6925]

I believe that if a particular Tesla senses a road problem, this is put transmitted to the database. Other Tesla's that use that piece of road then "know" that the obstruction/problem is there, and act accordingly.

[@davidhawkins7138]

@@brunosmith6925 I'm not sure if this is working yet, but it is certainly in Tesla's plans. This feedback would have huge value for road maintenance (filling a pothole early is easy and cheap), traffic flow (feedback to traffic control systems and onboard congestion mapping for other vehicles), and even accident reporting (i.e.: automatically upload video of accidents seen by the car). Some of this data is valuable enough to provide another revenue source for Tesla.

[@rickkay9548]

I have seen massive pot holes avoided and the car just drove around it. Still needs some work to make it more consistent though

[@tommornini2470]

They can detect such things, but it currently doesn’t appear to react to it. There’s zero reason to believe it won’t in the near future. In this video see the portion about “surface details.”

[@datamatters8]

@@tommornini2470 I agree. Road debris on the freeway like shredded truck tires still relies on the human supervisor to avoid. At 70 mph there is not a lot of time for FSD (or the human) to detect it and react but I really don't know. The higher res HW4 cameras should help though. Still there is always the possibility of vision being blocked by a vehicle ahead of ego which swerves out of the way at the last second. A good reason not to tail gate at 70+ mph.

[@polishfish]

Love Asimov!

[@fjalics]

The number of recent miles is very important, but it's not everything. Waymo and Cruise don't drive everywhere. Tesla wants to find all the edge cases. Teslas go more places. If they start shipping the cybertruck, they will go even more places.

[@tommornini2470]

If they ship the CyberTruck? That was an unlikely but reasonable concern before they started popping up here, there and everywhere recently.

[@jonbowes5999]

"Waymo and Cruise don't drive everywhere" because they cannot drive everywhere! That throws a very bright spotlight on the key limiting factor of any geo fenced system. This not only reduces exposure to edge cases in the wider world but simultaneously reduces the TAM of the product so drastically that it becomes financially unviable. Coupled with this using expensive and redundant sensor suites further erodes any potential commercial gain. Tesla on the other hand have managed to produce non geo fenced general capability with the minimum cost sensor suite and funded this with payments from customers using this for driver assistance capability. No customer has been sold autonomy leaving Tesla free to further profit once autonomy reaches sufficient ability.

[@tommornini2470]

@@jonbowes5999 To be fair, Tesla had NOT yet produced general autonomous functionality. Let’s keep it real. 👍🏻

[@jonbowes5999]

@@tommornini2470 Tom what is your definition of general autonomy? For me the general part is location agnostic, FSD does not care where it is driving and therefore by that definition is general. As for autonomy Tesla has not sold an autonomous system yet, however what they have produced can work autonomously if Tesla was ready to allow that. It is therefore autonomy capable. You will no doubt argue it isn't ready yet, clearly this is true I believe this will be addressed by substantially increasing the training data volumes. Those training data volumes currently constricted by available trainging and labelling compute. Tesla will 100x that capacity over the next year!

[@tommornini2470]

@@jonbowes5999 We don’t disagree on likely outcomes, just whether FSD Beta is currently autonomous.

[@RPHelpingHand]

We don’t need HD maps. We have a vector space ground truth. See it’s clearly a different name…

[@fredhearty1762]

Once you have the Foundation World model -- all vehicles will be driving in the same world (vector space) -- then communication between vehicles becomes viable. An emergency situation seen by one can be broadcast 'locally' like an emergency vehicle siren is local. Ultimately, when the population of equipped vehicles is high enough, vehicles could send intentions to each other and avoid 'surprises.'

[@tommornini2470]

There is zero need to have v2v communication, and I’m willing to bet it will never happen, or not until non-autonomous vehicles are not allowed on roadways. Even then, it introduces a new complexity, communications failure. Each car can see all the cars around it and directly determine the vast majority of what v2v could communicate in any case. I believe people’s desire for v2v is mostly a byproduct of the idea that centralized planning is more efficient than decentralized solutions.

[@fredhearty1762]

@@tommornini2470 There was zero need to have portable phone communications or links to the internet... global internet coverage, or self driving cars. A bit ago, there was zero need to have an auto-mobile.

[@tommornini2470]

@@fredhearty1762 And today there’s zero benefit of having v2v communication. You’d have a ton of systemic instability as the protocol(s) would evolve over time. Putting it together and rolling out the cars with it would dramatically delay the move to autonomy for very little benefit.

[@health101DOTorg]

36 FPS is the video frame rate. This is different from "Compute Rate". 24 FPS is film, 30 FPS is TV video, and FSD cameras will be 36 which is an improvement over standard video FPS.

[@Alarix246]

5:13 the speed of Tesla: we've grown to expect that the purpose of competition is to ensure they have a lead, and the competition always follows in your heels and pushes you forward. Once it is A in the lead, then B, then C and perhaps A again. A constant race. But can we apply the same logic for Tesla? No way! Tesla is a "runaway technology" imho. They are uncatchable as long as their miraculous working ethos continues. Perhaps what we see from the competition and media (lies, misunderstanding, scorn, underestimating) is a reflection of what a typical human mind does when it encounters something out of the ordinary? Perhaps the world at large cannot comprehend Tesla? What have I just said? Perhaps in Nikola Tesla's time he was impossible to comprehend as well. If Elon Musk was able to re-create some of this incomprehensible genius, then it is very fitting he chose Tesla name to honor the originator's brilliant mind. I always think (I only imagine) that perhaps when Elon was young and wanted to achieve Tesla's greatness, his mum told him: well it's all very nice, but do you realize he had died of hunger and penniless? If you want to succeed in technological progress, you have to study also the economy in order to financially survive! And in a stroke of luck, this little boy exceptionally heeded his mum's advice, and that is, my dear children, the end of tonight's fairy tale!

[@justingregoris3164]

How come nobody talks about operating FSD in heavy rain or heavy fog or heavy snow in States and provinces that have extended periods of adverse weather conditions?

[@wr2382]

The Tesla dev discussed this in the Q&A section of his talk.

[@justingregoris3164]

@@wr2382 oh okay I did not know this. Are you able to send me this information?

[@wr2382]

@@justingregoris3164 He just said that it is very difficult compared to sunny conditions and it occupies a lot of their time.

[@justingregoris3164]

@@wr2382 so then how will full self-driving truly advanced and sales increase if this software is not able to operate effectively in adverse weather?

[@SirHargreeves]

Those conditions make it hard to humans to drive. It will likely have to drive slower, like us.

[@justinjja2]

Passively watching a video at 24fps is fine. But when we are controlling something (a car or a video game) 24 fps is too low.

[@tommornini2470]

No, it’s not. Some others gave the numbers above. Human vision is 30ms, response time is 150ms at best.

[@justinjja2]

@@tommornini2470 Humans aren't binary. You can can see a flash of light less than 1ms if it's bright enough. I can easily tell when my phone is running at 60hz vs 120hz just scrolling up and down in these comments.

[@tommornini2470]

@@justinjja2 Yes, I agree, but that’s far different than reaction time to visual stimulus. It’s been measured, carefully, for a very, very long time by many, many people. The cameras have far faster detection, and can accurately measure the length of time of a flash, regardless of the brightness. Human eyes cannot.

[@FrunkensteinVonZipperneck]

The key adverb is “passively.” The transformer is active.

[@Ant3_14]

It sounds like they first do occupancy network that determines positions of things and add layers for cars, VRU's, paths, roads. Later you don't care about most of space and leave only important data that goes to NNs and determines actions of ego car.

[@tommornini2470]

They’ve said that occupancy is enough to guide the car in an emergency situation, so it may eventually shut down post processing once it realizes it’s in a dire situation.

[@MARILYNANDERSON88]

If it operates 100x faster than humans, does it make mistakes 100x faster, then can it make and correct errors 100x faster

[@Preciouspink]

Possibly but it probably modeled situations that fast and taking action on the probable outcome of a particular action. Either way, we can use all that compute power to further the goal of fsd

[@bru512]

This graph looks impressive for Tesla, but I would argue that Cruise and Waymo are collecting nothing useful to solve autonomy and build a profitable business. Within 12 months I predict that both Cruise and Waymo will give up on this approach.

[@tommornini2470]

I doubt they’ll give up, they’ll map out the areas where robotaxi rides are very dense, and it’ll be an 80/20 situation. Not sure why anyone would want an 80% solution, but fixing up at this point would be the equivalent of corporate suicide.

[@bru512]

@@tommornini2470 I can't disagree with your reasoning. It's hard to give up on sunk costs. My reason for saying that they will give up this approach is that it will become very clear to those outside the Tesla community within 12 months that Tesla is on the right track. It's clear to me as I supervise FSD driving me back and forth to work every day.

[@tommornini2470]

@@bru512 We agree on what should happen, disagree on giant corporation’s ability to admit defeat so late in the game. For their sake, I hope you’re right.

[@FrunkensteinVonZipperneck]

@@tommornini2470You’re both being logical - not LEGAs strong suit. Cash will determine when Cruise is abandoned.

[@faramarzmokri9136]

Without a question FSD software and hardware architect will be used by ALL EVs in the future. No one buys any EV without FSD capabilities. It might even be required by all insurance companies for discounted rates. IMO.

[@craighermle7727]

Define "the Future" Musk was said FSD , note lack of Beta, would be available in 2019.

[@FrunkensteinVonZipperneck]

Not “required.” But priced. $100/month IFF FSD; $1000/month for non-

[@FrunkensteinVonZipperneck]

@@craighermle7727Back to …

[@z4zuse]

The blue section has virtually no FSD details. It is generic, and I understand they can use it for the robot

[@Kenlwallace]

36 frames a second is 28 milliseconds vs about 200 to 300 ms for humans

[@stephenboyington630]

FSD: never more hype over something that will never amount to a hill of beans. If I were Tesla, I would make it sound like it will generate profits, too. What better way to pump the stock? FSD is the new Blockchain.

[@brunosmith6925]

Pigeons' frame rate is around 200 per second...

[@tommornini2470]

So? Humans reaction times are 150ms or higher, and pigeons don’t drive.

[@gretco1]

Elon Musk for King 👑🤖 King of the World 🌎🌍 Musk-eteer 🦮🚀 SpaceX 🌌🚕 Mars-bound woohoo 🤚 Go Tesla woohoo 🎉

[@tribalypredisposed]

I just don't understand why Waymo and Cruise are still open and burning cash. Their only pathways to profitable operations start with "Tesla completely fails to achieve level five FSD, and then..." The Waymo and Cruise people have to be smart enough to watch the same things you are watching and come to the same conclusions, with the additional conclusion for them that their companies are doomed.

[@wreckinball11]

I understand this is a difficult task, but with millions of cars inputing data I would think it would be solved faster.

[@tommornini2470]

They’re *just* getting to the point where data is the limiting factor. They’ve had to write procedural code to bootstrap the driving ability as they developed the knowledge to get where they are. V12, with end-to-end neural networks, will have the continuous flywheel described over the years.

[@Rsmith420]

What circumstances do you have to compare in order to make the claim that “I would think it would be solved faster”?

[@jonbowes5999]

This is an interesting question, the answer lies in the latest posts on twitter from Tesla AI... Tesla are accelerating their acquisition of compute in the shape of Dojo and other GPU based systems. WHY are they doing this? The answer is they have too little capacity to train their models currently. What this means is they are having to restrict the quantity of training material in each training cycle because they simply do not possess the compute to train using all the data they would like to use in a reasonable timeframe! For example FSD currently appears to be restricted to a relatively constricted elapsed time horizon for planning. I would guess they would want to extend this however longer video training clips require more data and thus more compute to train into the models... Then there is the question of how many clips can be used for each model? We see in the release notes for each version that they are constantly juggling so many extra clips for A or B situation. With the current limitations they will likely be training as many clips as the current compute can cope with and so when adding clips to one problem they will have to remove some from another... The cure for this is to 100x the available compute over the next year! This is why the building foundations and frame analogy is so apt the foundations are the models and the training will build the frame up as compute allows...

[@craighermle7727]

@@Rsmith420 Well, there's Elon Musk, clearly stating that FSD without BETA would be rolled out in 2019.

[@craighermle7727]

@@jonbowes5999 Don't forget the inputs devices, which are being upgraded to HW4. I think that aspect is being totally over looked. To have a system, whick is not upwardly compable says alot doesn it?

[@mnhsty]

FSD baffles me. The whole purpose Is to let you disengage mentally from the piloting process, at least for the most part. Yet somehow the driver is expected to be responsible to recognize and be immediately alert to dangers than FSD can't handle properly. So the occupants may be safer in expected conditions, but much less safe in a black swan event. Not sure this is a major advance.

[@singularity844]

Humans see well over 100 frames per second.

[@tommornini2470]

This is not entirely true. 30ms is approximately the time casual observers describe as “instantaneous.” And visual stimuli to physical reaction is 150ms or greater.

[@FrunkensteinVonZipperneck]

Glad I ain’t human❤

[@singularity844]

@@tommornini2470 that’s the wrong measurement to make. Humans can easily tell when an iPad is scrolling at 90hz vs 30hz. Reaction speed is not relevant to this discussion

[@tommornini2470]

@@singularity844 Reaction speed isn’t relevant to autonomous vehicles?

[@singularity844]

@@tommornini2470 course it is. Just making a tangential point. Reaction speed gets solved via prediction.

[@Andy-sj2hl]

Gotta disagree with your comment at kzbin.info/www/bejne/gXvZn2ubqLWmsLc . The non-ultrasonic cars definitely cannot currently park themselves. Auto Park is completely non-existent, and if you were to try and park purely using the Park Assist view in these cars you will definitely have a collision in a relatively short space of time. Tesla need to get this feature up to parity and make it a useful tool rather than a novelty tech demo.

[@taijistar9052]

Why he got to talk about this as if he developed this? In effect his boss already talked about this at the investor day meeting?

[@tommornini2470]

Because they’re recruiting the best engineers who always wish to work with the best engineers. It’s not as though you can go from zero to aha! without millions of data collectors, and deep understanding of every little box on those diagrams, and the training scaffolding and hardware, which none of their competitors have.

[@1bdollars]

How does a scam become the top comment here?

[@Starckgazer]

Can wehaveavlozsary of terms, many, many of which Iforoneam unfamiliar!

[@doodlebug1820]

We can do all this but we cant remove scam comments from youtube videos

[@davidking1138]

2 things. 1. China will never allow FSD data out of the country. 2. The Pentagon will want the FSD data from all countries and then if they wish to invade send autonomous armoured vehicles.

[@guslevy3506]

None of this is relevant. Evil martians have been monitoring Earth from the dark side of the moon and will destroy humanity soon…

[@tommornini2470]

Which is why Tesla built, or is building, a data center in China. As for what the governments of the worlds do, se la vie.

[@FrunkensteinVonZipperneck]

On 9/11, “The Pentagon” did not have a fire truck to respond when Saudis bombed E-Ring.

[@Dr_PhilK]

1st to watch this😂

[@rogerforsman5064]

I believe it when it is tested by a non idependent source!

[@davidhawkins7138]

That's what beta testers are. It looks like there are currently about 400k independent sources testing now.

[@tommornini2470]

Skepticism is healthy, but not seeing clear evidence of rapid progress is bias.

[@rebelinthef.d.g.7245]

After years of FSD development, it still does stupid things like turning into wrong lanes, towards a ditch, or stopping when it shouldn't. I saw a video - I believe it was from dirty tesla - in which FSD, Waymo and Cruise were compared, and to be honest, I was more impressed by Cruise. He sat in the back seat and had a beer while the car did the driving. Granted, it's starting off in limited areas, but so did Google Maps. Now look at it.

[@tommornini2470]

The Waymo/Cruise solution can’t scale quickly enough to make a dent in the world, and requires too much energy to be effective in an EV, and too expensive to be in a consumer product. This is because they’re in the last stages of replacing mature procedural code with immature neural networks to escape the local maximum the procedural code is trapped in. Hold onto your hat, the improvements we’ll see over the next 15 months should be shocking.

[@blengi]

it's interesting that a human can learn to drive competently and generalizably after a few 1000km practice on the road yet FSD needs 100s of million of km to do similar. Kind of implies the foundational vision model of human from just ordinary real world existence creates a latent space which can do the driving bit tens of thousands of times more efficiently. Maybe FSD should just generalise to the real world full stop and add a driving layer after the fact.

[@litestuffllc7249]

Sure; I'll believe it when FSD can recognize a police car trying to pull it over; or a hand signal from a traffic cop. FSD so far is super lame with the level of cognition of a 3 year old.

[@sumofbitch]

So your fsd is gona brake laws. Priceless

[@litestuffllc7249]

@@sumofbitch not my fsd; it currently doesn't recognize any of those now - so a police car can't stop it unless it gets in front and slows down; it can't read hand signals at all and probably never will. The list of serious problems is long. It can't navigate in a parkng garage; it can't pass on a 2 lane highway.

[@tommornini2470]

It’s called bootstrapping. You can’t get the cognition of a 16 year old without being 3 first. 😂

[@tommornini2470]

@@litestuffllc7249Can you not understand that this is a matter of training of “edge cases.” 1. Drive around without hitting anything. 2. Collect data. 3. Learn, understand, improve software. 4. Train neural networks. 5. Rinse and repeat. They’ve spent years of time on 3 & 4 and the rate of progress is rapidly increasing.

[@sumofbitch]

@@litestuffllc7249 neither can my wife she still drives

[@litestuffllc7249]

Oh Frunkenstein; so FSD could pass a driving test huh? I didn't know it learned write; thats news. Could it drive as good as a Novice driver? No, a novice driver can navigate a parking garage. A novice driver could understand handsignals Oh and the verbal directions of the person giving the test -If the test giver says; put on your brake; FSD would do nothing - FAIL. If the test person said turn left; FSD would do nothing - FAIL. FSD can't hear ; it can't read. FSD would not pull over if an emergency vheicle was coming - FAIL. FSD wouldn't stop if a huge yellow bus had its flashers going and its stop sign out - FAIL. Your theory that edge cases are not tested is wrong. Edge cases include obeying the posted speed limits FSD - mostly fail. Vision is tested FSD FAIL. FSD can't see far enough to determine if an on coming car is going to kill you on a two lane highway - go ahead and try it; make sure there is an oncoming vehicle 1000 ft away coming at 60 mph.

[@tech-utuber2219]

Too many examples of enthusiastic Tesla fans who are completely lacking in objective understanding that a Robotaxi fleet system should not only be able to accurately represent 3-D space, it MUST also have the additional capability of understanding and prediction regarding human behavior. Optimus even more. None of that is an evidence in these presentations.

[@datamatters8]

Interesting claim re: "understanding and prediction regarding human behavior". Please elaborate. There are a large number of pedestrians and bicyclists injured and killed every year by motor vehicles. Today when I use FSD Beta and it sees a pedestrian or biker it steers to the left of them to give them plenty of room. If it can't safely do that because of an oncoming vehicle or a blind curve FSD Beta slows down and follows behind them. 1000's of pedestrian lives would be saved every year in the US if all vehicles had this capability. FSD Beta is still improving but as I see it this is already a pretty big win. For ref see "cdc pedestrian safety". "More than 7,000 pedestrians were killed on our nation’s roads in crashes involving a motor vehicle in 2020. ... There were also an estimated 104,000 emergency department visits of pedestrians treated for non-fatal crash-related injuries in 2020."

[@tech-utuber2219]

@@datamatters8 I appreciate your thoughtful question. An example regarding human behavior for FSD to recognize and take appropriate action would be that of a fireman standing in the street, gesturing with his arms and hands to slow down and pull over or to actually reverse to make room for more responders to enter an area during a critical situation. Another example, I was driving on the freeway years ago and pulled my car over after a motorcyclist had just gone down and was barely moving. I got out of my car and walked into the lane where he was lying with the downed motorcycle ahead of him. Since it was not easy to make out what had happened, I simultaneously pointed down at him, and with my other hand, I was gesturing to oncoming cars to slow down and go around into the adjacent lanes. FSD would need to recognize this type of behavior as an indication to brake, slow down and request the driver to take over. EDIT, I remembered another example when I was driving at night on the freeway, and my friend in the passenger seat, recognized the erratic behavior of a drunk driver well ahead of us. He pointed ahead and said in a raised voice, “Watch out, stay away from that guy!“. Realizing what my friend had identified, I hung back, and we both observed the car,making sure it was not about to cause an accident ahead of us, so that we would be prepared. It would be great if FSD could recognize erratic behavior, and request for me to take over.

[@datamatters8]

@@tech-utuber2219 These are good examples. It would be very interesting to see someone tackle the machine learning problem of interpreting American sign language. I'm no expert but it seems like this is an AI trainable problem and will eventually be done. Perhaps Google and Apple are already working on it so your smart phone could translate ASL. Traffic related gestures are a smaller group of gestures and will be required for Level 4+ along with complex construction zones and road emergencies. Today FSD beta handles people in and entering crosswalks along with j-walkers. It stops until they clear. Also I have to say you were very brave to help that motorcyclist on the freeway. There are so many accidents with inattentive and sleep deprived drivers. I haven't seen any tests of this yet but I would expect FSD Beta to slow down and perhaps go around you even though it couldn't understand your gestures. Or it might request an immediate driver take over.

[@tech-utuber2219]

@@datamatters8 another thoughtful reply, which I appreciate. The best overall strategy we could expect from an auto pilot system is to request takeover whenever it recognizes potentially problematic situation, or that it clearly identifies a category of situation’s were human judgment should be included. In the case of Optimus the robot, it absolutely needs to eventually understand all human behaviors and responses, including an ever-growing variety of scenarios where humans should make a critical decision, while it pauses all actions due to an inability to predict any successful outcome.

[@FrunkensteinVonZipperneck]

@@tech-utuber2219 “recognize erratic behavior” Let’s train FSD on Trump…

[@HashsirHaroon1]

You talk too much. Stay on topic.

[@noleftturns]

Since 2019 there have been 719 crashes linked to FSD - so says NHTSA and 17 deaths. 11 deaths since May of 2022. Thanks, Elon, for the FSD beta testing. There are many KZbins showing the latest in LIDAR - something FSD will eventually have to use. Each point returned has the following data (for a car): 1) Distance to object 2) Speed of object relative to car 3) Texture of object - ice - water - snow - sand - etc Knowing the distance to the object, it's child play to determine what each point is part of. FSD is fundamentally flawed - wrong software approach (AI vs navigation), and hardware - no LIDAR or RADAR just low resolution cameras. How long before Elon does a "refresh" on FSD?

[@tommornini2470]

Saying it is so is not proof it is so. I believe they’re on the right path, so our opinions cancel each other out. The proof is in the pudding, and the pudding is improving rapidly. Also, you’ve misstated the accident data. The numbers you quote are cases under investigation, not all of which can or will be attributed to FSD Beta, most of them are likely AutoPilot which shares little to nothing with FSD Beta at this point. Also, as a clearly described level 2 system, FSD Beta cannot be held responsible for an accident any more than classic or adaptive cruise control can be held responsible for an accident.

[@robertweekley5926]

Have you ever Driven on a Foggy Road? Safely? How about during high wildfire Smoke? Safely? How is this any different than using Low resolution camera's?

[@Preciouspink]

Are you corrected or have we misunderstood your comment,please continue,do you have something else to offer

[@nguyep4]

You are purposely misleading the accidents and death... 719 crashes but how many mile driven? It matters and important to drawn any conclusions. It is vastly different when 1 accident per million miles or 1 accident per 10 million miles. Most of the death if not all is under autopilot but that is ultimately under the driver ignorance or abused of the system.

[@datamatters8]

In addition to the data you cite here is some more data. There are a large number of pedestrians and bicyclists injured and killed every year by motor vehicles. Today when I use FSD Beta and it sees a pedestrian or biker it steers to the left of them to give them plenty of room. If it can't safely do that because of an oncoming vehicle or a blind curve FSD Beta slows down and follows behind them. 1000's of pedestrian lives would be saved every year in the US if all vehicles had this capability. FSD Beta is still improving but as I see it this is already a pretty big win. For ref see "cdc pedestrian safety". "More than 7,000 pedestrians were killed on our nation’s roads in crashes involving a motor vehicle in 2020. ... There were also an estimated 104,000 emergency department visits of pedestrians treated for non-fatal crash-related injuries in 2020." FSD Beta is not a Level 3+ system and no one claims it is. FSD is still beta software and needs driver supervision which is plainly explained when one activates the software. I see the role of beta testers as a human supervisor over the software running the car not unlike my role when I taught my children to drive. But I'm in the driver seat ready to take over steering, breaking or the acceleration at any time. So currently, the system being tested is the human supervisor PLUS the software and that is made perfectly clear. In the same vein FSD Beta is watching out for me with emergency breaking, lane changing when the blind spot is clear, distance handling from the car in front, stop and go traffic, lane following even on tight turns, etc. It has seen pedestrians I have missed and I've seen steady improvement over the last 2 years. I've used it for long road trips and in congested cities. FSD beta significantly reduces the driving load and over time I and other testers have acquired a sense of where it does well and where extra vigilance is needed. And in some cases there are situations it is not ready to handle, e.g. construction zones, parking lots and road debris, so I just drive manually. I look forward to its continued improvement.