REVEALED: What Tesla FSD OCCUPANCY NETWORKS Are--And How They'll Soon be MUCH BETTER! AI Day 2

Let's take a deep dive into Tesla's foundational Occupancy Networks, as described by Ashok and Phil during AI Day 2022. What is the state of the art for occupancy networks as Tesla has built and is using them? What about how NeRFs are critical to occupancy networks? And what is coming next for occupancy networks, including continuously variable voxel resolution. It's no exaggeration to say that occupancy networks are and will be a game changer for Tesla!
My video on Ashok's recent talk: • Don’t Crash, EVER!! TE...
**To become part of our Patreon team, help support the channel, and get awesome perks, check out our Patreon site here: / drknowitallknows . Thanks for your support!
**Want some awesome Dr. Know-it-all merch, including the AI STUDENT DRIVER Bumper Sticker? Check out our awesome Merch store: drknowitall.itemorder.com/sale
For a limited time, use the code "Knows2021" to get 20% off your entire order!
**Lift pucks for your Tesla: amzn.to/3vJF3iB
**Emergency tire fill and repair kit: amzn.to/3vMkL8d
**If you're interested in a CO2 Monitor, you can check it out here: amzn.to/3PsQRh2
**Check out Artimatic: www.artimatic.io
**For 30% off the amazing OneCarStereo instrument display for Tesla Model 3/Y, use this link: bit.ly/3vd7NRv
**Get 20% off an awesome Ekster wallet and/or wallet locator(!), check out my affiliate link here: shop.ekster.com/DrKnowitall
**Want to get in on WeBull's awesome Crypto and stock fun? Check out this link, and get started trading stock and Crypto!
a.webull.com/i/DrKnow-it-allK...
**If you are looking to purchase a new Tesla solar roof, solar tiles or PowerWall, just click this link to get up to $500 off! www.tesla.com/referral/john11286. Thank you!
**You can help support this channel with one click! We have an Amazon Affiliate link in several countries. If you click the link for your country, anything you buy from Amazon in the next several hours gives us a small commission, and costs you nothing. Thank you!
* USA: amzn.to/39n5mPH
* Germany: amzn.to/2XbdxJi
* United Kingdom: amzn.to/3hGlzTR
* France: amzn.to/2KRAwXh
* Spain: amzn.to/3hJYYFV
**What do we use to shoot our videos?
-Sony alpha a7 III: amzn.to/3czV2XJ
--and lens: amzn.to/3aujOqE
-Feelworld portable field monitor: amzn.to/38yf2ah
-Neewer compact desk tripod: amzn.to/3l8yrUk
-Glidegear teleprompter: amzn.to/3rJeFkP
-Neewer dimmable LED lights: amzn.to/3qAg3oF
-Rode Wireless Go II Lavalier microphones: amzn.to/3eC9jUZ
-Rode NTK Studio Microphone: amzn.to/3vjMomW
-Focusrite Scarlette 2i2 audio interface: amzn.to/3l8vqDu
-Studio soundproofing tiles: amzn.to/3rFUtQU
-Sony MDR-7506 Professional Headphones: amzn.to/2OoDdBd
-Apple M1 MacBook Pro: amzn.to/3wPYV1D
-Docking Station for MacBook: amzn.to/3yIhc1S
-Philips Brilliance 4K Docking Monitor: amzn.to/3xwSKAb
-Sabrent 8TB SSD drive: amzn.to/3rhSxQM
-DJI Mavic Mini Drone: amzn.to/2OnHCEw
-GoPro Hero 9 Black action camera: amzn.to/3vgVMrH
-Suction car mount for camera: amzn.to/3tcUfRK
-Extender Rod for car mount camera: amzn.to/3wHQXsw
**Here are a few products we've found really fun and/or useful:
* NeoCharge Dryer/EV charger splitter: amzn.to/39UcKWx
* A totally fun Tesla coil toy--awesome for the whole family: amzn.to/30rvzbI
* The Humane Interface, by Jef Raskin: amzn.to/3ajM7YT
* AI Superpowers, by Kai-Fu Lee: amzn.to/3dgRSs3
* A camping mattress for your Tesla model S/3/X/Y: amzn.to/3chSRrQ
*And of course the amazing "Elon Musk: Tesla, SpaceX and the Quest for a Fantastic Future": amzn.to/36hqdmS
**Music by Zenlee. Check out his amazing music on instagram -@zenlee_music
or KZbin - / @zenlee_music
Tesla Stock: TSLA
**JOWUA:
*For an awesome wireless charging phone holder mount, check out: bit.ly/3tRMSQR
use the Promotion Code: DRKNOW5
*Tesla USB Hub: bit.ly/2PXHPit
use the Promotion Code: DRKNOW5
*Powerful wireless iPhone/Android charger for your Tesla Model 3 or Model Y, check this link: bit.ly/3b5fHSW
use the Promotion Code: DRKNOW5
**EVANNEX
Check out the Evannex web site: evannex.com/
If you use my discount code, KnowsEVs, you get $10 off any order over $100!
**For business inquiries, please email me here: [e-mail]
Twitter: / drknowitall16
Also on Twitter: @Tesla_UnPR: / tesla_un
Instagram: @drknowitallknows
**Want some outdoorsy videos? Check out Whole Nuts and Donuts: / @wholenutsanddonuts5741
OCCUPANCY NETWORKS and Tesla FSD Beta: The Present--And WHAT COMES NEXT! AI Day

[@BongoWongoOG]

Mimicking how the fovea sees. Close up detail handled by a part of the eye which has a more dense pack of rods and cones (higher details) for close up work. Further away the outer retina is used which has fewer rods and cones but 'registers' the photons of objects. By varying the voxel sizes, we're effectively creating a digital 'fovea' and reducing the compute necessary based on distance. This is a great solution IMHO. I'd love to work on this :)

[@peterprocopio2192]

great video! Thank you for stopping the presentation and explaining the important and technical aspects. You did a great job for us non-engineers. I am so impressed at the Tesla team's ability to engineer new solutions so quickly. I can't believe that other car companies will be able to compete with the FSD technology. This also gives me confidence that Tesla will solve FSD before anyone else and introduce robotaxi service before anyone else. When is still the question, but I am more confident that they will.

[@jkev1425]

This is why I initially subscribed your channel 👍👍👍

[@RajIndia31415]

Thanks for explaining it so well. I think you are the only one doing it for a technical audience while keeping it approachable.

[@barnabydunning5424]

You’re a fantastic communicator! Thanks for this vid.

[@keitho9508]

Hey John, please keep this up, you are helping so many people understand what our great company is doing.

[@RjDavis000]

Great video John, would have been in the dark for a while on the AI day content without this

[@williamjulien5858]

Live in a high-rise and park in a large parking deck. For fun I often experiment/play with Beta in the parking deck. It has always gone straight at solid concrete walls (or pillars that stick out) until 69. It will now turn and not head straight at them 100% of the time.

[@mb345]

Really like this format using OBS. Your overlay to the video with pauses for explanation was great. If you had a pen tablet and some obs addon for screen mark up then you could annotate, draw attention to important parts of the screen or switch over to a white board view and explain a concept.

[@FredPauling]

It's mind blowing that this can be done at 100Hz. Will be following this topic very closely

[@kunletim8464]

Hey John, you are a great teacher; I'm sure your students always enjoy your classes😊 Thanks for the beautiful explanation.

[@davidc3463]

Just a heads up “Ashok” is pronounced “uh-sh-Oh-k”… not uh-sh-oC-k. A lot of KZbinrs are misspronouncing his name, ;)

[@loweryjk]

I've been waiting for news on applying variable resolution based on distance to the occupancy network. Thanks so much for finding and sharing this awesome information!

[@jbarvideo12]

Dr KIA Thanks for stopping many times during the Tesla discussion and explaining in understandable terms (and communicating Well to us non gamers) what is happening. It is amazing to understand what is being accomplished in refining FSD.

[@WilliamQ20]

WOW John, what an amazing explanation of what is happening @ Tesla. I felt from last Friday thet they were on the good track but after this video i am convinced that they are on the right track. How pity i is that so many people do not get it and start complaining about the supprimation of those 12 stupid sensors. Thank you for explaining all this in humann language.

[@giovannidegeronimo8941]

I just don't understand how the occupancy network can detect things that the cameras can't see. For example, lets say the car is off, and I put a cone in front of it that is outside of the field of vie of the camera. It has no way of seeing the cone. How can that ever substitute ultrasonic sensors without adding another camera at the front of the car that sees right in front of the bumper?

[@thoughtfulrebel2246]

@@giovannidegeronimo8941 I agree. I wish someone could provide reasoning as to how they will deal with this. Maybe the cameras are recording even when the car is off to see if something goes into the 'blind spot' in front of the car and then it will not go until the cameras see it again outside of the blind spot?

[@explor794]

Perfect analogies and simple explanations for complicated things. excellent work

[@Martinit0]

Excellent summary and explainer. I remember voxels from the 90ies helicopter flight simulator game Comanche. What surprised me here is that the Tesla voxels are not on a fixed grid and they are not full (or empty) voxels but there are many partially filled voxels. So there is a complexity to each individual voxel-

[@mhfs61]

Thank you for a great lecture, John.

[@skipugh]

Thank you. Great presentation 👍👍

[@billl1768]

Thanks for the explainer. Very helpful.

[@SoleLo]

Great video! Thank you for breaking it all the way down. Although I understood some of it, you really allowed me to understand more of the details.

[@TanujBolisetty]

Great explanation. Keep going

[@jubjub7406]

Well done!

[@gdaddy9410]

John - fantastic video!!

[@oisiaa]

It's amazing how much they are able to do with the HW3 CPUs since none of this tech was invented when that hardware came out.

[@ericchild3363]

Excellent John, thank you

[@stevedowler2366]

That one is very fascinating. It peels back a lot of the unknown layers of how Tesla engineering builds a 3D recognizable image from raw pixel data. Thanks much, Dr. KIA (wait, that reads a little scary in military acronyms).

[@sk.n.9302]

Great video, amazing explanation!

[@bukurie6861]

Thank you for your video

[@firescratcher.]

Finally a well prepared video. It's been quite a few videos since John has prepped before presenting. IE yesterday when he tried to bring up an image of a bumper with a sensor, Fail. How hard would it have been to have that picture ready? This one was good!

[@gressex7]

Wow, awesome video. Thank you!!

[@kevryfabroni]

great job!!!

[@richardgoldsmith7278]

I loved the explanation of the variable size voxel map - implications are enormous and dawning on me like a wave …

[@yangni007]

Thanks for this inspiring video ! but is this occupancy map the output of a good lidar ?

[@gaborszollosy2153]

This episode was rather for computer NeRFs :). OK, joke aside great explanation, I like this format that we deep dive into shorter parts of the extra long AI Day.

[@DanyPell]

so excited cant wait for the next massive upgrades an AI day 3!!

[@bootiemacarthur9182]

Rectifying speed instantaneous is amazing to me , during training as a railroad engineer (over 40 years ago) we made recorded trips with motion, sound, speed etc (a wrap around screen in front of the engine (visual progress was regulated to train speed)

[@northernouthouse]

Great video. So, my interpretation is that Tesla took 8 cameras to create its own version of lidar without using the lidar sensor.

[@sandmehlig]

Critical objects far away are smaller, voxel size should not increase with distance but with invariance/insignificance. Even working with outer boundaries, especially for near objects, should decrease complexity and keep functionality.

[@robstewart2213]

Loved this explanation. I guess I'm a geek too. Much appreciated your summary. Keep up the excellent work! Impressive what the team at Tesla has accomplished to date.

[@cpleng7]

thanks you very much , now I have basic idea for the FSD .

[@longboardfella5306]

Thanks John. Great explainer for us. How far do you think they are going on the mid period strategic planning you called out as an issue on a number of occasions? Seems they’d need 10 secs or more of forward planning.

[@richiehart7858]

Very good video.

[@TheElectricMan]

Thanks For Posting Quality #Tesla Content

[@stephm9261]

Great video even a year later!

[@billgibson2520]

🥶 very well done....tks

[@hhal9000]

Great explanation as usual.This is really exciting stuff and it looks like this could follow a similar trajectory to the history of 3D rendering and particularly the use of Games engines to do real time rendering.If they get enough compute power to get the granular detail down to 1mm voxels at very close range just imagine the accuracy of the Occupancy space.

[@davidfellowes1628]

Super interesting

[@jangeiss8693]

You are amazing

[@eugeniustheodidactus8890]

*Thank you Thank you Thank you Thank you Thank you Thank you!*

[@ashsilverwizard3275]

The smallest lod would probably vary with speed as well.

[@paulimhof714]

John, thanks for doing this video. I watched AI day and wasn’t that engaged and didn’t realize how important occupancy networks are. Thanks for breaking it down and making it much more enjoyable and understandable 😊. From your Intel friend you met at Fully Changed Live in San Diego. BTW: Hit me up if you are interested learning about GPUs/AI Accelerators/IPUs/DPUs/SiPho/etc.

[@andyhamilton5926]

One thing I've not seem much discussion of is how Tesla determine the right speed of the vehicle if there's nothing in front. The posted "maximum" is not always "safe" due to corners, weather, and elevration changes. In the UK, our roads are narrower and twistier and I usually find Autopilot going TOO FAST even for slight bends on big roads, resulting in centrifugal force making the car drift to the outside of a not so gentle turn. Ideally the vehicle would slow down or ANTICIPATE the outward drift and apply steering input early. Can you discuss this please? Thanks! Andy

[@alanrickett2537]

So asking this question again with this veiw if you have a high D radar that can sense in blocks of the same size that the vision is using at that distance and just lay the two over each other then you would not have sensor clashing issues .Is this a workable option.

[@CorwynGC]

I think there is misunderstanding about various 'spaces', at least as Mathematicians would describe them. "Raster Space" is a 2D space based on a plane being viewed (like a video screen) Every spot on the plane has a value. "Voxel Space" is a division of a 3D space into small box shapes. Every box has a value. "Vector space" is a representation of items using (basically) arrows pointing to the item in 2D, 3D or even higher dimensional volumes. Fewer items means a smaller vector space. An oscilloscope works on vector space, with beam being moved directly to the spot of interest, rather than scanning back and forth over the whole screen.

[@Mark-kt5mh]

Hey @19:39, that clip is indeed from New York! The chimney is there because of a district steam leak. The articulated bus has the black and yellow New York license plate.

[@cmw3737]

Higher resolution of voxel space will really matter to Teslabot which needs to focus on what is in its hands in order to manipulate it while not needing to know about its surroundings so long as it keeps a distance unless to needs to interact.

[@glen7567]

This is all great and highly technical and impressive but I’m wondering when the car will be able to do U turns, identify road closed signs, identify arrows on ground, identify blinkers on others cars marking intent of other drivers, proper lane selection and timing, not zig zagging when lanes widen, not trying to move around cars that are clearly just stopped at a light, etc.

[@BboySnake71]

Another amazing video. Great job. I now ACTUALLY understand what is going on. Why the late, hard braking on the highway at high speeds, why they will be able to get rid of the Ultrasonic sensors.

[@kipling1957]

Physical occupancy is a bit of a misnomer, in that it ignores gases and possibly transparent liquids. Air velocity and liquid state water volumes may be relevant in certain environments.

[@cgamiga]

Great explanations, thanks! My concern re dropping the ultrasonics, is... lack of close-up camera placement. Just as the AP cameras are fine for street/traffic vision, but Tesla can't do 360o birds-eye parking view: there are no cameras to provide full near-scale perception, especially in front of the bumper, looking down by the front wheels, etc... rear seems to be bit better, w/ the fish-eye backup cam and the rear side repeaters (can see wheels/curb at least)... but I'm not sure the A-pillar cams see that far down? Certainly, not in front of the hood.

[@ChrisFleck]

It will be great to avoid curb rash!! Are the cameras in the correct place? What about curbs in front below the view of the camera?

[@kcx047]

For Optimus, the LOD needs to reduce to 1mm or less. Fine motor skills will eventually need 500 microns or so. At 50 microns, Optimus can perform most surgeries. Very cool

[@chrisd6716]

I'd love to see you do a breakdown of George Hotz breakdown. He didn't seem very impressed at all, and still thinks Tesla's approach to all of this is incorrect. Interested in your thoughts on his approach with Comma versus Tesla's.

[@635574]

I watched half of that and he just comes from a smaller company who cannot even try to make these complex systems. It is far more useful to have a general system that recognizes 3D motion and knows how to avoid things, more important than understanding how to classify the objects in the way. As long as comma AI doesnt have a solution for general unexpected case. IMO they will be as dangerous as the old self driving before 69 and crash into unrecognized obstacles.

[@TheOmega1971]

Maybe this is because there are hardware 4 in the cars now…?

[@johnhanson6039]

Teddy bear looking thing is a Kerbal 🤣

[@m_sedziwoj]

About feature implementation, where is one important difference between games and driving, some elements are not important eg static buildings in distance, and some are very important person laying on ground. So in my opinion cube size should not only dependent on distance but content too and it placement (something looking as human on side of road is not as important as at your lane even with 100m distance could be less than 3s before hitting it). EDIT and one thing, on distance you have limited information because of resolution of camera, with distance one pixel taking more and more physical space (look at Unreal Engine 5 and it Nanite technology, each pixel is polygon, but no more than one, it is final form of LOD system)

[@RandomGuyOnYoutube601]

To me it looks like brown objects are garage doors or gates. It is very impressive that the car can actually discern it.

[@jessiejanson1528]

at roughly 30:00 .. They only have so much they can process at once, so for city driving they ignore things above a certain height, focus more in front of them and to the sides with a bit behind them. But for highway driving the sides almost dont matter, and the back also doesnt matter while at speed. they can focus much further ahead while driving and while stopped they could focus more on whats behind them. I think it is probably relatively easy for them to transfer from a voxel highway front to rear model to a voxel city model as needed. Im not sure LODs would be possible. but given that its a highway they might simply prefer to focus on the lane itself and whats in that lane as well as whats to the sides if its visible. given the limited space they need to search, their normal approach seems better suited for tracking far off objects where the voxels are more suited for 'whats around me that i might hit'.

[@Nakatomi2010]

Brown appears to be either "Openable object", like a Garage door, or Gate.

[@wholmes7177]

Thank you for another excellent video. Do you think SpaceX will use FSD tech to land Starship and Super Heavy?

[@miiihaaas]

17:29 - interesting (off-topic) moment, you covered mic with your hand and the only sound was able to hear is echo... :) I wonder how Tesla's AI networks can use sound in their navigation system.

[@shak7185]

Thanks for your fantastic video! The dynamic Level of Detail problem they are facing currently would potentially be much simpler if there was LIDAR data available which has a much higher level of detail without the need for heavy (and slow) video processing. New LiDAR solutions are coming online now which are about the price of those ultrasonics they just removed.

[@lylestavast7652]

great job breaking things down ... what's the potential value of having increasingly more dense imagery on all/some of the cameras for finer detail and distant object detection ? Obviously the image processing pipeline would require a lot of upgrading... Given today's video things like 4k, 6k, 8k... say they made a jump for the forward facing camera up to whatever the top is today - maybe 8k ... would the increased image density (and maybe optimized depth of field given time of day/f-stops) and leaving the others be cheaper cameras - would it be straightforward for them to dumb down the image on that frame for skeletal info, yet use the virgin of it for some of the distant object determination (or accuracy of velocity changes) using the much higher res ? and in that vein, if doable at all - how about an additional, non-statically positioned camera where the sensor moves in the direction of intended movement as a time leading predictor element ? Maybe that camera can be moved to look in any direction of concern - like backing up , knowing motion from one side seems sketchy etc... I don't know the space at all, just wondering where it goes down the timeline and what's possible ...

[@MostViewedTop40]

Pretty awesome stuff. It would be cool to know how much of the available compute this takes up versus all the other things it is doing. As each new version of FSD computer comes presumably they can have higher and higher numbers of tiny voxels. I always think they are limited by having to keep the hardware in the car the same for so long.

[@Martinit0]

Given Tesla's foresight to use defined interfaces between car components I am pretty sure they could field-replace the FSD computer. The price of FSD certainly has enough margin to do that.

[@WarrenLacefield]

Also the new Samsung cameras are far higher resolution. So they can actually see the "up close" much better than the older cameras and to see the "much farther away" in sufficient resolution. In the future, another dimension of the "video stream" might well be the (variable) focal/depth of view of the cameras. That might be a way of implementing variable "voxel resolution" of the 2-3 second dynamic occupancy network.

[@mjr7991]

Is vector space running in the latest version? My experience would suggest no. My Tesla does not see a parking arm that is down and will plow through it if I let it.

[@goingballisticmotion5455]

They did that already w/o NeRFs? Wow, the combination will be remarkable.

[@garretthoneycutt3432]

Red means the object is potentially dynamic.

[@cheronecom]

If Tesla Vision is good enough to prevent me from smashing into my wall or trailer or curbs, then great. I sure hope this isn't a mistake.

[@de-kat]

It's time for Tesla to bring out a few vacuum cleaning robots, with this technology they can easily outperform all existing manufacturers in terms of detection and production costs.

[@capslock9031]

I'm really looking forward to your video here, since I just watched some of what George Hotz had to say about this stuff: kzbin.info/www/bejne/ooS7qHywrNxpi9U My impression is, that there are lots of philosophical differences between what he likes to do and what Tesla does, but that his criticisms are at least very well informed and fair, on a technical level. Two take-aways were that he views the occupancy nets as a reconstruction of lidar and that he thinks Tesla still - and to their disadvantage - does quite a lot of hard coding. I'd really love it if you could bring on George Hotz along with James Douma and maybe even Scott Walter to discuss some key points of Tesla's hard- and software approaches. That would be an epic discussion, I believe.

[@terrymatic]

DKIA, greetings from Jamaica! Could the occupancy network be used to translate 2D games to 3D VR worlds? Gaming and VR have the same problem as EVs and charging ports...everybody is doing their own thing. Could there be a way for Tesla to convert all the 2D objects in any game and translate it into a universal VR format since they could have the physics for every object it would make things frictionless.?!

[@barrydaugherty5528]

With all their doing, it seems like it would be extremely easy to present a Birds Eye view of the car when parked or exiting a space. Wonder if they’ll ever present that to users?

[@dareldavies2674]

Looks like my kids Minecraft yet will be a major turning point.

[@colinmaharaj50]

Dr Know-it-all, I like that name, Im new here..

[@andyfeimsternfei8408]

The problem with eliminating ultrasonics is the cameras can't see close to the car all around. The occupancy networks can be inferred for objects moving into the blind spots if the cameras are on, however if an object gets very close the the car while parked it will be unknown to the network. Example, a small animal or a child's toy placed near the car while parked. This would not be a problem if the cameras had 100% coverage of the vehicle. In my opinion, if Tesla had upgraded the camera locations FSD would be much simpler and farther along by now. The team is struggling with lots of elaborate measures that would be avoided if the front fender repeaters were located at the front of the car, perhaps on the sides of the headlights.

[@russadams3008]

The problem with the occupancy network AT THE PRESENT is it avoids "objects" that it should not. I had FSD attempt to avoid puddles recently. I'm not talking about flooded roads. I'm talking about a puddle that occupies 1 lane of a multi lane road. What it did was attempt to change lanes every time it encountered a puddle. This resulted in weaving from lane to lane. It became unsafe and I took over. I captured this behavior by pushing the feedback button every time it happened. I wonder if Tesla even noticed my feedback.

[@alexi7787]

Hmmm thinking about the occupancy network but still think a camera is needed on the front bumper as how can it guess a hydraulic retractable bollard if it can t see it before ?? @elonmusk @Tesla in Europe when we go in cities roads are now being equipped with those bollards but if the car saw it raised before we approached but then (this goes to the blind zone of the car as we get close) then the bollard retracts but if it did see it before , will it say something is in front ?

[@partyboeller]

So the occupancy network is basically LIDAR without LIDAR?

[@amusableackeem]

So basically, they're training our cars to play a video game in the real world lol

[@bujin5455]

30:07. You probably wouldn't want to use dynamic detail quite like that. The faster you go, the further ahead of yourself you want to plan, and the more you want to narrow your field of focus. You can't make fast maneuvers, so up close detail isn't too important, but you definitely want to understand something is in the road far enough out that you can plan and not hit it (even if it's a relatively small something). So the majority of the detail needs to shift out in front of you, and it needs to be focused however many seconds out it needs to be for you to be able to make meaningful/safe course corrections. Basically there is a judgement window, and the faster you go, the farther out that window is, and the narrower that window becomes. (That is in fact how humans do it.) And you want to prioritize your resources around that judgement window, and you want to shift that window according to the situational context. 33:28. That is how the eye works, because it can't resolve detail at a distance, but that is not how human focus works. Cameras will have fewer and fewer pixels to work with as you go farther and farther off into the distance (just like an eye), but the AI's focus (that area where all the resources are spent) needs to be optimize around the ideal decision window, as I described. They may also be able to do dynamic voxels sort of like how the new Unreal Engine does dynamic triangles, where the voxel becomes content aware. Using fewer voxels to represent less complex items. Like how not many voxels are needed to map the occupancy space of a bus. That way you could increase your area of focus for the same resource expenditure.

[@notspm9157]

13:48 MAJOR mistake. That is not a YUV space, raw sensor data is not in YUV space! You can't capture YUV space with an image sensor. Want to make that clear. What IS being sent is the raw camera sensor data for each color in RGB, where there are 2 pixels representing green (as the sensors typically there are 2 green sensors for every red/blue sensors...due to needing it square and we are more sensitive to green). In all the post processing you convert the RGB space into YUV space. So yes they are now working in RGB space, instead of prior they were likely in YUV space...so pretty much reverse of what you said

[@jacksonmatysik8007]

Wonder when the occupancy network reckonises potholes

[@christophercobb6352]

Another way to say "thirty centimeters" is "one foot" (approximately).

[@richardgoldsmith7278]

Be honest, the nodding was you nodding off 😜 😘

[@MGCAUSTIN]

This is a great research project but it's not ready for public release. Can we just get the cruise control work work reliably without slamming on the brakes at high speed? Then we can move on to the fancy stuff.

[@anbuchinnappillai7763]

FSD is long way to go

[@jkev1425]

What I don't understand is how Tesla can replace the input from the ultrasonic in areas the cameras cannot see. Next to the bumper the ultrasonic sensor can "see" but the camera can't. If the scene is moving it is fine, but if you parked your car and now something is next to your bumber. You can't...

[@tldrinfographics5769]

Can’t believe they just flipped like this on software

[@strejf]

Nice video, but they do compress the images right? It's not RAW image data they send to their servers.

[@strejf]

@Torr-Net I don't think so.

[@nettlesoup]

@@strejf Yes, the videos uploaded to servers are each several seconds (>10) worth of lossless compression (i.e. RAW). This is ccording to earlier conversations James Douma has had with Dave Lee. Video camera capability is 1280x960 12-bit per colour. I guess we don't know the frame rate any more since now they're using raw photon counts but it used to be 36 Hz in the old days. Bear in mind that FSD Beta testers regularly talk about getting home and watching their Tesla uploading several GB of data to the mothership when they check the home router. This is mostly lossless compressed video, presumably from all 9 cameras (if we include the cabin camera).

[@strejf]

@@nettlesoup When I store videos on my usb drive in the car they are compressed though, not lossless.

[@nettlesoup]

@@strejf That's correct. Because the USB drive has to hold tens of minutes worth of video, which would be impossible in most cases to store losslessley due to size requirements and rate of data transfer to the USB device. These 10 or 15 second clips are compressed slightly but they need to be lossless because Tesla needs to see exactly what the camera saw originally and be able to use this raw data to train the neural net at the server side. Ten seconds of lossless video is going to be about 1 GB so not a huge issue and I guess it's worth it if Tesla can be strict about the conditions when they collect such data, e.g.: 1) When an FSD Beta tester presses the data collection icon (limited number of presses since local storage is limited). 2) When a crash occurs. 3) When a campaign filter is triggered, meeting a very specific requirement where they want high quality examples of some edge cases.

[@strejf]

@@nettlesoup I think George Hotz said in his analysis video that the images are compressed, not lossless. Comma AI does it and also Tesla he claimed.