Tesla Confirms MCU/eMMC Recall For Tesla Model S, Model X. But What's An MCU? And What's An eMMC?

[@Graham_Wideman]

Nikki: As an EE, I would like to credit you with giving a very sensible and balanced description of eMMCs, what role they play, their pattern of wearing out essentially inevitably, and the impact that has. Nicely done!

[@tylerreeves8026]

Yeah seriously, it was a pleasant surprise to see that level detail for this video. Heck, most "tech" yt channels don't do as good as a job explaining it.

[@celsostarec6735]

I would add it's a deeper problem with reckless design in Musk's steered culture. A glimpse of the board shows a small cooling fan similar to those used in notebooks. What's it's expected lifetime ? The board is crammed like many consumer electronic gadgets, not like high reliability industrial, military or automotive designs. What are the temperature ranges for the equipment and components? Are they fit for extremes? To be just, they seem to use solid capacitors to avoid one of the most evident failure modes of consumer electronic. eMMC are mostly cheap and disposable pieces of hardware, meant to be used where lifespan is not an issue. Changing eMMC, even for larger ones that spread the use over more memory cells reduce a bit the rate of falure, but not the failure mode. High reliability durable hardware would use other kinds of memory (Nand x Nor; SLC x MLC; etc.). A component that is known to be failure prone should be designed into the system as a replaceable component or subunit. Tesla is clearly opting to show bells and whistles instead of design for reliability. If at all they mean to have the whole module as an expendable one, it should fit the vehicle in a standard way to allow retrofitting newer standard models, like was done with radios, CD players, with the 1 DIN and 2 DIN slots. Designing the standard interface for many models, openning up the architecture doesn't seem to be within Tesla's plans. Than I remember Musk x Mobileye, wanting to oversell what was achieved and secure to use. This way of thinking evident in SpaceX of rapid iteration without excessive design for reliability indeed has it's merits. But there is also a dark side to it, mostly hidden by the anxious Tesla cheerleaders disguised as independent commentators. (Some times they look as circus seals clapping hands in shows). Tesla won't allow a owner to repair his own car independently. They reserve the right to change the product after selling it, as if the were part owners. They design in failure modes far different from the alleged 250.000 mile battery or the millions mile battery. One buys the car or leases the car? A 1990s GM has all the noises os cheap plastic, but when it fails you just buy an original or aftermarket piece of hardware with original design or upgraded and put it back to work. Imagine the fuss if owners had to be allowed by GM to repair their stuff. It would be a scandal. Tesla's supposed treehugger followers like very much stock valuation embedding a Standard Oil like monopoly on transportation and all kind of anti consumer tricks with their blind love. Tesla does indeed contribute decisively for the progression of the electrification agenda. They have lots of merits. But a more balanced approach to evaluating its products and policies is needed. Refreshingly, even discording from Nikki many times, I recognize her effort in maintaining a balanced truthful approach.

[@rtfazeberdee3519]

Tesla finally joins the automotive world, recalls are a fact of life for most manufacturers

[@Hans-gb4mv]

It is not the first recall for Tesla.

[@AnonymousFreakYT]

And buying new technology vehicles guarantees it. My early Arcimoto has had a dozen recalls already. Their even earlier "available for commercial sale but really preproduction" models they recalled and replaced with full-production rather than "fix" all the recalls those had.

[@EVPaddy]

@@user-to2rf1rj5v it‘s rare because they don‘t do recalls unless they‘re forced to do so.

[@EVPaddy]

@@user-to2rf1rj5v Yeah, if you think so. China has forced them to do a recall over their suspension. Guess who doesn't get a recall... Teslas have TONS of issues, from the doorhandles, drivetrains, batteries, lack of primer on parts where aluminum touches steel, lack of other insulation where those (listen to musk vs. munro) crazy many parts in the trunk meet and often leaves holes in the trunk. Also from the wheel wells dirt and humidity gets in. Suspension of model X develops problems often after 1000 miles. Air suspension rattles ... lots of things that have never been fixed.

[@EVPaddy]

@@user-to2rf1rj5v That's good for you. It's quite easy to find 2 owners that weren't that lucky, though. Just go to any forum. Your experience has no statistical relevance.

[@chrisl2915]

Voluntary recall? The recall was enforced by regulators, Tesla did not do this voluntarily they were charging customer thousands to have this fixed. Very poor customer support and not a good look for a company that are the "goodguys" of the car industry.

[@scanspeak00]

Size and power always corrupts. Eg Google, Amazon, Facebook. Soon they will have control of your transportation (movement censorship).

[@buddy1155]

@@scanspeak00 Tesla has always been very eager to use its power even when small, just look at the insane amount of lawsuits Tesla has started against their smaller rivals. And very curiously not against the big boys that are so big that their operational capacity is not affected by a lawsuit.

[@buddy1155]

@@brunobgl I was afraid I had to lookup numerous lawsuits, but guess what, it is so insane someone made a page on wikipedia: list of lawsuits and controversies: en.wikipedia.org/wiki/List_of_lawsuits_and_controversies_of_Tesla,_Inc.

[@buddy1155]

@@brunobgl Just google yourself: Tesla lawsuit against: Neo, Zoox, Rivian, Fisker & Xpeng, That is from the top of my head, but there are lots more lesser known startups sued by Tesla as well.

[@jacobuserasmus]

Thx again for a great report. As always well research and actually getting the details right is just not common in journalism.

[@Hans-gb4mv]

It is sad that Tesla is blaming Nvidia in its announcement. The choice for the chip was made by Tesla and the software that runs on it is designed by them as well. And it is Tesla that failed to disable some unneeded log files that is writing massive amounts of data.

[@carholic-sz3qv]

Tesla should have used a way bigger emmc capacity

[@Hans-gb4mv]

@@carholic-sz3qv The size in itself is not the root cause of the issue. Putting in a bigger chip only postpones the issue. Tesla should have disabled all that logging that they do not need. It's not just logs from the car that destroy the storage. From what I remember in the past, a lot of the writing is done from logging on ordinary programs from the Linux system. Logs that don't have any value unless you are debugging the software itself.

[@carholic-sz3qv]

@@Hans-gb4mv the bigger capacity would have drastically increased the life of the emmc

[@jacobuserasmus]

@@Hans-gb4mv Actually the logs are very useful. But regardless the new eMMC chips are considerably better than the ones available in 2012. So a 64G eMMC will postpone the problem from 5 years to 40 years.

[@henrycruz45cal]

@@jacobuserasmus I don't see a memory chip lasting 40 years but nor will my Model S be with me for that long. I'll be happy with an additional 15 years.

[@22bendavis]

EXCELLENT job explaining how NAND flash wears and how SSDs compare for the uninitiated. Bravo!

[@no_more_free_nicks]

5:54 - ok now I know why in the instructions, of my Samsung NVMe drives, they asked to leave some space unoccupied.

[@celsostarec6735]

Most SSD with NVMe or SATA interfaces will have far more elaborate controllers to minimize risk of failure. One feature is wear leveling. When the host addresses a memory block it's not a physical block, but a logical one mapped internally by the controller to one of the cells. To avoid hot spots of cells that are used (written to) too much, the controller tries to always write to one os the least written physical cells at each time. Sometimes, the content of a rarely used physical cell is moved elsewhere to achieve this wear leveling. When there is a lot of spare space the controller can do it's background work fairly efficiently. When the whole SSD is used, just a small amount of cells are available, cleaned (they must be put to a blank state before being rewritten), and the processing becomes messy, slower, inefficient, possibly requiring lots of shuffling data back and forth. Most memory products, including IDE/ATA/SATA/SCSI hard drives (HDs), eproms, ram, are produced with spare cells that substitute for failed ones, so programmed by the factory, and sometimes at low level formatting (ie HDs). The eMMC shouldn't be different, but are probably not as sophisticated as a proper SSD, and have an inherently failure prone technology to begin with. It's still a good idea to leave some 10% of the raw space of your SSD unallocated, just to avoid accelerating the wear that happens when nearly fully written.

[@ModPhreak]

So in Europe, the cars that have had an eMMC replaced will get more info from Tesla soon. We were told mid February, probably money back, as we payed for the repair. The ones that got an MCU upgrade/replacement might get info as well, as to any money back. The eMMC problem is known for years as you have described, but early excessive logging is the fault, not frequent OTA updates. Though OTA adds to the wear and tear of the flash as well, but to a much smaller degree. Tesla underestimated the issue and refused to help out while private companies would. That did reflect poorly on them. Same with the durability of a few minor parts in the early door handle designs of MS, which should also have been recalled, but is now a much cheaper repair than before. If you upgrade to MCU2 now, there is also an option to keep FM radio for 500$ extra, in addition to the reduced 1.500$ for the MCU2. Last year there was no FM option and you could choose between MCU1 with std. Screen for 1.500$ or keep old screen for a reduced price 1.200$ I think. Alternatively you could pay 2.500$ for MCU2, with the new screen. Before this it was even more expensive to replace the MCU1. If you get a new eMMC (the recall or otherwise), which is fine if you have an AP1, please keep in mind to also get 4G/LTE. If you get a MCU2 upgrade/replacement, which I would recommend even for older AP1, you will get 4G with it.

[@sarran1955]

Hello Nikki, As we both own MMCs..Morris Motor Cars.. Havn't things moved on... The speedo needle would bounce and the indicator bulb needed blinker fluid.. Now, like the ashtray of old the computer is all clogged up..and needs a tech to fix it.. Log files, together with a hammer and screwdriver are in a bag in the boot. Still, it gives them an opportunity to check the suspension dashpots and the exhaust system.. Don't you just love it...... Cordialement,

[@transportevolved]

Nikki here - I sold my morris minors in the UK - but Kate still owns hers up in Washington State :)

[@Top12Boardsport]

Tesla will refund those that have paid for this so far they informed in a newsletter.

[@attist178]

Under a Safety Recall, they don't have a choice.

[@Top12Boardsport]

@@attist178 well in our case they seem to do have a choice. We have three Model S cars in our family 2014 and 2015 260.000km on each all out of warranty unfortunately. Still happy as we have invested a lot in Tesla shares.

[@danzen800]

Do you think they will also refund those who bought the MCU2 upgrade? I literally bought the mcu upgrade 2 days before receiving the email for recall and my car is currently at the service center having the MCU swapped for the new one. I had previously agreed to the estimate for 2500 not knowing that they had slashed the price to 1500 just very unfortunate timing. I haven’t paid yet , do you think I can request for them to adjust the estimate to reflect the 1000 dollar rebate?

[@Top12Boardsport]

@@danzen800 sorry to say that that I don’t believe is possible. You have chosen to get a upgrade with much more functions and much more advanced MCU. I was considering that also as one of our cars has AP but to take full advantage we would need a newer model with 8 cameras and we only have two.

[@danzen800]

@@Top12Boardsport yeah but Tesla slashed the cost of mcu upgrade literally days after o ordered mine . I don’t see why they couldn’t agree to charge me the new cost since they are installing it today

[@TheRgnvideo]

Thanks for this update, it's not before time. I wonder what the situation will be for UK owners?

[@koma-k]

Same as for Norway I'd guess - I got mine replaced a month ago; after the Christmas software update the car told me the eMMC ("center display storage unit") was degraded and to schedule a service appointment through the app.

[@ruaraidhmcdonald-walker9524]

It's already agreed for UK. TOGUK have pushed hard for this. Will Fealey (TOGUK Pres) had a mention on a Sky news article yesterday to discuss it.

[@christopherbaudelaire]

They should just upgrade owners to MCU 2

[@M1911jln]

This was not a small fault that couldn’t have been predicted. This was a ridiculous engineering choice that was guaranteed to fail quickly, all to save just a few dollars per car.

[@levenkay4468]

Apt-so-frakking lutely, and good on NHTSA for holding Tesla to account! Hopefully, Tesla will gain an appreciation of the difference between a couple dollars' manufacturing cost and a couple *thousand* dollar mandatory recall on every car built, and the effect on its brand image and customer goodwill.

[@Holywood07]

Thanks for the update - lets hope this recall will reach Europe as well! This is exactly one of the points that I'm fighting with Tesla. It's indeed high risk at the moment: driving in the cold with the whole family, MCU needs restart because of a software crash (which happens regularly), AC turns off, and before the MCU completed its reboot (over 3 minutes), the windows are dangerously fogged. This means: as soon as the MCU crashes, I need to start watching out for a place to stop to not risk a crash. And up to now they said: live with it! As a former fanboy (next car will not be a Tesla anymore) I enabled ("send logging data home") for Tesla to help to speed up autonomous software development. This clearly wore out the memory a lot more; and this is how Tesla says thanks now (or at least used to)? Really hope they give in now and replace the memory (as explaned by the service center: the replacement won't be a new MCU, but just a used one with the new memory chip).

[@Amuzic_Earth]

Who has already paid for the replacement will get full reimbursement. It's official.

[@MarkLLawrence]

This is the same reason that I use an SSD for my dashcam instead of an SD Card. I hope whoever proposed an SSD and got rejected in favor of the eMMC is getting an apology right now. The cost difference wouldn't have been prohibitive and in the long run cost would have been lower.

[@Wookey.]

Or they could just have turned the logging down a bit so it lasted 50 years instead of 7.

[@marriocharles4168]

Transport Evolved, that is very useful information. There's always a need to keep on improving Technology. Please inform us what is next.

[@erickayson588]

Just entered my VIN on the Tesla Recall Check and realized I have 3 open recalls needing to be completed, which includes the emmc. Got service appointment next week, so we'll see if there's another discounted MCU2 upgrade.

[@bernardfender5147]

It's about time Tesla formally acknowledge the issue

[@robertperry6048]

I had this problem with our 2016 model X P100d. The vehicle was still covered under the extended warranty. I recorded this malfunctioning MCU while I was driving. I alerted Tesla about this issue. Several months later I learned about this recall. I went in to have the unit replaced. I was told that since the unit did not record any problems in it’s data records, they would not replace the unit. I reviewed this video and have decided to recontact the service center and attempt to get this unit replace. According to my email records, Tesla has sent a replacement unit several months ago and so it should be still waiting for me. I will let you know what happens.

[@karlbloss]

Thanks for delving into the details! Would you recommend doing the $1,500 upgrade or just getting the replacement if one is planning to keep the car longer term? What are the pros and cons of the new MCU?

[@erickayson588]

Good question. I take my car for service next week for recalls and wondering if they're gonna give me a MCU2 option less than $1500.

[@henrycruz45cal]

@@erickayson588 let us know please

[@erickayson588]

@@henrycruz45cal - bringing in to service tomorrow. I'll report back

[@erickayson588]

Unfortunately the part needed to complete the emmc recall was not available yet. They are telling me it won't be readily available till March. Also, if I did want to replace to the MCU2, it would cost $1500 to complete the upgrade. They're not sure if and when a refund for those who paid the original $2500 will be applied. I'm assuming they will. Since I plan on getting the Plaid, I'll pass on paying the $1500 MCU2 upgrade.

[@iansemailbox]

Hello 👋🏼 does anyone have an update on how Tesla replaced the eMMC ? Did they only replace the eMMC or the swapped the whole MCU ? If not, would it be worth paying to swap to MCU2 ?

[@wilkinlee5005]

This is great piece and well put summary! What is the view of the faulty door handles also? Surely it is a safety issue if you can't easily access people in the cars

[@passdasalt]

Sell tsla stock. I need to buy the dip.

[@barryw9473]

Nikki, are you going to investigate LG Chem cell issues across multiple vehicles? It seems the Bolt and Kona (others?) use identical cells.

[@Wookey.]

I could have told them that the emmc would wear out the way it's being used. It's bloody obvious if you know anything about nand flash filesystems. emmc hides a lot of the details in its internal firmware and algorithms so it'll work for a long time as spare blocks are mapped in to cover for worn-out ones, but eventually it'll run out of spares and then you get filesystem-level errors.

[@koma-k]

Seems the timeline is slightly different for Europe, unless I misunderstood something: I got an e-mail from Tesla on december 10th, about owners of cars that are within the 8 years/100000 miles (well, it said 160000km) from date of delivery will get a failing eMMC replaced at no cost, and that if it has already been replaced I may get reimbursed. By then I was already suspecting that the eMMC was degraded and near failure. Fast forward two and a half weeks and as we get home from visiting family for Christmas the car gets a software update, after which it popped up a warning that the "center display storage device" was degraded; it was OK to drive but I should schedule a service appointment. Tesla Service confirmed that the software update included what was previously a separate service centre diagnostic tool to check eMMC health. The small(ish) daughterboard carrying the eMMC was replaced a week and a half later (parts hadn't yet "trickled down" from the central warehouse in Oslo to my neck of the woods). The eMMC replacement fixed a weird error I was having with seat memory - often the seat back would not move fully forward to the stored position from the "easy entry" profile - sometimes selecting the driver profile again would fix it, but mostly I had to adjust manually and store the settings again. I also had the MCU completely shut down (or crash?) while parked on a couple of occasions, and taking several minutes to start up.

[@johnnychang4233]

Why a soldered eMMC and not a microSD card slot as a way of storage upgrade?

[@Wookey.]

Significantly more reliable. Especially in a vibratey automotive context.

[@johnnychang4233]

@@Wookey. It's almost no self generated vibration or harmonics frequencies unlike an ICE.

[@johnnychang4233]

@@brunobgl But as Tesla is taking more of an Apple approach and try to make their system as third party servicing unfriendly we may have not much to choose about.

[@no_more_free_nicks]

I thought that Tesla have give a choice to upgrade to the MCU2 for 1500$ instead of 2500$. I heard about this on YT, so no idea if it is true.

[@dewiz9596]

I wanted to replace the hard drive of my Windows XP machine with a SSD, but was informed that it would not be possible because XP did not do the “levelling” procedure. The nature of the work I was doing created a *lot* of temporary files. . . and being “of an age”, I decided it was time to close up shop and retire. Why XP? For some very good reasons. One piece of critical software I had written was 16-bit, written in Borland C. Some of the functionality has not been available in Microsoft C, without completely rewriting for a windows environment.

[@marc0523]

An SSD would be fine, unless you were running a _lot_ of data. Modern SSDs can cope better than old ones. Also moving the Page file to a different drive helps, though if you have enough RAM the page file wouldn't be used that much. If you WinXP program can be run inside a virtual machine, then you could run it on any modern PC (or Mac) to save yourself the hassle.

[@Wookey.]

That's bollocks. You were misinformed. SSDs do the levelling internally. The computer can't tell the difference between a hard drive and an SSD.

[@marc0523]

@@Wookey. Your computer can tell if a drive is an SSD or HDD. Windows will enable TRIM for SSDs and Defrag for HDDs automatically. Windows XP probably won't be able to do this though, but it is a long time since I checked. But again, modern SSDs can cope with rough treatment, you don't need to worry about the rules as much any more.

[@Wookey.]

@@marc0523 OK yes, there are minor differences - my point was that it's the same physical and logical interface. They can be treated exactly the same. There are these things which are done differently by way of optimisation, as you point out. But you can always plug an SSD in where an HDD went before, you just won't see the full advantage (and will put extra wear on it) without the OS using the extra ATA command TRIM.

[@ianburnett4605]

Voluntary recall. Baaaahaahaa.

[@theheathkitshop2424]

Wondering what you think GM will do with the Bolt EV battery issue? I have a 2017, never had the upgrade done yet, but using hill top reserve in the mean time. Wondering if LG chem will up for new packs or replacement cells?

[@ouch1011]

They will probably do the same as Hyundai/Kia, which is a software update to increase the fault detection sensitivity of the battery management system. Their goal/hope is to find the individual failures and replace the packs selectively, hopefully before they catch fire, rather than replacing all the affected packs. Hyundai/Kia use very similar LG Chem batteries in the Kona and Niro, and they had the same problem.

[@stvboxtvbox3024]

if u got a sun roof the mcu2 eliminates your am/fm radio

[@chrismcardell8137]

Good to hear that Tesla are doing the right thing eventually. Pity they didn't do it themselves, but no other manufacturer would have either.

[@iansemailbox]

Hello 👋🏼 does anyone have an update on how Tesla replaced the eMMC ? Did they only replace the eMMC or the swapped the whole MCU ? If not, would it be worth paying to swap to MCU2 ?

[@tonespeaks]

This is a strange recall, because there isn't a defect in those memory. Flash memory wears out, this is a known issue, their shelf life is limited to the number of writes performed, that is why some cars have never had the a problem and others have. Any computer with an SSD will have a limited life span. Most parts of the car, has a shelf life, usually failing just after the warranty period (hehehee). So for NTHSA to demand a recall is a little strange since, this is not a faulty part. It isn't really an safety issue as well, since there is a warning long before the part fails. On the other hand, as a courtesy Tesla should recall the part and especially since it's a minor cost. If NTHSA is going to suggest Telsa recall this part, get ready for many recalls of many parts.

[@EinzigfreierName]

Sure every product has a limitied lifetime. But If you buy a car with a limited lifetime of just a few years - that can't be right.

[@Conenion]

> Any computer with a SSD will have a limited life span. Yes, everything has a limited life span. But in this case it is way too short and most importantly represents a safety issue. There is no warning, AFAIK. The problem is that they didn't use a SSD which is more expensive and would have lasted many years longer. See the video. What they have used is more like an SD card.

[@kkobayashi1]

I suggest you watch the video before commenting.

[@danejurus69]

I prefer manual plastic knobs to control my vehicles functions. All this electronics in modern cars just means more parts to repair/replace over time.

[@ruaraidhmcdonald-walker9524]

Finally. Poor coding, poor hardware specification but mostly a complete lack of ownership of the problem and a total lack of respect for owners. 5th major failure of my S. Voluntary? Hardly. Only after they were forced by the NHTSA.....

[@no_more_free_nicks]

Technically they could be forced only by going to a court? So if there is no final sentence that would make it voluntary?

[@ruaraidhmcdonald-walker9524]

@@no_more_free_nicks No. The NHTSA can simply tell them to do it. The first step for the NHTSA was to *ask* them to do it. If they had refused they knew what was coming next so they "volunteered" to do it.... under duress. If it was truly volunteering they would have done it years ago and saved us all a load of grief and stress.

[@Hans-gb4mv]

Voluntary in that you are not required to have it fixed under the recall. If it was required, Tesla would have to do the change the next time they see your car for service. Now it is optional.

[@ruaraidhmcdonald-walker9524]

@@Hans-gb4mv Mine's needed doing for the last 6 months and they've been putting up every possible blocker and pushing it back to me every time, including trying to charge me £2.5k for an upgrade to MCU2 rather than just making my car work properly under warranty. Frankly I'm glad they've had their hand forced. Dreadful way to treat customers. You can argue about semantics all you like, the net result is that they are fixing the car no questions asked. As they should have done in the first place.

[@fjalics]

On a positive note, you are about to get a free(not including your time) upgrade. I think you will get not just a new chip, but a new MCU.

[@LewdCustomer]

I got my notice of repair. Felt good about my car for an hour and it's not broke yet.

[@JPSG3]

That memory should be modular not soldered to the pcb.

[@EinzigfreierName]

Not really. Directly soldering it to the PCB saves cost and increases reliability. It's just a design failure to use an eMMC the way they did.

[@JPSG3]

@@EinzigfreierName but if it fails you pay a premium.

[@carholic-sz3qv]

@@JPSG3 totally agree with you.

[@EinzigfreierName]

@@JPSG3 Well, if they had thought about this possibility, that would not have made the eMMC replaceable but change teh design in the first place.

[@danzen800]

I ordered the MCU2 upgrade literally a few days before receiving the email from Tesla about the recall. I unfortunately had not seen that they slashed the price before I agreed to the 2500 dollar estimate on the app. Do you think I can dispute this, and have them charge me the 1500?

[@celsostarec6735]

Asking is no crime. Let's see how they value their customers and commitments. It's a failure embedded in the design, clearly their responsibility. The trick of selling un upgrade is ok, but it should cost nothing to repair to original specs as it is a design flaw. The extra they charged you doesn't seem fair.

[@EVPaddy]

They could have fixed that problem years ago. It is known the eMMC dies because they‘re writing tons of useless unix logs to it. They just didn‘t care to change that, probably because they were happy to sell you a new MCU out of warranty.

[@berndniedermeier3012]

Tesla is somewhat the first one running into this issue with systems in cars moving from literally "read only" (i.e. flashed once with minor updates, nothing really written into storage etc.) towards a system we are used from mobile phones. Android being on board, new Apps possibly popping up after the car is already 3 years on the road, not being able to predict what those apps might actually do and how they deal with the system. Now everybody in automotive moves towards highly consolidated high-peformance-computing platforms, also referred to as domain controllers, zone controllers etc. The level of complexity of those systems combined with the number of companies developing code and applications for those systems - more than 20 is not an exception - is getting huge which creates the desire to log even more data for potential debug purposes. Digital video recording and/or event data recording including FullHD or even 4K camera streams is adding high bandwidth and huge data to the system as well. The trend towards separating hardware development from software development will add to the problem as you can't really blame software developers on a higher abstraction layer for not understanding physical hardware challenges and aspects in detail. It might become a challenge to establish design rules that can be enforced, followed and controlled to mitigate risk in a best way. Of course you could choose to simply put a 1TB SSD into the system and you might be able to forget about many of those challenges (unless the vendor increases page-size with higher capacity, but that'S a different story). But price for such a SSD won't be accepted by carmakers. So other measures have to be taken to mitigate risk and there are some good ones available. Believing that the flash memory controller is handling EVERYTHING for you is a big mistake. All software components can add to a phenomenon called Write Amplification - specified through the WAF (Write Amplification Factor). Easy take on it: if you want to write net data of 1MB and end up to actually write 1.5MB of data into the flash memory then WAF=1.5 - With current programming practices, databases involved etc. you can end up with 3digit WAFs for certain operating systems. Means worst case: you have 1MB net data but for some crazy reason 100MB and more get written into the flash. I am hoping that these risks will be mitigated well - Flash friendly filesystems e.g. would be a good start - os that we don't see similar recalls with much higher volumes in 5 years from now

[@christopherguy1217]

This was a design fault by the software and hardware engineers. In embedded software you NEVER log to your operating drive, logging is done to either a different drive or to its own partition. Secondly, you NEVER design software to fail because of logging errors, you design the software to be able to run when you can't log. The hardware engineers should have added an SD card slot for logging and the software engineers should have thought more about their logging. In the embedded world, it is common to log to RAM and only write necessary information to disk at shutdown. Tesla designed this fault through negligence or for planned obsolescence to force customers to pay in the future.

[@Hans-gb4mv]

It's not that the software fails due to logging errors, it fails due to errors on the storage device itself. Errors you also cannot prevent by partitioning the storage device as the wear levelling will move data around anyway. On an old hard drive partitioning would prevent data from being written into the storage area of another partition, but with solid state storage, we no longer have that principle. If the hardware engineers had added an SD slot, the software engineers probably wouldn't have used it, or if they did, they would have written the logs directly to it and destroyed that thing even faster than the eMMC chip. This is what you get for rushing software with inexperienced people. Young minds, fresh ideas but lacking experience.

[@MarkRijckenberg]

Thanks for this video. Tesla should have done this recall years ago..... I tweeted this issue to Elon Musk a year ago who never responded....

[@ragnarolofsson7554]

Many are we who got. problems with the MCU1 after changing the eMMC. Something Tesla refuses to acknowledge.

[@TheRgnvideo]

Sorry I meant the recall is not before time, not your update!

[@RealAndreGlegg]

Amazing explanation!! This channel is so amazing!

[@Johnbl261]

Anyone buying a 8 year old $100k dollars car should expect massive repair bills,

[@debendragurung3033]

The eMMC is a part that is doomed to break. And just 8GB for such a complex machine with Autopilot. What were they thinking? They should have had the backup plan right from the beginning. Anyway its not the end of the world - cause its easier fix than other parts. But Still why the freaking astronomical price to fix or update it.

[@donaldburkhard7932]

Tesla can redesign board so EMMC can be replaced just like camera SD card but have Tesla do it.

[@peterszczesiak6025]

It put me off buying a Tesla as its well known to be an expensive fault

[@nickclark2278]

Take 2

[@scanspeak00]

Just think, a 32GB memory module would have prevented all of these problems but they cheaped out on their $100k cars with just 8GB. I hope the autopilot system doesnt use that type of memory.

[@shanefiddle]

Glad to hear the government is holding Tesla's feet to the fire on this one! This is an important precedent to set for all auto makers. And it is also important Tesla is setting the precedent of having steady updates to vehicle software! Hopefully other car makers will be forced to follow suit. Thanks for the great coverage, looking forward to hearing about the Bolt EV recall too!

[@tony_25or6to4]

Tesla is offering $1000 off the new, upgrade MCU as an alternative. It might be worth it for some.

[@flossythepig5141]

I bet the new higher capacity chips cost less now than the original did when first installed.

[@chunckyfreshnut]

They should force tesla pay for the recall

[@davidbelin4123]

Recall in Denmark?

[@shoestringsev4529]

I find it surprising that they are not just reflowing a new better chip on the board like Rossmann does at his apple repair shop. Also Tesla should have known that all memory chips fail over time. They could have chosen to use a socketed version and then you could just swap out the chip.

[@ioannis69k]

too much sysloging ...

[@D0li0]

Yup. Should have use all our RasPi tricks for longevity.

[@ioannis69k]

@@D0li0 I recall a video I saw more than a year ago with Rich Rebuilds and another guy when he explained this but the difference was that at that time Tesla was charging for the replacement of the whole “computer” and not the eMMC that was the one failing. Correct dimensioning is sometimes tough when you start and you don’t foresee correct the use of a component.

[@EinzigfreierName]

Tha's what happens when software guys ignore the hardware they are running on. In our modern world of virtualization and cloud computing, noone seems to be interested in hardware any more :-(

[@ioannis69k]

@@EinzigfreierName Can’t agree more !!! For them it is an “ abstraction layer”. 😹😹😹

[@D0li0]

@@ioannis69k I doubt they are replacing the eMMC component though... I think @greentheonly has a video of doing the replacement, but it's soldered, so not for the faint of heart, I wouldn't attempt it.

[@zapfanzapfan]

MCU? Marvel Cinematic Universe? ;-)

[@gryzman]

This has been a known problem for 5+ years Nikki. It's a truly monster size of a f-up

[@brent_in_aussie]

Great informative video as always Nikki , it does make you wonder what other shortcuts or cost cuts Tesla did or still does ?? Why use a proven to fail tech ?

[@Wookey.]

There is nothing wrong with the tech, just the way they used it. _All_ nand flash everywhere (MMC, SD, microSD, SSD) is unreliable - always has been. But it's huge and cheap, so reliable NOR flash is only use for very specialised things these days. The unreliability is hidden by layers of software and spare blocks and clever filesystems, and then engineers still have to make sure they don't do more rewrites than the thing is designed for. Tesla did way more rewrites than the thing was designed for (pretty much by accidentally leaving a load of debug turned on). It really isn't the hardware's fault.

[@brent_in_aussie]

@@Wookey. I had no idea about the flash being so fragile I guess until this video Did Tesla know or not know this would happen do you think with the amount of writing and wiping ?

[@Wookey.]

@@brent_in_aussie They should/could have known. This is basic embedded-systems engineering. If I'd been on the team I'd have been looking at the flash filesystems in use and the data rates and calculating from the chip spec whether it could reasonably be expected to last the life of the vehicle (flash filesystems used to be my thing :-). But I suspect that they didn't have any experienced embedded engineers on the team just a load of UI designers, and people who knew about self-driving. Or maybe they did but that guy was overwhelmed with work and simply didn't have time to look at the system as a whole. It's still hard to understand how an important project like this left the logging set up in a pretty dumb way and never measured the flash wear rates. I'm pretty sure it was cock-up not conspiracy/mendaciousness, possibly due to trying to move fast, but it is a _really_ embarrassing cock-up to those of us who work in this space. I also don't quite understand why they didn't fix it sometime after release, unless the system architecture was such that they couldn't actually change the base-level stuff responsible for the logging, but that would be a very odd (and insecure) architecture.

[@brent_in_aussie]

@@Wookey. thanks for your experience in answering mate I found it very interesting and was surprised for sure We are hearing here in Australia they are unsure if these issues will arise due to lack of the number of Tesla’s here Ev’s are so rare here and no government support Thanks again

[@eddiestevenson-kaatsch6306]

You look nice in your glasses. XXX

[@jiankuo]

Commpewwwwterrrrr

[@SpinMChecker]

You have a limited amount of over the air updates 😜

[@scanspeak00]

To summarize, Teslas are getting digital alzheimers.

[@godtable]

And thats how you can become the richest man in the world.