When Software Kills: The Therac-25

The story of the Therac-25, where a race condition bug caused patients to be irradiated with massive uncontrolled doses; how and why it could happen. For my book on the spectrum, see: amzn.to/3XLJ8kY
Join Dave as we explore one of the most shocking and tragic stories in medical history - the story of the Therac-25, a radiation therapy machine that went horribly wrong. In this episode, we'll delve into the fascinating yet disturbing tale of how a seemingly advanced technology ended up causing catastrophic harm to countless patients.
We'll examine the Therac-25's inner workings, explore the flaws in its software and design, and discuss the devastating consequences of its malfunctions. You'll hear the heart-wrenching stories of patients who suffered from radiation overdoses, and learn about the heroic efforts of medical professionals who worked tirelessly to treat these victims.
This episode is a must-watch for anyone interested in technology, medicine, or the human side of innovation gone wrong. So sit back, relax, and get ready to uncover one of the most chilling tales in the history of science and medicine!
Thanks to BobT for the episode idea!

[@keleighshepherd345]

Radiotherapy linac engineer here, this is why we have layers and layers of interlocks, both physical and software. It is a very intentional act to strip back those layers (we can and do, but very rarely and always taking appropriate measures to ensure safe practice) The safety innovations of today are writen in the blood of the past

[@TeIegram_me_TheDavesGarage1]

PM LES’T DISCUSS PRIVATELY

[@ApeStimplair-et9yk]

there is many more about cancer and moneylooters to say

[@tonysolar284]

The only time those safe guards are removed/disabled is for testing/maintenance, at no other time should safe guards be removed/disabled.

[@keleighshepherd345]

@@tonysolar284 yup, and even then if we can do it without, we do that first The very first day of my apprenticeship I was told "overriding interlocks is an action of last resort. It is never to be done lightly and we test it has been reinstated before handing the machine back to clinical"

[@blevin591]

And of course, most of these can only be stripped away in service mode, which you are very intentionally told repeatedly to not treat any human or animal with.

[@austinsloop9774]

Some more details for those wondering what happened under the hood - It wasn't the lack of a beam spreader, it was the X-ray target that was incorrectly configured. We make medical use X-rays by hitting a [tungsten] target with high energy electrons. Most of the energy goes into heat, so >100x the amount of electrons need to hit that target to produce a similar X-ray dose vs delivering a beam of electrons for treatment. One reason this race condition was hard to catch was that it required the tech to erroneously select and electron treatment (machine begins to remove the target from the beam line) and reenter the correct setting of X-ray mode while the target was still moving. Since that console saw the target was in motion, it didn't check which way it was moving nor what state it ended in so X-ray outputs were applied with the target out of the beam line. The devices used to monitor the radiation produced by the machine do not operate well at 100x the dose rate and significantly under-measure what has left the machine, further increasing the dose to the poor patient. You normally can't feel radiation, but the current of electrons was so high the patients were essentially zapped like they touched a high voltage wire, that also shredded their cells and DNA. Unfortunate that these lessons had to be written in blood, but I'm glad to this day radiations devices have many layers of redundancy and interlock codes have gotten more descriptive.

[@ΝίκοςΙστοσελίδα]

Do you have a source for the computer-side technical stuff? I have read a lot of what's online about Therac-25 but never seen what exactly happened, e.g. that it didn't check which way the turn table was turning.

[@giornikitop5373]

so, the operator entered "execute small dose radiation" and the machine went "execute order 66". i've been hit with mains voltage some times but those electrons would have felt like hell's whip. man, those poor patients.

[@tammymakesthings]

@@ΝίκοςΙστοσελίδα the references section on the Wikipedia article about the Therac-25 have some good technical details. In particular, the articles by Nancy Leveson.

[@ΝίκοςΙστοσελίδα]

@@tammymakesthings Alright, thanks! Funny thing, I looked up Frank Borger (mentioned in another comment) and I ended up finding an IEEE publication I don't remember reading. The more info, the better.

[@f-s-r]

I wonder why on earth didn't the machine just have an "execute button" and THEN it started moving the target, check that everything is ok, etc. It seems like a LOT safer way to do it.

[@SpaceCop]

Thanks for touching on the "any idiot shouldn't be able to just call themselves a software *engineer*" subject

[@brianvogt8125]

Back in the 1980s & 1990s, I was a mainframe system administrator in a state govt. bureau agency. When our Director approved the creation of the first LAN, the task went to a member of the Clerical team because he was the only employee who had his own PC at home, and nobody in my team knew what to do. It took a lot of years to develop a culture of discipline in the "gifted amateurs" (as one of my colleagues referred to them). In 1977, I did a Fortran programming project in one of our client departments, and saw a piece of code writted by one of their Clerical officers. It achieved Integer Division by repeatedly subtracting the divisor from the dividend, in a DO loop counting the iterations! He had no idea that there were integer & floating point division instructions in a CPU.

[@Milosz_Ostrow]

The title "Engineer" is reserved in every jurisdiction in the United States. In order to bill oneself as an Engineer, one must pass professional exams and work under the supervision of an experienced Registered Engineer for a number of years in a sort of apprenticeship before acquiring a registration or license. However, industry has arranged exceptions in the statutes to allow them to grant "engineer" titles to employees who haven't spent a single day in a college classroom studying the discipline in which they are employed.

[@UncleKennysPlace]

@@Milosz_Ostrow OTOH, some of us dropped out of college, and ended up, say, programming for the DoD. It happens. You cannot discount "gifted amateurs". Is Bill Gates a "software engineer"?

[@HappyCat3096]

I worked for a company that thought programmers were simply overpaid clerks. Ironic when you consider that they were selling a software product. It was a hellacious experience dealing with these idiots.

[@perwestermark8920]

​@@Milosz_OstrowMany of the best software developers are mostly self-trained. Because it's a new science where there is a constant need for new learning. Requiring a huge amount of will to constantly improve. This isn't that compatible with a traditional engineering education.

[@FrederickMarcoux]

For those curious, the ring he mentioned is related to the Quebec Bridge in Canada, which collapsed twice during its construction. It’s a fascinating story.

[@hedlund]

Would you happen to know a good read or watch on that there collapsery?

[@SMA265]

@@hedlund lmk if you find one. Thanks

[@snoozevmw]

The US version is known as the Order of the Engineer, and is a stainless steel ring worn on pinky.

[@markmatlock9918]

One of the key people who helped find the software race condition bug was Frank Borger at the University of Chicago Medical school. He was active in the Chicago Area Real Time Society, the local DECUS group for the Chicago area. I met him at a number of these meetings but did not hear about his role in figuring out what happened to the THERAC systems until years later.

[@rowdyriemer]

Software engineering has taught me a lot about being intellectually honest and humble. Between compiler errors, my own pre-review testing, code review feedback, automated test failures, and bug reports, I've been repeatedly reminded over the years that no matter how confident I can be that I thought through something correctly, I can still make logical errors, fail to consider certain contexts, etc. I'm glad I work in the video game industry. Heheh, the stakes are much lower.

[@andersjjensen]

Yeah. Getting told by the boss that "Some gamers are so upset they started a new subreddit about the problem, which is in your code" is quite different from "The plane suddenly went into a nose-dive and all 300 passangers and the crew died... the problem has been identified as being in one of your code blocks".

[@rowdyriemer]

@@andersjjensen Imagine how stressful it would be to write code for launch systems used for nuclear weapons!

[@coldlyanalytical1351]

Anecdote about bugs: There is no such thing as bug free code. Once I was responsible for the safety of a software driven robot handling plutonium products. The developers were given 2 week to specify their smallish modules, 2 weeks to code ad deliver it bug free (?) .. and then I would use a Monte-Carlo rig that I developed to test their code on a very fast computer for EIGHT WEEKS. In every case I would find maybe 4 or 5 bugs in the first couple of days. Then 2 or 3 more in the first 2 weeks. Then it would go quiet .. BUT .. in a couple of cases a rare timing bug would be detected in week 6 or 7 after working through millions of random tests which passed. Just as well we found & fixed these bugs! (After this I used my Monte Carlo rig on many other less critical projects .. I found zillions of bugs in 'bug free' code)

[@trapfethen]

To take it one step further. There is no such thing as bug-free code because such a construct would require bug-free OS's, bug-free Firmware, and Bug-free Hardware. Based on issues we have seen in just the past few years when it comes to CPU hardware-level errors, it stands to reason that the conclusion that bug-free software is impossible is correct even for seemingly simple constructions.

[@JamieStuff]

I heard that there was a bug free version of "Hello World", written in machine language on bare metal, running on a COSMAC Elf, in 1977. Things were simpler then.

[@coldlyanalytical1351]

@@JamieStuff Oddly, I was an RCA 1802 programmer in 1978-1979.

[@wtmayhew]

⁠@@JamieStuff I remember the RCA COSMAC ELF and the 1802 processor. Subroutines were a pain to code. I worked for a lab which had a job developing a communication device to be deployed on a satellite. The 1802 was chosen for the processor because its CMOS architecture made it more impervious to effects of cosmic radiation in space. I believe the first CMOS Intel processor was a version of the 8085 in about 1985. We would have preferred the 8085, but it wasn’t going to be space qualified soon enough.

[@jess648]

@@trapfethen exactly, not sure such a thing is even possible because entropy

[@miscellaneousHandle]

I was a developer on a medical device when the therac-25 tragedy happened. it was a truly sobering event to those of us in the industry. you are absolutely correct that many lessons were learned, many regulations written, many procedures changed. but a handful of the core causes remain a problem to this day. More so in concurrent and real-time systems. My Hope around this is twofold. first that every developer realizes that these problems can occur regardless of the program and language regardless of the operating system, regardless of the quantity of unit tests. second, Nancy's seminal paper on the disaster should be required reading for every software engineer

[@gruntaxeman3740]

When doing something where people's lives are at stake, it requires very different mindset. Operating system: Preferably no operating system. I don't think there are many that can be used. Language: That is easy. When there is garbage collection cycle or memory allocation fails because of memory fragmentation, that can cause airplanes to drop or nuclear meltdown if used in wrong place. That safety critical scene is likely still C, Ada and perhaps very small subset of C++. Unit tests: Good for developing normal business application to cover something but they are nowhere near enough if human lives are at stake Software usually doesn't need to be error free. Quality, and process to achieve desired quality should be specified.

[@brianjuergensmeyer8809]

@@gruntaxeman3740 Exactly this - as a programmer with 30 years of professional experience (mostly in the medical field, no less), I'm constantly shocked at the number of "engineers" that we hire that have absolutely zero idea of how the full software stack that they're using actually works. They've been trained to see everything below them as a black box that is guaranteed to work every time and will magically solve problems with parallelism, threading, memory management, and data persistence. In a perfect world, I'd agree with you that we'd ditch everything but the application, and make the application its own operating system. In the real world, I'd very much like to see hardened, full real-time operating systems that can guarantee responsiveness in a given number of clock cycles. What I get is Windows. Even though my work is mostly EMR related (as opposed to medical device), I've just seen so much crap stacked SO HIGH that it seems a special miracle every time a patient encounter succeeds. In the medical field, if you don't understand the full stack from the machine code writing the registers all the way up the stack to the user interface, you're simply begging for trouble.

[@arsenii_yavorskyi]

the fact that you had to change the thumbnail to *this*…

[@TeIegram_me_TheDavesGarage1]

PM LES’T DISCUSS PRIVATELY.

[@revcrussell]

I work for the company (AECL was legally continued on to be CNL) that made the THERAC, we went on to make nuclear reactors that are controlled by software code. It is so expensive to make quality code, and the real difference between when someone calls themselves a "software engineer" and being an actual software engineer.

[@TeIegram_me_TheDavesGarage1]

HEADUP LES’T HAVE A CONVERSATION.

[@NyelaKearney]

I'm in my last year of engineering at UofA right now, and theres a mandatory risk management and safety class. So many computer / software engineers roll their eyes about safety. There was even that controversy a few years back with APEGA ordering job boards to stop using "Software Engineer" for non engineering jobs. All this to say that even today software is regarded as this ultimate safe tool, perhaps even more so because of the prevalence of the PC. Thanks for your breakdown Dave!

[@Erik_The_Viking]

Thank you for bring this issue up. I work in medical devices, mostly in automated robotic surgery. Reading the Therac-25 is mandatory for anyone working on mission critical code that could hurt/kill people. This was why ISO-62304 was created, which is the software live cycle requirements for software used in medical devices.

[@drozcompany4132]

This is one of the reasons why I have reservations about autonomous vehicles. The risk of loss of life is huge, and the automotive industry tends to rush development to be the first to market.

[@20chocsaday]

The insurance industry won't underwrite it.

[@Odin31b]

Agreed

[@chrisg6597]

The risk of loss of human life is also huge when humans are driving the vehicles. Would it still not be better to use autonomous vehicles (including bugs) if the death toll is less than human drivers?

[@20chocsaday]

@@chrisg6597 Try to solve the question of why an accident happened. You will have arguements going back to the mines where children pull rare ores out of the earth. As well as the years of testing software.

[@eadweard.]

No more so than with human drivers, I suspect.

[@appliedengineering4001]

There's one valuable lesson that every software engineer need to learn, and that is, "You can't use 9 women to make a baby in one month". In other words, don't rush it.

[@apveening]

Most software engineers understand that one but manglement usually has a problem understanding it.

[@patrikfloding7985]

But you can try..?

[@greggoog7559]

Tell that to the "AGILE" idiots 😃

[@TeIegram_me_TheDavesGarage1]

HEADUP LES’T HAVE A CONVERSATION

[@bdssltdprc]

"The Mythical Man Month"

[@jacobfalardeau676]

A family member of mine was friends with the woman who was injured by the Therac-25 machine in Hamilton, Ontario. Although she unfortunately later ended up passing away from her cancer within a few weeks, her autopsy revealed that her hip was completely destroyed and that she would have needed a full hip replacement. I remember the first time I heard about that and being absolutely horrified at the idea of being killed by the revolutionary machine meant to save you. AECL made a bunch of changes afterwards to try and make it safer but they didn't correctly identify the issue and more people ended up being hurt and killed afterwards.

[@TeIegram_me_TheDavesGarage1]

PM LES’T DISCUSS PRIVATELY

[@coldlyanalytical1351]

Excellent video. Sadly, having worked for decades on the software of nuclear and high security systems, I can safely say that fewer than maybe 5% of even highly experienced& qualified engineers have any regard or 'feel' for safety or security. Additionally I found that they CANNOT be trained to be safer or more security aware even if their weaknesses are noted. I also found that if you find major - but subtle - defects in the code produced by supposedly very senior engineers, they will vehemently assert that their code is perfect.

[@20chocsaday]

And they are willing to bet their job on that? There is testing and the results.

[@coldlyanalytical1351]

@@20chocsaday I would edit their code because they wouldn't. It would then pass the extensive tests I was using.

[@sheilam4964]

Thenin lies the ERROR. Humans have ERRORS. Not the code. Code is produced by HUMANS. We have yet to find a way to protect ours selves from OURSELVES. - - - - I wonder who produced our code? 😆😆😆😆😆

[@_Mentat]

Also, management won't allocate any time for a belt and braces approach where errors that get through one level of code get trapped by the next.

[@coldlyanalytical1351]

@@_Mentat For nuclear work, you get the time and money! For example, Canada used Formal methods to write a few lines of code which controlled the safety of their reactors. Those few lines of totally bug free code cost a fortune!

[@edwardholmes91]

As always Dave, a great video. I'd like to add though, it was a combibation of a race condition and an overflow error... the code that checked if the collimator was in place would return a zero, before the beam could fire. To make sure zero wasn't present until the check had been performed, code in the setup loop would increment the variable in question, named Class3. The variable Class3 was 8-bit, and would therefore overflow when it reached 255. This meant that the beam could fire even if the collimator wasn't in place, when Class3 overflowed, approximately 0.4% of the time. Courtesy of Matt Parker's brilliant book: Humble Pi - A Comedy Of Maths Errors.

[@gts2ludovicofratts404]

Hi Dave .. your videos are all amazing but this one is one of your best from my perspective. As an engineer in Canada and wearing the ring since 1990s, and going form structural to software careers, you articulated exactly what i tell myself, colleagues and younger aspiring programmers and engineers. This story should be part of anyone's studies in school.. university .. and work. I will share with many. Be well and thanks again for sharing on your channel.

[@romangeneral23]

To me what I find more interesting is how the programmer has never been identified or seen again.

[@miscellaneousHandle]

I will shamefully admit to a morbid curiosity in trying to identify the programmer. his name is known to a handful of University professors that worked the investigation. ironically, I do take great pleasure in thinking of it as one of the first internet celebrities that has never been doxxed

[@romangeneral23]

@@miscellaneousHandle "I will shamefully admit to a morbid curiosity in trying to identify the programmer" Same here. For years I have always wondered how this person has avoided being named. It is fascinating....

[@eadweard.]

Seen again?

[@nua1234]

The programmer wasn’t the person most responsible for the problem, it was whoever decided to remove the hardware safety interlock. The same software bug was on the previous model, and no one got injured because of a hardware safety interlock.

[@nua1234]

@@romangeneral23The programmer wasn’t the person primarily responsible, it was whoever decided to not have a hardware interlock like the previous model (which had the same software bug, but no one was injured).

[@johnandmegh]

Dave, people got into your channel because it felt conversational, with a knowledgeable guy who has a knack for walking through complicated situations and explaining them clearly. Gratuitously sexual thumbnails on a video about a tragedy, and weird AI images in the video itself, don’t feel like what “Dave’s Garage” should be about, IMO.

[@mqblowe]

Well said.

[@DavesGarage]

Can you explain how adding images to a script detracts? If it's the same content, but more interesting visuals, I'm not sure how that's a step backwards. But interested to learn!

[@CalcuClark]

@@DavesGarage I don't really understand what the *sexy* thumbnail has to do with this story. As for the AI images, it takes us out of the story when we are shown photos of obviously nonexistent machines and plastic looking people. Real historical pictures of similar machines and scenarios would be much better. Your audience tends to be one with a bit more technical knowledge, seeing utter gibberish displayed on a fake computer screen in a photo detracts from that technical and historical feel you have often with your channel. Not to mention, many people dislike AI images due to copyright and ethical usage issues surrounding it all right now.

[@marknewellmusic]

@@DavesGarage Allow me to explain Dave: Lose the tits on the inappropriate, sexually suggestive, insensitive thumbnail and regarding adding AI imagery for the sake of it, it's a matter of content for content sake, not always needed.

[@mlann2333]

He's right, not appropriate, bad judgement

[@capncoolio]

I was taught this case in a Software Quality Management course at uni, and it is singlehandedly responsible for my enduring insistence on quality process

[@bartoszjasinski]

My father was an electroradiology technician in a clinic. Many times as a child I was amazed with all that equipment from soviet times. It was like sci-fi movie in real life. I love that old design but scare as f*ck of radiation. If you as a kid see how that shit works, the noise when cathode are spinning, transformers buzzing, warning lights light up... damn its so cool and traumatizing at the same time.

[@halbos7637]

Like all those strange machines and devices in those old Frankenstein movies.

[@Bob3519]

You might like the “diode gone wild” KZbin channel. I think he’s in the Czech Republic. He’s had quite a few episodes demonstrating and taking apart Soviet era electronics. The guy is super smart.

[@belstar1128]

when i was a kid and i went to the hospital i wasn't even allowed to look at the machines i would ask doctors about how they worked but they didn't answer a very toxic environment .

[@bartoszjasinski]

@@belstar1128 that's sad

[@RB9522]

In the 1980's, while working at a computer/software consulting firm I received a request from a local university to develop a machine for psychological research. The machine was to administer electronic shocks to people taking tests. I took one look at the requirements, decided that it was an unethical and dangerous application and refused to participate. I'm sure our insurance company is still thanking us!

[@afaulconbridge]

But was the real research how many developers would write such code? (I hope so!)

[@chrismofer]

​@@afaulconbridgeWhy do you hope that?

[@gm2407]

​@@chrismoferBecause it is less unethical and dangerous than running the machines for the experiment they are asking the code to be deloped for. It is removed by several steps.

[@halbos7637]

​@gm2407 Great that you replied. Ethics seems like an afterthought these days in business and many other areas.

[@gm2407]

@@halbos7637 The thing that bothers older people about younger people is lack of forethought about others and consequences. But the truth is it is most people in a lot of situations that don't consider things deeply. Habit, time constraints, lack of reprocussions observed. Ethics is a subject as deep as the universe but can only really be estimated like the movement of celestial bodies via calculus.

[@nowayjose668]

I'm wondering what accidents will happen in the future because code has been taken from AI output, and only cursorily checked by some cheap developers in a "best cost" country.

[@TeIegram_me_TheDavesGarage1]

HEADUP LES’T HAVE A CONVERSATION

[@wtmayhew]

I knew someone who worked for Picker International (now part of Marconi/Philips) in Solon, Ohio in the late 1980s and he told me about the Therac-25 linear accelerator incident back then and it sent shock waves through the industry. I actually interviewed for a job at Picker and they were using PDP-11s, I believe 11/34, to control the systems built in the department where I interviewed. I was pretty humbled by what I heard about the incident and I did not trust myself to be working on code that had the potential to cause grave harm if something went wrong. We’re pretty accustomed to PCs being reliable these days, but in the 1980s DEC reliability was good but not perfect. What would happen in the event of a computer failure, or worse a partial failure such as disk head crash?…

[@patrikfloding7985]

"We’re pretty accustomed to PCs being reliable these days" Ehh.. I wouldn't trust any PC to run anything critical without much lower level safety catches in place (external to the PC).

[@wtmayhew]

@@patrikfloding7985 Also - in the late 1980s or maybe early 1990s, I went with friends to Cedar Point amusement parks near Sandusky, Ohio. There was a ride called Demon Drop that dropped a car about 50 feet and then dissipated energy by rolling with the occupants flat on their backs down a short track with retarders. The was a hut at the bottom of the ride. There was clearly visible through the window a generic tower case PC, probably ‘286 machine, with no-name amber screen monitor. I couldn’t make out what was on the screen, but I’m glad I saw that _after_ going on the ride. In all honesty my guess is the PC was there collecting performance data and the ride was controlled by ladder logic - at least I hope.

[@patlawler5532]

I was very interested when a co-worker doing embedded software told me about a document titled 'MISRA C' (Motor Industry Software Reliability Association, C Language). While some might read it as a simple list of good programming practices, I read it as a list of rules that were developed because something bad happened when it was done another way, and this new practice would address the issue. Much like the safety labels on step ladders, which were added to prevent another person from suffering that type of ladder injury.

[@jimrafert7372]

Very similar to the Federal Air Regulations. There's a saying in the aviation community, "The FARs are written in blood.". A large percentage of them are the result of lessons learned the hard way. Also, the primary reason for warning labels on ladders is not to prevent accidents, it is to prevent successful lawsuits when accidents do occur. It's possible to manufacture ladders that are many times safer than the ones you buy at the home store. It would just be impossible for a homeowner to afford, store, or even lift one. It's also why the user manuals for even complex power tools spend very few words on how to use the tool. Instead they are filled with "warnings" like "Do not operate with guards removed"..

[@TeIegram_me_TheDavesGarage1]

HEADUP LES’T HAVE A CONVERSATION

[@thaernejem7317]

As a cancer patient, I feel terrified of the idea that a device which meant to cure me, can be my killer because of a software bug!

[@ApeStimplair-et9yk]

as described: it was not a software bug - it was leaded at sabotage based on profits. perfect murder ?

[@blevin591]

Modern machines have both hardware and software interlocks. That same fault would be impossible to happen for several reasons - I work on these exact type of machines (Linear accelerators) and if something even 1/1000 as bad happened the machine would stop treating and alert everyone, and, if you're getting treatment in Manhattan, I might get a call.

[@thaernejem7317]

@@blevin591 thank you, that is relieving.. I live in Jordan and the country is well known with advanced healthcare treatments.

[@jazzmike_]

Glad you’re talking about the Therac, but I was hoping for a much more in-depth technical explanation of the code flaws from your expert perspective given how delightfully detailed you’ve been for many other technical topics on this channel. If you ever were inclined to do a part 2 of this with more details, I for one would be very interested to see it.

[@TeIegram_me_TheDavesGarage1]

PM LES’T DISCUSS PRIVATELY

[@jimmeade2976]

This story reminds me of a similar situation that happened in the rail transportation industry, where I spent the majority of my career. Prior to the 1980s, safety systems (called interlockings) for railroads and railways involved the use of relays all of which were designed and tested for failure conditions, with critical ones being guaranteed to be failsafe, thanks to springs and gravity. In the 1980s, the industry started developing electronic systems to replace the relays ... a large room of relays could be replaced with a rack of 2 or 3 microprocessors, an obvious economic and maintenance advantage. Since the relay circuits are, in essence, boolean expressions, the microprocessors and their software were written to handle similar boolean expressions. Each relay circuit was broken into its equivalent Boolean expression, and entered into the software as data for processing. Fortunately, a problem with this method was found during early testing. You see, relay circuits are in essence a large machine with massive parallel processing, while a microprocessor only does one thing at a time, even when many software threads are involved. This caused the kind of race conditions in the software that Dave describes in the video. Fortunately, this was found and rectified in two ways: processes were created involving multiple developers, validators and testers to ensure correct and safe operation; and some failsafe relays were maintained, just in case the software did do something other than intended. The industry has been using electronic interlockings for years now, with a wrong-side unsafe failure a very occurrence.

[@TeIegram_me_TheDavesGarage1]

PM LES’T DISCUSS PRIVATELY

[@PsRohrbaugh]

I have seen this covered by a half dozen KZbinrs and almost didn't click. I'm glad I did. You provided a much better representation of the low level issues than any other recount I've seen.

[@TeIegram_me_TheDavesGarage1]

PM LES’T DISCUSS PRIVATELY

[@Akens888]

As an industrial control systems specialist I have seen software bugs persist for 30+ years before being discovered due to the exact conditions needed to produce them arising. You can never be too careful with software that is controlling machines.

[@jothain]

It's quite disturbing that such device weren't rigorously tested before widespread use measuring radiation levels 🙁

[@patrikfloding7985]

At the time very few people in any engineering discipline understood some of the unique issues with software based systems, including most programmers. Testing would likely not have revealed anything as the problem surfaced when the operator did certain things at a certain pace (according to this video).

[@TeIegram_me_TheDavesGarage1]

HEADUP LES’T HAVE A CONVERSATION

[@jothain]

@@patrikfloding7985 I don't know. I think that's bad excuse. On such high safety level needing device every single possible scenario should've been taking into account. But it's bizarre that physically existing safety locks weren't there. Working in industry PLC programmers f'up all the time working in hurry and bit left handed. But equipment I'm involved doesn't doesn't hurt people if done wrong. Well in some equipment there exists safety relays to prevent door opening if machine is in production, even when program should handle it. But if there are such things in packaging equipment, it's absurd that radiation related devices where made in such haste. Especially considering how much money is involved in medical equipment.

[@mattilindstrom]

I've worked in a company in the MRI field, nothing therapy related just imaging. Still it's possible to e.g. overheat a patient through absorbed RF energy, and there are physical interlocks in the RF amplifier path. The Therac-25 case was in every engineer/scientist employee's onboarding orientation material, just to remind of past mistakes.

[@tinlizziedl001]

Concise and plainly stated. Well done! One of those sayings that really annoys me is, "Good enough for government work." I was a federal civilian welder in a navy shipyard - if people understood just how damned good our work actually had to be, particularly on SUBSAFE components, only perfectionists would use that saying.

[@tinad8561]

Perfectionism isn’t universal, sadly. Remember the metallurgist who got done for having pencil-whipped decades of strength test results on submarine steel because she thought the test standard was too extreme? We are fortunate that there was so much extremity built into the standard, because no amount of perfectionism downstream can compensate for that kind of confident incorrectness hidden in the supply chain.

[@halbos7637]

​@@tinad8561Very well put. Think of the nuclear power industry. Adm. Rickover was right in his management of the Navy nuclear power program. High standards in ALL aspects, human and machine.

[@NipkowDisk]

Yep. In government, we are presumed correct and therefore we have much greater responsibility to be so.

[@yeahgoood]

0:20 When Elvis gets THERAC'd

[@DarrenYung]

The Therac-25 incident is something studied in my computer engineering course in university. As a high school computer science teacher, I tell my students on Day 1 it's my job to teach them good programming skills and critical thinking because I do not want to die! I'm hoping my students go on to write great code and build awesome things and I want to set a good base for them to build upon. Yes code is written for computers and machines, but it's people who interact with it and people who are affected by what it computes. Something we must be responsible for.

[@TeIegram_me_TheDavesGarage1]

HEADUP LES’T HAVE A CONVERSATION

[@AlexKarasev]

The reason that code went into production as egregiously as it has is linked to the reason the individual responsible for it not only wasn't jailed or even fined; we aren't even allowed to know their name. When someone has had that level impunity, how long do you reckon they'll keep caring, realistically?

[@vicslive]

Dave, I found all the developers of the Therac-25, they must be working for GM writing the Infotainment system for their new EVs, as my Blazer EV for sure has never been tested in real world scenarios for flaws the infotainment contains. It is so bad that my car has had error codes from the day it left the dealership a year ago and no dealership has ever figured out how to fix it. Thank you for shedding some light on the importance of great code and better testing.

[@73vwkubel]

I’ve heard about this story before but it’s always interesting to revisit from different perspectives. This is one of those seemingly inexcusable events. I’d be curious to see the chain of decisions that led to this software control without redundancy. I worked in the elevator industry for a bit. Modern elevators are software-controlled and have been since the 70s or so but on highly specialized/hardened hardware and every piece of the software chain has an electromechanical backup of some sort. It was sometimes difficult to test design changes because we had to bypass 3-4 layers of redundant safeties to cause the “failure”, and even then, the components had such a large safety margin that almost nothing was truly catastrophic. That’s not to say failures don’t happen, but when they do, it’s almost never a systematic design issue.

[@rogerp5816]

Is it possible that the software engineer that was never identified is now working for Boeing as a senior quality assurance engineer???

[@perwestermark8920]

Nope. Boeing had great senior quality assurance engineers. The problem? They were too good. So they got fired. Because their found issues would affect the time plan. Managers wanted people who wasn't so strict when producing/evaluating all quality documents, and when verifying everything was done to spec.

[@wtmayhew]

That was 40 years ago, so hopefully the coder is retired by now. Not to give a pass to the coder, but race conditions in software can be very difficult to detect and duplicate for analysis, especially when they’re triggered by an anomalous state of peripheral devices. When I did mainframe work we had a GIGO (garbage in - garbage out) card deck of random unformatted data which we forced programs to process as input to see if the program properly handled totally unexpected exceptions. I’m not sure you could call it a race condition, but rather a cascading failure when the entire Bell System long distance telephone network collapsed for almost a day in 1990. The problem happened when a switching fabric experienced an overload. The exception was handled with a break statement which was intended to drop the thread of processing out of an _if_ conditional. The problem is that _break_ is only relevant to _case_ structures, but anywhere else in K&R C, _break_ is treated as a no-op. Execution simply continued in the _if_ leaving the switch locked up. That locked up switch caused load to transfer to other switches which in turn overloaded with the unexpected traffic until the entire system was gridlocked. The vagaries of C, an apparently inexperienced programmer, not enough code review, and a rare but possible condition conspired to set off a disaster.

[@Geek-A-Hertz8707]

boeing probably hired people based on everything with the exception of being good at their job.

[@apveening]

@@Geek-A-Hertz8707 As I understand it, having a (detectable) heart beat was and still is an absolute requirement.

[@duranamoescapist6969]

as a former STE, I am extremely happy to hear a dev say these things. Modern software development practices are falling back in the 80s direction, with eliminating QA, outsourcing dev, etc. Testing is perhaps even more important than development, and as the consequences get ever higher, testing should at least keep up. VCs, of course, push for 0 testing, and must be countered.

[@GTaichou]

I'm amazed that they did away with the mechanical safety fallbacks. My first four years of work experience was in nuclear power and for every electrical system we had a mechanical backup.

[@TeIegram_me_TheDavesGarage1]

HEADUP LES’T HAVE A CONVERSATION.

[@marknewellmusic]

Thumbnail choice needs some explaining...

[@rogerlevasseur397]

The Therac-25, the poster child of bad user interface design and bad software design meets death.

[@timeimp]

Taught this in first year computer science. Ethics with computers really makes you realise… Code kills. A tragedy for all involved 😢

[@JimmyJamesJ]

I’m a licensed professional engineer and I’ve never met or even heard of a software engineer who had a license to practice as a professional engineer or who even gave a though to how their work would harm the public. That’s why they write the EULA to reject all responsibility for the consequences of their work. All EULA’s basically state “We’re not responsible for anything, ever, under any circumstances, including gross negligence. Good luck.” Even where licensed engineers should absolutely be required to take personal responsibility for their work such as in aircraft control systems, this is not done. Reference Boeing MCAS. Given all this, I'm really not sure what you're talking about here.

[@jangelelcangry]

Medicine student: How do you cure Cancer? Therac-25: Kill the host.

[@andljoy]

I would love to think we have learned , but i work in IT in the medical sector so i know better.

[@patw1687]

Dave, Thanks for showing this case study. It was informative. This case is one of the case studies used in training System Safety and System Software Safety engineers. The case study points to the code reuse and the lack of hardware interlocks. This is the first I heard that they used a single, nonprofessional programmer. We know today it is possible for well designed software to contain flaws (look at the USMC's Osprey during test). Military and industrial entities want to minimize risk to their people and investments. Many put processes in place to minimize software risk through processes based off of standards (for example IEEE and MIL-STD-882). Design and code reviews are essential to this process. System Safety engineering got started with nuclear weapons and submarines, where minor flaws could be fatal. As and old Chief Petty Officer told me, "There isn't a safety regulation in the US Navy that wasn't paid for in blood."

[@TracyNorrell]

"I'm in this for the like and subs", proceeds to put unrelated eye candy in thumbnail for maximum click bait returns. You're better than that Dave.

[@DavesGarage]

Explain why it's bad. Maybe I have a blind spot. But I don't see why an attractive person in the thumbnail is a problem....

[@TracyNorrell]

@@DavesGarage sexy click bait is going to drag in views that you don't want. It damages the brand you've been building. You've done such an amazing job of being an authority on technology. I'd hate to see that reputation squandered. Ultimately it's your channel and you have the right to run it the way you please, best of luck.

[@nomore6167]

@@DavesGarage "Explain why it's bad. Maybe I have a blind spot. But I don't see why an attractive person in the thumbnail is a problem" - I know that clickbait thumbnails have become almost standard on KZbin, but there are reasons that people hate them. Speaking only for myself, I expect a thumbnail to be relevant to the video. Unless there is a Julie Gonzalo doppelgänger in this story, the thumbnail is not relevant, thus the only purpose it serves is to trick people into clicking on your video. That's another thing -- most people don't like to be tricked like that.

[@amcluesent]

I thought this would be about pagers!

[@paulstubbs7678]

Since when has Therac made a pager...

[@mlann2333]

I suspect the pager software was reprogrammed to trigger a small explosive on a specific text msg. Took some know how and quite a bit of planning in fairness.

[@wilsonj4705]

And now two-way radios aka Walkie Talkies. A bunch exploded today.

[@Fay7666]

​@@mlann2333 But like, did they have access to the _whole_ pager supply line? I wonder if this is more a Stuxnet situation, and they were just normal pagers that somehow had a way to weaponize (overvolt) their batteries?

[@_Mentat]

It's quite a sophisticated mod to add an explosive charge _and_ trigger it when a particular alphanumeric string is displayed - not received; displayed! And somewhere there must be an auto-dialer messaging each pager.....

[@SpamMouse]

Even the first code I wrote included sanity checks on entered data, perhaps comes naturally with a Fortran mindset !

[@BushyBrowsHD]

I work in industrial automation, I always conduct myself aware and knowing that both my life and other's lives are on the line when I'm working on or making changes to equipment, whether that be to the wiring or PLC programs. Never be complacent and do professional work you are proud of, never cut corners in our field.

[@TeIegram_me_TheDavesGarage1]

PM LES’T DISCUSS PRIVATELY.

[@KurtisRader]

The story of the Therac 25 is covered in the book "Normal Accidents" by Charles Perrow. It is a must read for every engineer, hardware or software, whose work has safety implications, but really should be read by every engineer.

[@Nick_1911]

And the scariest part is , that is not just "based on a true story" , it is a real deal and it happened with real people...

[@theantipope4354]

Dave, I'm your age & an engineer, & this story has horrified me since the 90s, when I first heard of it.

[@Meower68]

In "Digital Apollo," it's explained that the original X-15 had two sets of controls: one set of aerodynamic controls (traditional airplane controls) and another set of Reaction Control Systems thrusters. Seeing as how the X-15 left the atmosphere, you needed both systems. The pilots had to master BOTH sets of controls, making it considerably more complex. After having a hard crash which damaged one, they finally integrated the systems using Fly By Wire (FBW), such that one set of controls could handle both parts of the flight envelope. Why didn't they do that to begin with? Because pilots refused to trust their lives to FBW. They didn't trust the computers. They had to get over that. The LM, which Neal Armstrong piloted to the Lunar surface, was FBW. As you mentioned, NASA had very strict rules and processes WRT the development of that software.

[@petersage5157]

I'm a belt and braces guy. I like redundancies to have redundancies and safeguards on top of safeguards. Test it until it breaks, then test every fix to find out what the fix breaks. I wish this kind of history were only of the past, but quite recently a commercial airplane manufacturer got a bit of attention for simple spot check failures.

[@mattarnold198]

I hope if someone behind the development or management of it claimed "it'll never happen", they've severely rethought that. Making one's job easier should never be an excuse for a sub-par design and just because you use software a certain way doesn't mean everyone else will or should. I've worked in a company where a developer claimed "no one will ever right-click and select paste" when his attempt to disable pasting was short-sightedly limited to Ctrl + V and another that claimed "no one ever resizes their windows" to account for his poor CSS not scaling to the size of the window! Imagine if those people were developing medical equipment - would you trust them?

[@LMacNeill]

I remember reading about this back in the late '80s -- right when I'd started college, majoring in Computer Engineering (sort of a mix of Computer Science and Electrical Engineering). The idea of working on any system that could threaten a person's life still scares the hell out of me, even today, not that my job entails such a thing.

[@daflyinguy]

Great video Dave. I work in Healthcare IT and almost think you could replace ‘AI’ for ‘software’ when it comes to the vetting necessary for AI in healthcare. It’s one thing if Dalle draws a toad when you asked for a frog; entirely different when trying to offer possible medication suggestions based on potential disease state. We need to remember that AI in healthcare is there to aid in the process, not dictate it - it’s a tool.

[@YDV669]

"...combination of software errors, oversight failures, and misplaced trust in technology led to one of the deadliest medical device disasters in history..." So, the plot to Sword Art Online? But seriously, you'd think companies would've learned from earlier mistakes, but nope. I guess Stockton Rush's quote, 'At some point, safety is just pure waste' wasn't as unique as people like to imagine.

[@thesavo]

You are the fourth high-profile channel I have seen bringing this tragedy to light. I Iove that your rendition is unique, much like the others. For additional view points, see Kyle hull, low level learning and fascinating horror.

[@buggaboo333]

My grandmother suffered death from this. They burned her internal organs. She died from complications a few months later. I don't know if it was this machine or not, but same thing happened. We were told it was the tech's fault. My grandfather never sued the hospital, though he should have.

[@ChristopherHaws90]

I’m a St Software Engineer and I refuse to work on any software that could affect someone’s life. That’s just too scary for me. I can’t imaging ever saying that my code is error free…

[@stevetockey5913]

Dave: Another enjoyable post. We could have long conversations about what the term "Software Engineering" can, and should, mean.

[@TeIegram_me_TheDavesGarage1]

HEADUP LES’T HAVE A CONVERSATION.

[@SamuelLudden]

I work at a public aquarium with highly automated systems and rely on the same levels of care in our code and sustems design. It's truly scary how quickly and easily the house of cards can fall around you.

[@markmowbray1769]

Dave, makes me think about how much “old code” still lies in modern product, used by us in every day life?

[@scottydawg1234567]

IIRC some of his original Task Manager code is still in the current version. But I might be wrong.

[@TeIegram_me_TheDavesGarage1]

HEADUP LES’T HAVE A CONVERSATION.

[@HweolRidda]

This question makes me think back to the Y2K crisis. Code that assumed the first two digits of a year were "19" was likely to break in the year 2000. I know people who were fixing code written in the late 1970s, which was 20 years earlier. That code was sometimes written in languages that were only known by retired former employees, who made a fortune consulting!

[@gruntaxeman3740]

@@HweolRidda Lessons were learned. Some companies make big offers for information systems to government. To maximize profits, they just put there Oracle Forms, make things around proprietary CRM/ERP or other turd. The they make fortune later and stocks go up!

[@connorskudlarek8598]

I think Prime covered this before, too. Absolutely insane that 1 person had total authority over this thing. Would LOVE to see a collab between you two.

[@dgpsf]

Excellent video, Dave. A great reminder for all of us in software and related fields. Especially for those of us who weren’t around when this disaster happened.

[@shaunjackson6366]

I remember writing weather related software for the RAF Back in the early 1990s that converted air pressure from Imperial to metric as the data I received was imperial but the display needed to be metric. I had a small rounding error in my calculation which a squadron leader spotted, the air pressure was used to calibrate plane altitude and the SL explained my rounding error was the equivalent of about 25 feet altitude, not a lot you may think but important during landing. Suitably chastised, I corrected the code and learned to take software that affects lives (even indirectly) much more seriously after that. I was only 22, and no code or design reviews took place. Software engineering has come a long way since then thank goodness

[@TeIegram_me_TheDavesGarage1]

PM LES’T DISCUSS PRIVATELY

[@cwaldrip]

This happened at my local major hospital (Kennestone), in Kennesaw, GA. It's a very good hospital and its accidents like this that improve the safety in all hospitals, unfortunately.

[@HappyCat3096]

I worked in aerospace on the GPS satellite program for 20 years. The scientists often discussed stories of catastrophes in the industry that resulted from simple errors. One we often discussed was the Ariane 5 disaster. Basically, they reused code from Ariane 4 but it wasn't right for Ariane 5. BOOM. Fortunately we avoided problems on the GPS program thanks to rigorous testing and review.

[@raybod1775]

The real question is why the machine wasn’t pulled from the market after the first error was reported.

[@eadweard.]

The market?

[@belstar1128]

it was but it was already too late when they found out

[@andyevans2336]

First off, the developers refused to admit that they had a bad code… after multiple Doctors kept questioning the company, oh, and the Doctors were reluctant to admit that they were involved with the ‘problem’ as well.

[@SF-eg3fq]

dave is a man of culture, the thumbnail lol

[@wpuclark]

Ha yep, I have vague memories of this story from my CS ethics course. Definitely a tale to learn about for software engineers, but the thumbnail left me wondering what it had to do with the subject, lol. The AI generated vignette left me chuckling too. I guess Dave is trying to spice things up?

[@xoxo2008oxox]

Thumbnail got my view!

[@JoseJimenez-il5vs]

Considering how tragic this story is, I find your thumbnail in quite poor taste.

[@FranzKafkaRockOpera]

It's pretty bizarre. The vid had a more tasteful one initially, and the AI cheesecake isn't at all this channel's usual style.

[@DavesGarage]

So an ugly person in the thumbnail would have been better? Can you explain why?

[@rayers1000]

hm. i see the point on both sides. however, i do find it a bit risque unnecessarily. but we shouldnt let the image on the cover take away from a very well put together video.

[@FranzKafkaRockOpera]

@@DavesGarage The best way I could explain it is that I (and a few other commenters) feel that imagery that's designed to generate a prurient interest in the viewer cheapens the grim reality of the story being told. It's a question of tone mismatch as much as anything else.

[@martindooley4439]

Dave -- As someone who has donw real time software I dlove to hear your view on the Chinook Helicopter FADEC software. This was back in the late 90s early 2000s. Great content 😊

[@Capohanf1]

The patients should have done what a guy my dad knew did for his cancer treatments! My dad worked on a Nuclear Reactor. One of his and the other people that worked with him jobs was to "Refuel" the reactor by inserting new Uranium Fuel Pellets into it BUT first the old, now highly radioactive pellets had to be removed. They were pushed out of the reactor and into lead buckets that were then sealed. This one worker had been diagnosed with throat cancer. He went through Chemo but it did not work very well. On his next visit to the doctor, the doctor found his tumor has shrunk for some reason. It was finally revealed he had been exposing his neck to the old, highly radioactive, Uranium Fuel Pellets that were in the buckets before they were sealed. In reality he was giving himself Radiation Treatments! I understand he lived for a few years after he was made to stop his self-treatments.

[@dougkinzinger]

I love that Elvis is the subject of the pictures 😂

[@andyevans2336]

My dad had his first Therac treatment in Yakima,Wa. The next day, his chest swollen like a basketball, he passed away. Yes, it was a Threac 25………

[@hotlavatube]

Therac-25 was a case study for my undergrad Software Development class. There was a somewhat similar incident in 2001 at the Panama National Institute of Oncology in which 28 people were overexposed and at least 5 people were killed. In that incident, the machine was the Theratron 780-C Cobalt 60 teletherapy system manufactured by Theratronics Inc from Canada (same company that took over maintenance of Therac) but used a therapy planning software from Multidata software out of St. Louis. Apparently the software permitted incorrect forms of data entry which led to miscalculation of treatment times. I have no idea if the Multidata software directly interfaced with the therapy machine or if it was just used for planning. Regardless, people trusted buggy software and there weren't safeguards in place to prevent overexposure.

[@TeIegram_me_TheDavesGarage1]

PM LES’T DISCUSS PRIVATELY.

[@mcmaddie]

This is beyond the problem of the developer doing poor job. This was company wide issue. Wild that trying to blame operators for errors and not even checking their own product in this kind of situation. I'm working also on software field and none of the products I've worked with are putting user in risk and yet if customer complains about error/bug we'll investigate it and do not ignore it. But that was "Wild 80's" back then. This feels more like management seeing SW as 'cheaper option' for hw interlocks thus hiring the hobby programmer to do the job.

[@robert_the_great2842]

Great video Dave and thank you for showing us.

[@TeIegram_me_TheDavesGarage1]

HEADUP LES’T HAVE A CONVERSATION.

[@johnk2743]

One would expect that the software places the machine in an 'unsafe' state at the start and gradually with it's configuration set and checked more flags progress to a checked state. Only if all checks came out succesfully the machine should be allowed to enter a 'operational' stage. But please keep the physical safe-guards in the loop! Thanks Dave, this was really informative!

[@PeterRichardsandYoureNot]

This is bad, but it’s quite common. I was having lasik surgery and the was a room full of people waiting. The person ahead of me was rolled in. Then 5 mins later, rolled back out….without their procedure done. Huh? We all sat in wonder. 20 mins later I am escorted back. I asked what happened? They told me the machine that does the lasik would not accept their software programming for the procedure. Wait, what? And now you are putting me under and shooting a laser into my eyes? Yes, and be rest assured, also is ok.. uh, no.

[@Ayelmar]

Excellent story, Davy! In the course of my Software Development degree, I've used the example of the Therac-25 several times in essays, papers, and exam questions to illustrate how utterly vital it is to seek out edge cases and test them *thoroughly* and to document, document, document in every project, and most especially in any system that is health-, life-, safety-, or even just mission-critical. The Therac is a prime example of hubris and complacency leading to fatal cosequences, something we must fight against to keep it from happening on our watch.

[@TeIegram_me_TheDavesGarage1]

PM LES’T DISCUSS PRIVATELY.

[@Rorschach1024]

Gee, software in safety critical systems not being tested thoroughly? And written by non-professional programmers? Where have we heard about this before?

[@patrikfloding7985]

He or she got paid to program, so was a professional programmer. Pretty standard to be mostly self-taught back in those days.

[@TeIegram_me_TheDavesGarage1]

HEADUP LES’T HAVE A CONVERSATION.

[@AllenChildress]

Boeing managed to kill 300 with the same lack of testing and managers rushing a product to market. Boeing employees are striking now in protest to return those quality checks, but management is threatening them as a consequence. I'm afraid Boeing is going to continue to kill customers and astronauts.

[@SimonJM]

I'm a firm believer in the 'belt, braces, and piece of string in pocket' style of philosophy

[@_Mentat]

I heard of another such device with the much simpler problem that if the operator made a typo entering the dose and used DEL to correct it all was fine, but if they used BACKSPACE then the digits on the screen vanished but the number received by the control program contained _all_ the digits entered!

[@TeIegram_me_TheDavesGarage1]

HEADUP LES’T HAVE A CONVERSATION.

[@danondler8808]

Thanks Dave for this story highlighting that in critical systems where a failure in any component (software is also a component) could cause serious consequences. If fail-safes cannot be implemented then redundancy must be. When involved in designing equipment which safety is required you are responsible.

[@jwillisbarrie]

Thanks for adding actual captions for the Deaf and making video more accessible

[@ckturvey]

Thanks for sharing the story. I had not heard about the Therac-25 even when I working IT in a hospital back in the 90's.

[@lorddorker3703]

I won't say who I worked for but I was an electronic medical records dev. I found a device driver 'feature' on db2 that added a null terminator at the end of its buffer. Problem was we used rtf to store notes in the db. In some cases there were entire sections of chart notes just gone. Drs had no idea it was happening. I told management and I got "We don't take responsibility for FDA approved, that is on the Dr " . I quit and got out entirely. Thank you for shining a light on this issue!

[@TeIegram_me_TheDavesGarage1]

HEADUP LES’T HAVE A CONVERSATION.

[@TheEmbeddedHobbyist]

This was a case i had to look at during one of my software safety courses, did a spot of aircraft navigation code reviews and a few years later code reviews for a medical device as part of our FDA approvals. if i remember correctly there was another case where an error condition was a byte being non-zero, but they just added a '1' when an error was detected. every 256 errors the byte was cleared so you could have a lot of errors but still show as safe to proceed. This is why it's much harder to certify software than hardware. a big part of the certification is based on how you design the code and not just how it runs. FPGA code follows the same route as software in hardware certification as it's impossible to test an FPGA as there can be so many parallel processes running in side, hardware testing for fault conditions is next to impossible. The fun in working in industries where safety is everything and faults can kill. 😞

[@landroveraddict2457]

Dave you are an impressive and captivating story teller. I appreciate the time you take to make your videos.

[@geoffpool7476]

I studied the Therac-25 case as an undergrad in 2002.... thanks for reviewing this case! I've always wanted a more technical breakdown.

[@_Mentat]

My closest call was harmless. Me and colleague collaborated to write a print driver for a government document printer/binder. There were lots of options for colors, page sizes, orientations etc. Unfortunately we both 'handled' number of copies. We tested all the options extensively but always with one copy - didn't want to waste paper. We let it go live and overnight the agency requested 60 copies of a 100 page manual. The operators fed it paper for hours while it printed the 60^2 copies then at 3 a.m. phoned me to ask why it wasn't stopping? I said pull the power at the wall and I'll look at it in the morning. 😀

[@TeIegram_me_TheDavesGarage1]

HEADUP LES’T HAVE A CONVERSATION.

[@squirrelingaround]

Was that thumbnail really necessary? 🙄

[@RS-ls7mm]

It worked exactly as intended, so yes.

[@squirrelingaround]

@@RS-ls7mm Oh yeah and what did he intended with it? It would be nice that for once tech bros would understand that women work in this industry too and acting like wankers is just cringe, is not "fun" or "cool" in any way. I had respect for him but it seems this issue runs deep in the "culture" of men in tech industry, not beign able to see women like normal humans and the constant need to sexualize us.

[@RS-ls7mm]

@@squirrelingaround As your kind is so quick to point out, its not for you.

[@RS-ls7mm]

@@squirrelingaround How arrogant that you think everything has to be designed just for you.

[@romerogoon]

​@@squirrelingaroundI agree - the video is a bit of an abberation in terms of what Dave's videos are usually like, I.e. very good. This video went hard on the AI pictures - not necessarily just the thumbnail, but they detract from what was supposed to be a factual representation. The most egregious moment was the use of a stereotype sysadmin picture - long haired slim white dude fresh out of the 70s - when the script inferred nothing of the sort. Maybe lay off the genAI for a while - the videos are usually awesome😊

[@jimbeckert7946]

The software developer should not be responsible for determining the safety and reliability of the device. The device needs to be tested independently and software flaws identified and given as feedback to the developer. Having been involved in many product development projects in the past, I've never seen designers get it right the first time.

[@blakepace]

❤Thank You, Dave! ❤ Some people are sensitive to MRI, too. Chuck Norris' wife and some of my close family members also.😢

[@rhodaborrocks1654]

I've often wondered how the guy who wrote the original MCAS code for Boeing felt when it was realised that it was that system that was responsible for two fatal crashes, not that it was his fault, he did exactly what he was tasked with doing, but it must have been sobering to realise that his lines of code actually put the aircraft out of control.