When Software Kills: Fatal Bugs in the Therac-25

[@keleighshepherd345]

Radiotherapy linac engineer here, this is why we have layers and layers of interlocks, both physical and software. It is a very intentional act to strip back those layers (we can and do, but very rarely and always taking appropriate measures to ensure safe practice) The safety innovations of today are writen in the blood of the past

[@ApeStimplair-et9yk]

there is many more about cancer and moneylooters to say

[@tonysolar284]

The only time those safe guards are removed/disabled is for testing/maintenance, at no other time should safe guards be removed/disabled.

[@keleighshepherd345]

@@tonysolar284 yup, and even then if we can do it without, we do that first The very first day of my apprenticeship I was told "overriding interlocks is an action of last resort. It is never to be done lightly and we test it has been reinstated before handing the machine back to clinical"

[@blevin591]

And of course, most of these can only be stripped away in service mode, which you are very intentionally told repeatedly to not treat any human or animal with.

[@blevin591]

​@@tonysolar284they usually can't be. Treatment, at least on the Varian machines, needs to happen in "record and verify" mode. No one should be in the room when it's beaming on in service mode.

[@austinsloop9774]

Some more details for those wondering what happened under the hood - It wasn't the lack of a beam spreader, it was the X-ray target that was incorrectly configured. We make medical use X-rays by hitting a [tungsten] target with high energy electrons. Most of the energy goes into heat, so >100x the amount of electrons need to hit that target to produce a similar X-ray dose vs delivering a beam of electrons for treatment. One reason this race condition was hard to catch was that it required the tech to erroneously select and electron treatment (machine begins to remove the target from the beam line) and reenter the correct setting of X-ray mode while the target was still moving. Since that console saw the target was in motion, it didn't check which way it was moving nor what state it ended in so X-ray outputs were applied with the target out of the beam line. The devices used to monitor the radiation produced by the machine do not operate well at 100x the dose rate and significantly under-measure what has left the machine, further increasing the dose to the poor patient. You normally can't feel radiation, but the current of electrons was so high the patients were essentially zapped like they touched a high voltage wire, that also shredded their cells and DNA. Unfortunate that these lessons had to be written in blood, but I'm glad to this day radiations devices have many layers of redundancy and interlock codes have gotten more descriptive.

[@ΝίκοςΙστοσελίδα]

Do you have a source for the computer-side technical stuff? I have read a lot of what's online about Therac-25 but never seen what exactly happened, e.g. that it didn't check which way the turn table was turning.

[@giornikitop5373]

so, the operator entered "execute small dose radiation" and the machine went "execute order 66". i've been hit with mains voltage some times but those electrons would have felt like hell's whip. man, those poor patients.

[@tammymakesthings]

@@ΝίκοςΙστοσελίδα the references section on the Wikipedia article about the Therac-25 have some good technical details. In particular, the articles by Nancy Leveson.

[@ΝίκοςΙστοσελίδα]

@@tammymakesthings Alright, thanks! Funny thing, I looked up Frank Borger (mentioned in another comment) and I ended up finding an IEEE publication I don't remember reading. The more info, the better.

[@f-s-r]

I wonder why on earth didn't the machine just have an "execute button" and THEN it started moving the target, check that everything is ok, etc. It seems like a LOT safer way to do it.

[@jazzmike_]

Glad you’re talking about the Therac, but I was hoping for a much more in-depth technical explanation of the code flaws from your expert perspective given how delightfully detailed you’ve been for many other technical topics on this channel. If you ever were inclined to do a part 2 of this with more details, I for one would be very interested to see it.

[@coldlyanalytical1351]

Anecdote about bugs: There is no such thing as bug free code. Once I was responsible for the safety of a software driven robot handling plutonium products. The developers were given 2 week to specify their smallish modules, 2 weeks to code ad deliver it bug free (?) .. and then I would use a Monte-Carlo rig that I developed to test their code on a very fast computer for EIGHT WEEKS. In every case I would find maybe 4 or 5 bugs in the first couple of days. Then 2 or 3 more in the first 2 weeks. Then it would go quiet .. BUT .. in a couple of cases a rare timing bug would be detected in week 6 or 7 after working through millions of random tests which passed. Just as well we found & fixed these bugs! (After this I used my Monte Carlo rig on many other less critical projects .. I found zillions of bugs in 'bug free' code)

[@trapfethen]

To take it one step further. There is no such thing as bug-free code because such a construct would require bug-free OS's, bug-free Firmware, and Bug-free Hardware. Based on issues we have seen in just the past few years when it comes to CPU hardware-level errors, it stands to reason that the conclusion that bug-free software is impossible is correct even for seemingly simple constructions.

[@JamieStuff]

I heard that there was a bug free version of "Hello World", written in machine language on bare metal, running on a COSMAC Elf, in 1977. Things were simpler then.

[@coldlyanalytical1351]

@@JamieStuff Oddly, I was an RCA 1802 programmer in 1978-1979.

[@wtmayhew]

⁠@@JamieStuff I remember the RCA COSMAC ELF and the 1802 processor. Subroutines were a pain to code. I worked for a lab which had a job developing a communication device to be deployed on a satellite. The 1802 was chosen for the processor because its CMOS architecture made it more impervious to effects of cosmic radiation in space. I believe the first CMOS Intel processor was a version of the 8085 in about 1985. We would have preferred the 8085, but it wasn’t going to be space qualified soon enough.

[@jess648]

@@trapfethen exactly, not sure such a thing is even possible because entropy

[@miscellaneousHandle]

I was a developer on a medical device when the therac-25 tragedy happened. it was a truly sobering event to those of us in the industry. you are absolutely correct that many lessons were learned, many regulations written, many procedures changed. but a handful of the core causes remain a problem to this day. More so in concurrent and real-time systems. My Hope around this is twofold. first that every developer realizes that these problems can occur regardless of the program and language regardless of the operating system, regardless of the quantity of unit tests. second, Nancy's seminal paper on the disaster should be required reading for every software engineer

[@gruntaxeman3740]

When doing something where people's lives are at stake, it requires very different mindset. Operating system: Preferably no operating system. I don't think there are many that can be used. Language: That is easy. When there is garbage collection cycle or memory allocation fails because of memory fragmentation, that can cause airplanes to drop or nuclear meltdown if used in wrong place. That safety critical scene is likely still C, Ada and perhaps very small subset of C++. Unit tests: Good for developing normal business application to cover something but they are nowhere near enough if human lives are at stake Software usually doesn't need to be error free. Quality, and process to achieve desired quality should be specified.

[@brianjuergensmeyer8809]

@@gruntaxeman3740 Exactly this - as a programmer with 30 years of professional experience (mostly in the medical field, no less), I'm constantly shocked at the number of "engineers" that we hire that have absolutely zero idea of how the full software stack that they're using actually works. They've been trained to see everything below them as a black box that is guaranteed to work every time and will magically solve problems with parallelism, threading, memory management, and data persistence. In a perfect world, I'd agree with you that we'd ditch everything but the application, and make the application its own operating system. In the real world, I'd very much like to see hardened, full real-time operating systems that can guarantee responsiveness in a given number of clock cycles. What I get is Windows. Even though my work is mostly EMR related (as opposed to medical device), I've just seen so much crap stacked SO HIGH that it seems a special miracle every time a patient encounter succeeds. In the medical field, if you don't understand the full stack from the machine code writing the registers all the way up the stack to the user interface, you're simply begging for trouble.

[@DarkLink606]

Development of the therac-25 was criminally negligent, to be sure, and thank goodness responsible devs such as yourself took notice of the incident ans standards were improved. I agree that bugs can happen with any language, but really, C language and compilers already existed at the time, was it really necessary to program in assembly? Machine code is much harder, error prone and not intuitive at all, even Fortran and Cobol are a breeze in comparison. Unbelievable the manufacturers of therac-25 decided to cut corners by hiring a single amateur programmer for such a difficult and critical task.

[@lorddorker3703]

I won't say who I worked for but I was an electronic medical records dev. I found a device driver 'feature' on db2 that added a null terminator at the end of its buffer. Problem was we used rtf to store notes in the db. In some cases there were entire sections of chart notes just gone. Drs had no idea it was happening. I told management and I got "We don't take responsibility for FDA approved, that is on the Dr " . I quit and got out entirely. Thank you for shining a light on this issue!

[@SpaceCop]

Thanks for touching on the "any idiot shouldn't be able to just call themselves a software *engineer*" subject

[@brianvogt8125]

Back in the 1980s & 1990s, I was a mainframe system administrator in a state govt. bureau agency. When our Director approved the creation of the first LAN, the task went to a member of the Clerical team because he was the only employee who had his own PC at home, and nobody in my team knew what to do. It took a lot of years to develop a culture of discipline in the "gifted amateurs" (as one of my colleagues referred to them). In 1977, I did a Fortran programming project in one of our client departments, and saw a piece of code writted by one of their Clerical officers. It achieved Integer Division by repeatedly subtracting the divisor from the dividend, in a DO loop counting the iterations! He had no idea that there were integer & floating point division instructions in a CPU.

[@Milosz_Ostrow]

The title "Engineer" is reserved in every jurisdiction in the United States. In order to bill oneself as an Engineer, one must pass professional exams and work under the supervision of an experienced Registered Engineer for a number of years in a sort of apprenticeship before acquiring a registration or license. However, industry has arranged exceptions in the statutes to allow them to grant "engineer" titles to employees who haven't spent a single day in a college classroom studying the discipline in which they are employed.

[@UncleKennysPlace]

@@Milosz_Ostrow OTOH, some of us dropped out of college, and ended up, say, programming for the DoD. It happens. You cannot discount "gifted amateurs". Is Bill Gates a "software engineer"?

[@HappyCat3096]

I worked for a company that thought programmers were simply overpaid clerks. Ironic when you consider that they were selling a software product. It was a hellacious experience dealing with these idiots.

[@perwestermark8920]

​@@Milosz_OstrowMany of the best software developers are mostly self-trained. Because it's a new science where there is a constant need for new learning. Requiring a huge amount of will to constantly improve. This isn't that compatible with a traditional engineering education.

[@gts2ludovicofratts404]

Hi Dave .. your videos are all amazing but this one is one of your best from my perspective. As an engineer in Canada and wearing the ring since 1990s, and going form structural to software careers, you articulated exactly what i tell myself, colleagues and younger aspiring programmers and engineers. This story should be part of anyone's studies in school.. university .. and work. I will share with many. Be well and thanks again for sharing on your channel.

[@JPs-q1o]

But how did someone manage to put WinDOS code on the Therac-25? OR was this Therac-25 code such sloppy hot garbage that it appealed enough to Dave to base DOS/Windows on it? YES! That would explain how he knows enough about it to make a video about it!

[@swittman9123]

The United States now has a variant of the Engineer's Ring. My diploma doesn't technically say "engineer" so I wasn't invited, but my partner was.

[@NyelaKearney]

I'm in my last year of engineering at UofA right now, and theres a mandatory risk management and safety class. So many computer / software engineers roll their eyes about safety. There was even that controversy a few years back with APEGA ordering job boards to stop using "Software Engineer" for non engineering jobs. All this to say that even today software is regarded as this ultimate safe tool, perhaps even more so because of the prevalence of the PC. Thanks for your breakdown Dave!

[@brianwithnell3570]

I used to be an AP computer science teacher. The students absolutely were given information about the Therac-25 (though with the naming of the machine/company removed to prevent unintended liability). They learned of other critical software failures that had far reaching consequences and the importance of software quality. That was a long while ago (my teaching) but I hope that those students learned that not just from a software basis, but also for any profession they entered.

[@FrederickMarcoux]

For those curious, the ring he mentioned is related to the Quebec Bridge in Canada, which collapsed twice during its construction. It’s a fascinating story.

[@hedlund]

Would you happen to know a good read or watch on that there collapsery?

[@SMA265]

@@hedlund lmk if you find one. Thanks

[@snoozevmw]

The US version is known as the Order of the Engineer, and is a stainless steel ring worn on pinky.

[@revcrussell]

I work for the company (AECL was legally continued on to be CNL) that made the THERAC, we went on to make nuclear reactors that are controlled by software code. It is so expensive to make quality code, and the real difference between when someone calls themselves a "software engineer" and being an actual software engineer.

[@nolifenerdwhohasnevergotten]

A family member of mine was friends with the woman who was injured by the Therac-25 machine in Hamilton, Ontario. Although she unfortunately later ended up passing away from her cancer within a few weeks, her autopsy revealed that her hip was completely destroyed and that she would have needed a full hip replacement. I remember the first time I heard about that and being absolutely horrified at the idea of being killed by the revolutionary machine meant to save you. AECL made a bunch of changes afterwards to try and make it safer but they didn't correctly identify the issue and more people ended up being hurt and killed afterwards.

[@rowdyriemer]

Software engineering has taught me a lot about being intellectually honest and humble. Between compiler errors, my own pre-review testing, code review feedback, automated test failures, and bug reports, I've been repeatedly reminded over the years that no matter how confident I can be that I thought through something correctly, I can still make logical errors, fail to consider certain contexts, etc. I'm glad I work in the video game industry. Heheh, the stakes are much lower.

[@andersjjensen]

Yeah. Getting told by the boss that "Some gamers are so upset they started a new subreddit about the problem, which is in your code" is quite different from "The plane suddenly went into a nose-dive and all 300 passangers and the crew died... the problem has been identified as being in one of your code blocks".

[@rowdyriemer]

@@andersjjensen Imagine how stressful it would be to write code for launch systems used for nuclear weapons!

[@kingofichigo]

Yeah, I could never write software people's lives depend on. I'm waaaaaay too clumsy and forgetful

[@edwardholmes91]

As always Dave, a great video. I'd like to add though, it was a combibation of a race condition and an overflow error... the code that checked if the collimator was in place would return a zero, before the beam could fire. To make sure zero wasn't present until the check had been performed, code in the setup loop would increment the variable in question, named Class3. The variable Class3 was 8-bit, and would therefore overflow when it reached 255. This meant that the beam could fire even if the collimator wasn't in place, when Class3 overflowed, approximately 0.4% of the time. Courtesy of Matt Parker's brilliant book: Humble Pi - A Comedy Of Maths Errors.

[@patlawler5532]

I was very interested when a co-worker doing embedded software told me about a document titled 'MISRA C' (Motor Industry Software Reliability Association, C Language). While some might read it as a simple list of good programming practices, I read it as a list of rules that were developed because something bad happened when it was done another way, and this new practice would address the issue. Much like the safety labels on step ladders, which were added to prevent another person from suffering that type of ladder injury.

[@jimrafert7372]

Very similar to the Federal Air Regulations. There's a saying in the aviation community, "The FARs are written in blood.". A large percentage of them are the result of lessons learned the hard way. Also, the primary reason for warning labels on ladders is not to prevent accidents, it is to prevent successful lawsuits when accidents do occur. It's possible to manufacture ladders that are many times safer than the ones you buy at the home store. It would just be impossible for a homeowner to afford, store, or even lift one. It's also why the user manuals for even complex power tools spend very few words on how to use the tool. Instead they are filled with "warnings" like "Do not operate with guards removed"..

[@Erik_The_Viking]

Thank you for bring this issue up. I work in medical devices, mostly in automated robotic surgery. Reading the Therac-25 is mandatory for anyone working on mission critical code that could hurt/kill people. This was why ISO-62304 was created, which is the software live cycle requirements for software used in medical devices.

[@PsRohrbaugh]

I have seen this covered by a half dozen KZbinrs and almost didn't click. I'm glad I did. You provided a much better representation of the low level issues than any other recount I've seen.

[@coldlyanalytical1351]

Excellent video. Sadly, having worked for decades on the software of nuclear and high security systems, I can safely say that fewer than maybe 5% of even highly experienced& qualified engineers have any regard or 'feel' for safety or security. Additionally I found that they CANNOT be trained to be safer or more security aware even if their weaknesses are noted. I also found that if you find major - but subtle - defects in the code produced by supposedly very senior engineers, they will vehemently assert that their code is perfect.

[@20chocsaday]

And they are willing to bet their job on that? There is testing and the results.

[@coldlyanalytical1351]

@@20chocsaday I would edit their code because they wouldn't. It would then pass the extensive tests I was using.

[@sheilam4964]

Thenin lies the ERROR. Humans have ERRORS. Not the code. Code is produced by HUMANS. We have yet to find a way to protect ours selves from OURSELVES. - - - - I wonder who produced our code? 😆😆😆😆😆

[@_Mentat]

Also, management won't allocate any time for a belt and braces approach where errors that get through one level of code get trapped by the next.

[@coldlyanalytical1351]

@@_Mentat For nuclear work, you get the time and money! For example, Canada used Formal methods to write a few lines of code which controlled the safety of their reactors. Those few lines of totally bug free code cost a fortune!

[@wa4aos]

Great video Dave ! If you are not aware already look up the radium girls who painted dials on clocks and meters. They had no idea how dangerous radium could be and considered it a cool, almost novelty material. They often wet the tips of their brushes in their mouth having no idea how dangerous this was. Then the stalling of the companies involved to recognize the problem and later the insurance companies didn't want to pay on the claims. Many of these young ladies spent the last months/weeks of their lives in agony as they were tragically coming to an end; so very sad !! They should NEVER be forgotten !!

[@jimmeade2976]

This story reminds me of a similar situation that happened in the rail transportation industry, where I spent the majority of my career. Prior to the 1980s, safety systems (called interlockings) for railroads and railways involved the use of relays all of which were designed and tested for failure conditions, with critical ones being guaranteed to be failsafe, thanks to springs and gravity. In the 1980s, the industry started developing electronic systems to replace the relays ... a large room of relays could be replaced with a rack of 2 or 3 microprocessors, an obvious economic and maintenance advantage. Since the relay circuits are, in essence, boolean expressions, the microprocessors and their software were written to handle similar boolean expressions. Each relay circuit was broken into its equivalent Boolean expression, and entered into the software as data for processing. Fortunately, a problem with this method was found during early testing. You see, relay circuits are in essence a large machine with massive parallel processing, while a microprocessor only does one thing at a time, even when many software threads are involved. This caused the kind of race conditions in the software that Dave describes in the video. Fortunately, this was found and rectified in two ways: processes were created involving multiple developers, validators and testers to ensure correct and safe operation; and some failsafe relays were maintained, just in case the software did do something other than intended. The industry has been using electronic interlockings for years now, with a wrong-side unsafe failure a very occurrence.

[@drozcompany4132]

This is one of the reasons why I have reservations about autonomous vehicles. The risk of loss of life is huge, and the automotive industry tends to rush development to be the first to market.

[@20chocsaday]

The insurance industry won't underwrite it.

[@Odin31b]

Agreed

[@chrisg6597]

The risk of loss of human life is also huge when humans are driving the vehicles. Would it still not be better to use autonomous vehicles (including bugs) if the death toll is less than human drivers?

[@20chocsaday]

@@chrisg6597 Try to solve the question of why an accident happened. You will have arguements going back to the mines where children pull rare ores out of the earth. As well as the years of testing software.

[@eadweard.]

No more so than with human drivers, I suspect.

[@capncoolio]

I was taught this case in a Software Quality Management course at uni, and it is singlehandedly responsible for my enduring insistence on quality process

[@wtmayhew]

I knew someone who worked for Picker International (now part of Marconi/Philips) in Solon, Ohio in the late 1980s and he told me about the Therac-25 linear accelerator incident back then and it sent shock waves through the industry. I actually interviewed for a job at Picker and they were using PDP-11s, I believe 11/34, to control the systems built in the department where I interviewed. I was pretty humbled by what I heard about the incident and I did not trust myself to be working on code that had the potential to cause grave harm if something went wrong. We’re pretty accustomed to PCs being reliable these days, but in the 1980s DEC reliability was good but not perfect. What would happen in the event of a computer failure, or worse a partial failure such as disk head crash?…

[@patrikfloding7985]

"We’re pretty accustomed to PCs being reliable these days" Ehh.. I wouldn't trust any PC to run anything critical without much lower level safety catches in place (external to the PC).

[@wtmayhew]

@@patrikfloding7985 Also - in the late 1980s or maybe early 1990s, I went with friends to Cedar Point amusement parks near Sandusky, Ohio. There was a ride called Demon Drop that dropped a car about 50 feet and then dissipated energy by rolling with the occupants flat on their backs down a short track with retarders. The was a hut at the bottom of the ride. There was clearly visible through the window a generic tower case PC, probably ‘286 machine, with no-name amber screen monitor. I couldn’t make out what was on the screen, but I’m glad I saw that _after_ going on the ride. In all honesty my guess is the PC was there collecting performance data and the ride was controlled by ladder logic - at least I hope.

[@arsenii_yavorskyi]

the fact that you had to change the thumbnail to *this*…

[@DavesGarage]

I guess it proves viewers liked that version better. What else do YOU think it proves?

[@johnandmegh]

Dave, people got into your channel because it felt conversational, with a knowledgeable guy who has a knack for walking through complicated situations and explaining them clearly. Gratuitously sexual thumbnails on a video about a tragedy, and weird AI images in the video itself, don’t feel like what “Dave’s Garage” should be about, IMO.

[@mqblowe]

Well said.

[@DavesGarage]

Can you explain how adding images to a script detracts? If it's the same content, but more interesting visuals, I'm not sure how that's a step backwards. But interested to learn!

[@CalcuClark]

@@DavesGarage I don't really understand what the *sexy* thumbnail has to do with this story. As for the AI images, it takes us out of the story when we are shown photos of obviously nonexistent machines and plastic looking people. Real historical pictures of similar machines and scenarios would be much better. Your audience tends to be one with a bit more technical knowledge, seeing utter gibberish displayed on a fake computer screen in a photo detracts from that technical and historical feel you have often with your channel. Not to mention, many people dislike AI images due to copyright and ethical usage issues surrounding it all right now.

[@marknewellmusic]

@@DavesGarage Allow me to explain Dave: Lose the tits on the inappropriate, sexually suggestive, insensitive thumbnail and regarding adding AI imagery for the sake of it, it's a matter of content for content sake, not always needed.

[@mlann2333]

He's right, not appropriate, bad judgement

[@tinlizziedl001]

Concise and plainly stated. Well done! One of those sayings that really annoys me is, "Good enough for government work." I was a federal civilian welder in a navy shipyard - if people understood just how damned good our work actually had to be, particularly on SUBSAFE components, only perfectionists would use that saying.

[@tinad8561]

Perfectionism isn’t universal, sadly. Remember the metallurgist who got done for having pencil-whipped decades of strength test results on submarine steel because she thought the test standard was too extreme? We are fortunate that there was so much extremity built into the standard, because no amount of perfectionism downstream can compensate for that kind of confident incorrectness hidden in the supply chain.

[@halbos7637]

​@@tinad8561Very well put. Think of the nuclear power industry. Adm. Rickover was right in his management of the Navy nuclear power program. High standards in ALL aspects, human and machine.

[@NipkowDisk]

Yep. In government, we are presumed correct and therefore we have much greater responsibility to be so.

[@73vwkubel]

I’ve heard about this story before but it’s always interesting to revisit from different perspectives. This is one of those seemingly inexcusable events. I’d be curious to see the chain of decisions that led to this software control without redundancy. I worked in the elevator industry for a bit. Modern elevators are software-controlled and have been since the 70s or so but on highly specialized/hardened hardware and every piece of the software chain has an electromechanical backup of some sort. It was sometimes difficult to test design changes because we had to bypass 3-4 layers of redundant safeties to cause the “failure”, and even then, the components had such a large safety margin that almost nothing was truly catastrophic. That’s not to say failures don’t happen, but when they do, it’s almost never a systematic design issue.

[@BushyBrowsHD]

I work in industrial automation, I always conduct myself aware and knowing that both my life and other's lives are on the line when I'm working on or making changes to equipment, whether that be to the wiring or PLC programs. Never be complacent and do professional work you are proud of, never cut corners in our field.

[@thesavo]

You are the fourth high-profile channel I have seen bringing this tragedy to light. I Iove that your rendition is unique, much like the others. For additional view points, see Kyle hull, low level learning and fascinating horror.

[@DavesGarage]

Yup, Kyle did a great job. Enough so that I had to go in a totally different direction, so did!

[@lunar0129]

I've heard about this before already and I still find this video to be informative and entertaining. Nice job, keep up the good work.

[@mattilindstrom]

I've worked in a company in the MRI field, nothing therapy related just imaging. Still it's possible to e.g. overheat a patient through absorbed RF energy, and there are physical interlocks in the RF amplifier path. The Therac-25 case was in every engineer/scientist employee's onboarding orientation material, just to remind of past mistakes.

[@patw1687]

Dave, Thanks for showing this case study. It was informative. This case is one of the case studies used in training System Safety and System Software Safety engineers. The case study points to the code reuse and the lack of hardware interlocks. This is the first I heard that they used a single, nonprofessional programmer. We know today it is possible for well designed software to contain flaws (look at the USMC's Osprey during test). Military and industrial entities want to minimize risk to their people and investments. Many put processes in place to minimize software risk through processes based off of standards (for example IEEE and MIL-STD-882). Design and code reviews are essential to this process. System Safety engineering got started with nuclear weapons and submarines, where minor flaws could be fatal. As and old Chief Petty Officer told me, "There isn't a safety regulation in the US Navy that wasn't paid for in blood."

[@GTaichou]

I'm amazed that they did away with the mechanical safety fallbacks. My first four years of work experience was in nuclear power and for every electrical system we had a mechanical backup.

[@vicslive]

Dave, I found all the developers of the Therac-25, they must be working for GM writing the Infotainment system for their new EVs, as my Blazer EV for sure has never been tested in real world scenarios for flaws the infotainment contains. It is so bad that my car has had error codes from the day it left the dealership a year ago and no dealership has ever figured out how to fix it. Thank you for shedding some light on the importance of great code and better testing.

[@Akens888]

As an industrial control systems specialist I have seen software bugs persist for 30+ years before being discovered due to the exact conditions needed to produce them arising. You can never be too careful with software that is controlling machines.

[@IMBlakeley]

Plainly Difficult is an excellent channel for covering many of these sort of cock ups, see also his video on this goof up.

[@bartoszjasinski]

My father was an electroradiology technician in a clinic. Many times as a child I was amazed with all that equipment from soviet times. It was like sci-fi movie in real life. I love that old design but scare as f*ck of radiation. If you as a kid see how that shit works, the noise when cathode are spinning, transformers buzzing, warning lights light up... damn its so cool and traumatizing at the same time.

[@halbos7637]

Like all those strange machines and devices in those old Frankenstein movies.

[@Bob3519]

You might like the “diode gone wild” KZbin channel. I think he’s in the Czech Republic. He’s had quite a few episodes demonstrating and taking apart Soviet era electronics. The guy is super smart.

[@belstar1128]

when i was a kid and i went to the hospital i wasn't even allowed to look at the machines i would ask doctors about how they worked but they didn't answer a very toxic environment .

[@bartoszjasinski]

@@belstar1128 that's sad

[@markmatlock9918]

One of the key people who helped find the software race condition bug was Frank Borger at the University of Chicago Medical school. He was active in the Chicago Area Real Time Society, the local DECUS group for the Chicago area. I met him at a number of these meetings but did not hear about his role in figuring out what happened to the THERAC systems until years later.

[@JPs-q1o]

But how did someone manage to put WinDOS code on the Therac-25? OR was this Therac-25 code such sloppy hot garbage that it appealed enough to Dave to base DOS/Windows on it? YES! That would explain how he knows enough about it to make a video about it!

[@Amstelchen]

@@JPs-q1o No idea what nonsense that is about. Moreover I have no idea whether it makes sense to try to explain things to you.

[@JPs-q1o]

@@Amstelchen Don't feed the troIIs, no matter how hungry we look.

[@philipcraig3266]

Very nice work DP. We give thanks for your films. Best wishes from Byron Bay Australia.

[@geoffpool7476]

I studied the Therac-25 case as an undergrad in 2002.... thanks for reviewing this case! I've always wanted a more technical breakdown.

[@shaunjackson6366]

I remember writing weather related software for the RAF Back in the early 1990s that converted air pressure from Imperial to metric as the data I received was imperial but the display needed to be metric. I had a small rounding error in my calculation which a squadron leader spotted, the air pressure was used to calibrate plane altitude and the SL explained my rounding error was the equivalent of about 25 feet altitude, not a lot you may think but important during landing. Suitably chastised, I corrected the code and learned to take software that affects lives (even indirectly) much more seriously after that. I was only 22, and no code or design reviews took place. Software engineering has come a long way since then thank goodness

[@keiwo_tritiyos_muketo]

Your best video so far. Graphics and.videos spot on 🙂

[@danondler8808]

Thanks Dave for this story highlighting that in critical systems where a failure in any component (software is also a component) could cause serious consequences. If fail-safes cannot be implemented then redundancy must be. When involved in designing equipment which safety is required you are responsible.

[@stevetockey5913]

Dave: Another enjoyable post. We could have long conversations about what the term "Software Engineering" can, and should, mean.

[@landroveraddict2457]

Dave you are an impressive and captivating story teller. I appreciate the time you take to make your videos.

[@dcc1165]

GREAT episode. Both informative AND entertaining! -- btw -- the programmer you showed looks like Allen Collins, guitar player for Lynyrd Skynyrd back in the 70's/80's :)

[@DarrenYung]

The Therac-25 incident is something studied in my computer engineering course in university. As a high school computer science teacher, I tell my students on Day 1 it's my job to teach them good programming skills and critical thinking because I do not want to die! I'm hoping my students go on to write great code and build awesome things and I want to set a good base for them to build upon. Yes code is written for computers and machines, but it's people who interact with it and people who are affected by what it computes. Something we must be responsible for.

[@AlexKarasev]

The reason that code went into production as egregiously as it has is linked to the reason the individual responsible for it not only wasn't jailed or even fined; we aren't even allowed to know their name. When someone has had that level impunity, how long do you reckon they'll keep caring, realistically?

[@Ayelmar]

Excellent story, Davy! In the course of my Software Development degree, I've used the example of the Therac-25 several times in essays, papers, and exam questions to illustrate how utterly vital it is to seek out edge cases and test them *thoroughly* and to document, document, document in every project, and most especially in any system that is health-, life-, safety-, or even just mission-critical. The Therac is a prime example of hubris and complacency leading to fatal cosequences, something we must fight against to keep it from happening on our watch.

[@kimlundgren562]

The thumbnail gave me a chuckle. A/B testing the cleavage to click ratio? Would love a video on the results! Great video as always.

[@DavesGarage]

Yup. I A/B tested it on a lark (it's an AI creation, and I didn't ask for cleavage!), and it performed vastly better.

[@kyay10]

@@DavesGarage I do wonder about the viewer retention difference though. Did people who clicked on the cleavage thumbnail still stick around? Please make a video about this!

[@robert_the_great2842]

Great video Dave and thank you for showing us.

[@daflyinguy]

Great video Dave. I work in Healthcare IT and almost think you could replace ‘AI’ for ‘software’ when it comes to the vetting necessary for AI in healthcare. It’s one thing if Dalle draws a toad when you asked for a frog; entirely different when trying to offer possible medication suggestions based on potential disease state. We need to remember that AI in healthcare is there to aid in the process, not dictate it - it’s a tool.

[@dgpsf]

Excellent video, Dave. A great reminder for all of us in software and related fields. Especially for those of us who weren’t around when this disaster happened.

[@johnk2743]

One would expect that the software places the machine in an 'unsafe' state at the start and gradually with it's configuration set and checked more flags progress to a checked state. Only if all checks came out succesfully the machine should be allowed to enter a 'operational' stage. But please keep the physical safe-guards in the loop! Thanks Dave, this was really informative!

[@vicmiller7191]

Great video, Dave. Thanks. Need to see more like this if possible. Again thanks and take care

[@martindooley4439]

Dave -- As someone who has donw real time software I dlove to hear your view on the Chinook Helicopter FADEC software. This was back in the late 90s early 2000s. Great content 😊

[@SirAzTop]

Very informative and insightful. Dave, as always, your wisdom and education do not disappoint.

[@fesaopilger]

Dave, you did an awesome content. Congratulatios for the quality of your videos.

[@mcw1593]

We had to study this in Human and Computer Interface class (HCI) over twenty years ago. Good to see people haven't forgotten about it. This story is also the reason behind the tile of the book "Set Phasers on Stun: And Other True Tales of Design, Technology, and Human Error"

[@johso87]

I see the whole movie for me David. Absolutely terryfying! Thanks for the story.

[@blakepace]

❤Thank You, Dave! ❤ Some people are sensitive to MRI, too. Chuck Norris' wife and some of my close family members also.😢

[@ckturvey]

Thanks for sharing the story. I had not heard about the Therac-25 even when I working IT in a hospital back in the 90's.

[@oleleclos]

"If an operator entered commands rapidly... the software could misinterpret the instructions." Reminiscent of the Horizon system run by Fujitsu for the British Post Office, which you may have heard of. One of many different problems was that it could freeze for various reasons, and if the operator then tapped the Enter key repeatedly to "wake it up" - as many did - the system would register this as multiple entries and the poor postmasters would end up with shortfalls in their tills. Nearly a thousand of them were jailed for fraud, countless more forced to repay money that never existed in the first place, and some committed suicide as a result of the financial and reputational ruin. All because everyone involved in developing and running the software ignored the professional and ethical duties you describe so well. "A troubling lack of formal software testing and a casual approach to safety verification... It had simply been assumed to work because it appeared to do so on the surface" perfectly describes that case too - and this was decades AFTER Therac-25. Makes me both mad, sad and ashamed on behalf of my profession!

[@jondrew55]

Great presentation on system safety and how things can go wrong. I'm going to recommend this during our next SW peer review

[@marknewellmusic]

Thumbnail choice needs some explaining...

[@yeahgoood]

0:20 When Elvis gets THERAC'd

[@DeanMichaelOwen]

Excellent video. I had read about this issue years ago, and it’s great to hear your take on it.

[@romangeneral23]

To me what I find more interesting is how the programmer has never been identified or seen again.

[@miscellaneousHandle]

I will shamefully admit to a morbid curiosity in trying to identify the programmer. his name is known to a handful of University professors that worked the investigation. ironically, I do take great pleasure in thinking of it as one of the first internet celebrities that has never been doxxed

[@romangeneral23]

@@miscellaneousHandle "I will shamefully admit to a morbid curiosity in trying to identify the programmer" Same here. For years I have always wondered how this person has avoided being named. It is fascinating....

[@eadweard.]

Seen again?

[@nua1234]

The programmer wasn’t the person most responsible for the problem, it was whoever decided to remove the hardware safety interlock. The same software bug was on the previous model, and no one got injured because of a hardware safety interlock.

[@nua1234]

@@romangeneral23The programmer wasn’t the person primarily responsible, it was whoever decided to not have a hardware interlock like the previous model (which had the same software bug, but no one was injured).

[@mkb_de]

I'm wondering what accidents will happen in the future because code has been taken from AI output, and only cursorily checked by some cheap developers in a "best cost" country.

[@GlennHamblin]

Thanks for resurrecting my favorite cartoon! I still want a copy of that clip. 😁

[@dubyaelle5533]

Malfunction 54. The unfortunate and ultimate BSOD. I thought from previously researching this, it was triggered by changing modes and applying before the physical filter had a chance to lock into place. If I recall, the first round of attempted mitigation was to remove the tab key from the keyboard to avoid the unintended workflow.

[@andljoy]

I would love to think we have learned , but i work in IT in the medical sector so i know better.

[@SpamMouse]

Even the first code I wrote included sanity checks on entered data, perhaps comes naturally with a Fortran mindset !

[@JT-kz7kq]

Great video. Any chance of a more in-depth technical explanation of the code flaws as a follow-up? I would be extremely interested in that - especially as I did a lot of dec assembler programming in the 1970-80s.

[@LMacNeill]

I remember reading about this back in the late '80s -- right when I'd started college, majoring in Computer Engineering (sort of a mix of Computer Science and Electrical Engineering). The idea of working on any system that could threaten a person's life still scares the hell out of me, even today, not that my job entails such a thing.

[@_Mentat]

My closest call was harmless. Me and colleague collaborated to write a print driver for a government document printer/binder. There were lots of options for colors, page sizes, orientations etc. Unfortunately we both 'handled' number of copies. We tested all the options extensively but always with one copy - didn't want to waste paper. We let it go live and overnight the agency requested 60 copies of a 100 page manual. The operators fed it paper for hours while it printed the 60^2 copies then at 3 a.m. phoned me to ask why it wasn't stopping? I said pull the power at the wall and I'll look at it in the morning. 😀

[@_Mentat]

I heard of another such device with the much simpler problem that if the operator made a typo entering the dose and used DEL to correct it all was fine, but if they used BACKSPACE then the digits on the screen vanished but the number received by the control program contained _all_ the digits entered!

[@UP_NS]

Excellent program. It's hard to imagine within today's corporate greed that additional costs will be funneled to ensure robust and secure software.

[@timeimp]

Taught this in first year computer science. Ethics with computers really makes you realise… Code kills. A tragedy for all involved 😢

[@stevephillips3952]

Outstanding episode!! Thanks for making this!

[@vcv6560]

I remember studying this one in school, its featured in Tannenbaum's Operating Systems text. It wasn't presented nearly so dramatically.

[@thomasjosephlamarque2927]

Ah “SIL levels”. Dave thank you for this. I have asked for this to be used as a briefing at my next technical briefing for Network Rail Wales and Western, UK.

[@geographicaloddity2]

The cat killing litter box comes to mind. IEC and ANSI standards for safety instruments systems and robotics safety took years to develop thanks in part to folks like you, Dave, making people aware of these issues. Even now, things that are well understood and have standards and industry organizations backing those standards face push back from managers and companies who either refuse to see the need for safer systems and testing prior to installation or they simply don't care to spend the money.

[@jwillisbarrie]

Thanks for adding actual captions for the Deaf and making video more accessible

[@KurtisRader]

The story of the Therac 25 is covered in the book "Normal Accidents" by Charles Perrow. It is a must read for every engineer, hardware or software, whose work has safety implications, but really should be read by every engineer.

[@GeneCash]

That's a hell of a book, I second the "must read"

[@connorskudlarek8598]

I think Prime covered this before, too. Absolutely insane that 1 person had total authority over this thing. Would LOVE to see a collab between you two.

[@duranamoescapist6969]

as a former STE, I am extremely happy to hear a dev say these things. Modern software development practices are falling back in the 80s direction, with eliminating QA, outsourcing dev, etc. Testing is perhaps even more important than development, and as the consequences get ever higher, testing should at least keep up. VCs, of course, push for 0 testing, and must be countered.

[@jothain]

It's quite disturbing that such device weren't rigorously tested before widespread use measuring radiation levels 🙁

[@patrikfloding7985]

At the time very few people in any engineering discipline understood some of the unique issues with software based systems, including most programmers. Testing would likely not have revealed anything as the problem surfaced when the operator did certain things at a certain pace (according to this video).

[@jothain]

@@patrikfloding7985 I don't know. I think that's bad excuse. On such high safety level needing device every single possible scenario should've been taking into account. But it's bizarre that physically existing safety locks weren't there. Working in industry PLC programmers f'up all the time working in hurry and bit left handed. But equipment I'm involved doesn't doesn't hurt people if done wrong. Well in some equipment there exists safety relays to prevent door opening if machine is in production, even when program should handle it. But if there are such things in packaging equipment, it's absurd that radiation related devices where made in such haste. Especially considering how much money is involved in medical equipment.

[@pianoman4Jesus]

Good summary of the AECL Therac-25 story, Dave. So it was written in plain Assembly. I wondered what language the software had been written in. I knew that it ran on a PDP/11. Same computer that a CT scanner I remember my father being trained on used to control it.... that was a Technicare 2060.

[@GeneCash]

Wow. The screen at 0:10 looks EXACTLY like letters when I get my migraines... That was so eerie suddenly seeing that! Edit: as a s/w engineer for 35+ years, the Therac-25 has been always in the back of my mind. Edit part deux: and it's no fun when your code turns to gibberish.

[@hotlavatube]

Therac-25 was a case study for my undergrad Software Development class. There was a somewhat similar incident in 2001 at the Panama National Institute of Oncology in which 28 people were overexposed and at least 5 people were killed. In that incident, the machine was the Theratron 780-C Cobalt 60 teletherapy system manufactured by Theratronics Inc from Canada (same company that took over maintenance of Therac) but used a therapy planning software from Multidata software out of St. Louis. Apparently the software permitted incorrect forms of data entry which led to miscalculation of treatment times. I have no idea if the Multidata software directly interfaced with the therapy machine or if it was just used for planning. Regardless, people trusted buggy software and there weren't safeguards in place to prevent overexposure.

[@itprankconfessions414]

I seem to remember this case in one of computer grad classes in the early 90s. I seem to recall the code had a 2 digit limit on a declared variable. When the operator made adjustment to the location of the treatment for the patient, the counter clicked up 1. If there was 100 adjacent the counter set to 00 and I guess this gave the code to give full radiation. I remembered this when I coded for Y2K. Great story will terrifying results.

[@_Meai_]

i've heard this story several times before with the dark historic story telling channels, however this was extremely interesting, being able to hear it from your perspective made it even more chilling...

[@rogerlevasseur397]

The Therac-25, the poster child of bad user interface design and bad software design meets death.

[@Zayka007]

I encounter these kind of hardware/software redundancy in cars, for example the throttle pedal contains two potentiometers which have different resistances and very secured, that's why they have a lifespan more than the car itself, automatic and semi-automatic transmissions have multiple hardware security layers too and I did intentionally trigger some fault like cutting wires while the engine running to see how the computer behaves and so far everything looks fine, I didn't go further testing each weird theory that comes to my mind because I didn't want to damage the ECU or anything else. The main issue here is when it comes to customer security, car manufacturers know how to design very good products like the throttle pedal, brakes or ABS, but any other thing unrelated to security it has to be poorly designed in order for the customers to buy that part again and again.

[@austinsloop9774]

Awesome breakdown video Dave. Quick note though, on your mock GUI (4:30) the energy would be in MeV for therapeutic treatments. 25KeV is crazy soft even for mammo radiography.

[@TheEmbeddedHobbyist]

This was a case i had to look at during one of my software safety courses, did a spot of aircraft navigation code reviews and a few years later code reviews for a medical device as part of our FDA approvals. if i remember correctly there was another case where an error condition was a byte being non-zero, but they just added a '1' when an error was detected. every 256 errors the byte was cleared so you could have a lot of errors but still show as safe to proceed. This is why it's much harder to certify software than hardware. a big part of the certification is based on how you design the code and not just how it runs. FPGA code follows the same route as software in hardware certification as it's impossible to test an FPGA as there can be so many parallel processes running in side, hardware testing for fault conditions is next to impossible. The fun in working in industries where safety is everything and faults can kill. 😞

[@j777]

I learned about this in College, so I was glad when I went to work on hot tub controllers for an internship and saw there was a dedicated circuit with it's own temperature sensor to cut off the heater in case of a problem.

[@martykong3592]

: ( SADLY SO TRUE! ! My youngest daughter works in QA, and had had her job eliminated now several times :( COST SAVINGS for the expense of NO QUALITY ASSURANCE ! Trying to send her back to school for aother line of work as there IS NO FUTURE for QA : ( THANKS MUCH for sharing and ALL the BEST! Cheers : )

[@ricardoveras3433]

This was brought up in one of my CS classes last semester!