The Computer Hack That Saved Apollo 14

[@StormSilvawalker]

Nasa safety check: we have a possible issue involving the "Burn Baby Burn!" command. Everyone: *turns slowly to that one guy who would call the engine ignition command that*

[@cdnarmymedic]

Disco Stu? I'll show myself out.

[@DrTssha]

Knowing what I know about the astronauts, they'd probably be grinning their butts off at that one. Granted, you were probably talking about the ground crew... :P

[@merinsan]

As a software developer, I'd have to say there'd likely only be 1 person who WOULDN'T be a suspect in that situation.

[@kubel83]

Storm Silvawalker hhehehehehe😂😂😂😂😂 made my day sir👍

[@timothymclean]

Please tell me someone there was familiar with the LAP-programmers' sense of humor...

[@mortcs]

As a hardware guy I think Apollo 13 was the greatest helldesk call in history. It had everything from bad reception, thermal hardware failures, cranky customers, computer reboots, hardware/process hacks and yes even a firey decent.

[@GigAnonymous]

It didn't have critical sections coded by THAT one intern.

[@scottmanley]

Oh yeah, Apollo 13 was an amazing achievement. And that's why most documentaries don't even bother to talk about glitches that happened on 14-17 because 13 was the natural climax of the story.

[@harriewheatley275]

Maybe a mini-series on this stuff? Sure, we could go research it ourselves but your presenting style and delivery makes it 10x more enjoyable. As a computer scientist to-be, this stuff fascinates me.

[@Snapshot01]

"Try SCE to AUX." What an obscure recall. It always makes my hairs stand on end when I hear that on the Flight Directors loop. It's an often forgotten glitch though. Love your work Scott!

[@b1laxson]

You forgot there was a virus. Wasn't one of the Astronauts sick too?

[@dewiz9596]

I remember this when it was happening. I was amazed, and confused at how they could “reprogram” at a Lunar “arm’s length”. At about the same time, I was working on a computerized typesetting machine (Photon Pacesetter), which had for-bit words. The machine developed a hardware problem with the character flash power supply. I was able to access the computer via a control panel, keying in values which slowed the machine so the flash power supply was able to recover for the next character. The machine ran at about 1/3 speed, but we got the paper out, and the next day, the repair technician replaced the power supply. I was so impressed with having been able to “fix hardware” with software, that I eventually made software my career.

[@lyingcat9022]

As a Programmer I love hearing about how you guys worked with these earliest of machines! Makes me feel nostalgic even though I wasn’t even born yet! Ha

[@keiyakins]

ugly hacks around broken hardware are one of the oldest traditions in the software world, and I don't foresee it stopping any time soon. Usually it's pretty easy these days, just pretend the broken feature doesn't exist, but sometimes you still have to get creative

[@JustWasted3HoursHere]

I saw a T-shirt once that had an octal joke on it: "Why do programmers get Christmas and Halloween confused?" "Because DEC25 = OCT31" Very clever, I thought!

[@5roundsrapid263]

That’s a good one! I remember a Far Side strip had a programmer writing out “0, I, I0, II, IV”! 😆

[@JustWasted3HoursHere]

​@@5roundsrapid263 Ah, I miss The Far Side. Gary Larson was great. He wrote a children's book called "There's a Hair in my Dirt: A Worm's Story" which was about ecology (but in disguise...). Did you mean "0, 1, 10, 11, 100"? Another programmer's joke that I saw on someone's T-Shirt said: "There are 10 types of people in the world: Those that understand binary and those that don't"

[@trollobite1629]

That's because there are 10 types of people in the world, those that understand binary and those that don't.

[@trollobite1629]

@@JustWasted3HoursHere dude you beat me to it 😁

[@JustWasted3HoursHere]

@@trollobite1629 :)

[@Adamzychu]

The only one appreciated tech support guy in the history of tech support guys.

[@RAFMnBgaming]

To be fair i ever went to tech support and it only took them 50 minutes to fix my problem i'd appreciate them a hell of a lot more.

[@davidcooper2589]

HASEnoncorperated "fix"

[@johngayceowens7236]

David Cooper Lol

[@IdleByte8000]

To be fair its not super easy to fix a problem unless it has a common solution. Because they gotta go through a list of solutions

[@a64738]

Tech support is always like "hold my beer" when they hear what problems I have... Then a hour or two later :( sorry I can not help. I always end up fixing it myself somehow anyway while not really knowing what I am doing as my computer problems is always unusual and strange. In DOS age of PC the tech guys used to say 99% of errors was user error and then they might have been correct, now it is other way around...

[@helge000]

The issue was actually caused by a small metal fragment left in the switch at assembly. In low gravity it floated around and could shorten the switch contacts on every acceleration. Source: Gene Kranz, 'Failure is not an option'.

[@ryanspence5831]

apparently the QC on the Apollo spacecraft wasn't very good...

[@jeremiefaucher-goulet3365]

What I've always wondered, is how the hell they know the cause. It's not like they brought back the LM to Earth for forensics.

[@5Andysalive]

Alternative source: Scott Manley in this very video. He just said it in a fancier way. @Ryan Spence There is no 100% guarantee that something as ludicrously complex as a spacecraft will never have a problem.

[@anondimwit]

Jérémie Faucher-Goulet how did you think they got back

[@keeganharris186]

@God they leave the LM in orbit around the moon after they get back in the command module

[@indianajones91]

Well researched video! In the case anybody wondered, on Apollo 15 and later the astronauts had a very easy way to tell the AGC to ignore the abort button and the abort stage button. That way the same issue couldn't have happened again. The AGC specialists developed a bunch of procedures for all the different switches that interact with the computer. So if any of these switches became faulty, they were prepared for it.

[@scottmanley]

+indianajones91 got point, there’s always things I miss...

[@492lautaro]

Scott Manley .

[@geocachingwomble]

indianajones91 a

[@HuntingTarg]

+Scott Manley I'm impressed by your ability to not only find facts, but find arcane footage and documentation that some of us would not believe to still be extant - you're not just a good programmer and avid space enthusiast, you have a talent for research. Outstanding job in this video!

[@n-wordjim1724]

It's called learning from your mistakes.

[@tmage23]

I knew I could count on Scott to use "computer hack" in its original context

[@o0alessandro0o]

IKR :)

[@RealCadde]

I will hack my way into your encryption by pressing random keys on my keyboard for 1 minute. Cue beeping sounds and lots of pointless graphics.

[@alienavatar7946]

That is odd. I was thinking that he used the word hack incorrectly. I could be wrong but I do not think reprogramming the computer you have been authorized to reprogram is hacking.

[@agarceran]

The original use of the word hack was to use a product in certain ways to make it do things it was not designed to do. In this way Mc Giver would be an Uber hacker.

[@RealCadde]

Even programmers with source code can do "hacks" in their code. While the general idea of hacking is to make something do something it wasn't intended to do. There's a different kind of hack that is defined as making something do a thing NOW temporarily so we can do a proper fix later. Sorta like your hose breaks. Instead of replacing the hose, you use duct tape to fix the immediate issue until the hose can be replaced. Or in the case of software, you find a bug (or broken switch as it may be) and you fix the bug as quickly as you can by "hacking together a piece of code" with the intent of returning later and fixing it. In fact, in programming you tend to comment "nasty" code with "// HACK: It may be ugly but it works for now" rather than "// TODO: Do nothing because I can't find any way to make it do what i want it to" It's another way of saying "Sorry, but this is the best i could come up with in such short time... I will return when i have a better solution". Which by the way was fixed in later missions where they had contingencies for issues like broken switches. Without the need to re-program the computer live.

[@noodlesthe1st]

"Houston we have a problem" *Indian voice* "Hello sir my name is Rajheed how may I help you today?"

[@TheLK641]

Have you tried restarting the computer ?

[@HungryHunter]

Yes and it dont help. Its still a mess. Its like this thing keeps going where it left... and is seen to be stuck at this task. How do i fix it? I running out of time.

[@didotb01]

okay so we have to install an antivirus would you please visit this website and download the file so we can remote control your computer and put syskey on it

[@azmanabdula]

In a tone of mistrust "You have a virus sir!" *we are fucked* "We need you to go to remote viewer" "Oh no"

[@almostfm]

Steve. His name is Steve.

[@LordFuturama]

A little bit OT but I recently notices KSP saying "flying safe" in the loading screen - i love it!

[@GreatgoatonFire]

Those woven memory units looks like some bonkers sci fi. Pretty cool.

[@TheRealMentat001]

It was called rope core memory and it was woven together by professional seamstresses. NASA engineers weren't good at sewing and it had to be done error free. Google image search rope core memory and you'll find images of an old lady weaving the wire into the memory unit.

[@GreatgoatonFire]

Oh cool cool. BTW I found this gem on wikipedia: "Software written by MIT programmers was woven into core rope memory by female workers in factories. Some programmers nicknamed the finished product LOL memory, for Little Old Lady memory."

[@almostfm]

I don't know the exact number either, but 3-6 months sounds about right. And if there was a problem and they missed the launch window, some of the memory would have to be rewoven and replaced. Not the whole thing, but stuff like the ephemeris data was only valid for the length of time of the mission.

[@RAFMnBgaming]

Give it a couple years and hipsters will be wearing those. Or have them on their moustaches or something.

[@mikemac2888]

GreatgoatonFire - Even more in-depth kzbin.info/www/bejne/b4qkaItrrMprpMk

[@Pieh0]

7:37 Manley.exe has stopped responding. End program?

[@LavenderSystem69]

DISK A:\ COULD NOT BE READ. ABORT, RETRY, FAIL?

[@sourpapasmurf]

Yes

[@julesscangojr.35]

Pieh0 Hahahaha this made me laugh out loud.

[@jfrtbikgkdhjbeep9974]

😳 damnnnn

[@mikestewart8928]

Great video! Don is still around and is still awesome. The listing shown in the video is from Don's own personal collection, which he has allowed us (at the VirtualAGC project) to have scanned over the course of the past year. We actually just finished transcribing the last of them, Zerlina 56, this week. Other great programs Don has given us include: * Aurora 12, a "fork" of the LM system test program, which has enabled us to greatly improve the accuracy of our emulator * Sunburst 37, an early development build of the Apollo 5 software * Sunburst 120, the flown Apollo 5 unmanned LM software * Luminary 69, almost the flown Apollo 10 LM software (69 Rev 2 is what flew; this one is missing a last-minute lunar gravity model change) * AP11ROPE, a 1970 remake of the Apollo 11 software (which has increased our confidence that Luminary 99 Rev 1, which we've had for a while, is what actually flew) * Luminary 116, the flown Apollo 12 LM software * Luminary 131 (shown in the video), an early release for the Apollo 13 LM (this underwent many changes before flying, however) * Luminary 210, the flown Apollo 15-17 LM software * Zerlina 56, a super advanced LM program featuring a variable servicer and P66 LPD, which you can read about on Don's site: www.doneyles.com/LM/Tales.html This was considered too risky, so NASA didn't improve the major changes for Luminary. He actually has more listings than every other source we've had put together! In an ironic twist of fate, Apollo 14 is one of the only two LM missions he doesn't have software for (the other being Sundance from Apollo 9). We're still searching for it though! You can find his collection as well as all of the other listings we've managed to have scanned in our Internet Archive collection here: archive.org/details/virtualagcproject Or if you like GitHub, they have all been transcribed and are in our repository here: github.com/virtualagc/virtualagc

[@oasntet]

"You have six seconds to key this in after the engines fire. And whatever you do, don't make a typo."

[@patrick6897]

And THIS is one of the reasons I subbed. So many well-researched space shenanigans and, of course, checking the staging

[@OriginalToma]

I truly enjoy stories from the history of computer science. This kind of ultra low level command of wire jungles flying space ships to the Moon has such an appeal to me. And Scott, you are an incredible KZbinr and an all around awesome human being (from what we see in the videos!). You just can't fake the passion for spaceflight that you have. Fly safe!

[@jackalovski1]

Was his first response "just switch it of and on again"?

[@duanevp]

No, it was to ask if it was plugged in.

[@Reactordrone]

They did that one on Apollo 12.

[@jackalovski1]

So which mission did they pull a cartridge out and blow on it to get it to work?

[@idiotpatroloftheduckempire7405]

Apollo 26

[@lawrencedoliveiro9104]

They did something like that on Apollo 10.

[@Voodoodrul]

THIS is the kind of content I love. Thanks Scott!

[@JeffreyBue_imtxsmoke]

Jesse Skrivseth me too!

[@elizabethwinsor-strumpetqueen]

+Jeffrey Bue get a room !

[@SamiJumppanen]

Agree. Not complaining about other videos, but I liked this a lot.

[@merinsan]

Good thing they found the problem before landing.... I can just imagine them all bouncing around on the moon, only to see the lander "abort" and fly off into orbit.

[@liljasere]

@DressedInRags pull the handbrake

[@ffggddss]

@Scotland Dobson You mean, drop anchor? Fred

[@elgoog-the-third]

As I understood it in the video, the system has the abort routine disabled completely while docked, landed and ascent.

[@BlueLightSpecial2023]

@@elgoog-the-third Actually, the AGS (Abort Guidance System) handled the normal ascent from the moon. So, in effect, a normal lunar ascent used the abort routines.

[@user-mp3eq6ir5b]

merinsan ☆ Scott does this very thing in Kerbal all the Time! Waiting for a Rescue Mission from Kerman...

[@kipparimies]

Well the guy at 1:40 certainly looks he was woken up midnight during his sleep

[@Keith136ful]

I know. Who was there with a camera to take his picture?

[@rickharper4533]

kipparimies I don’t think he’s ever had sleep...

[@ben-2368]

He looks like he knows what he’s taking about. Thin, kinda strange looking with glasses. Clearly hasn’t slept in days. He’s a BAD ASS engineer.

[@ben-2368]

Nerds can be Bad Ass.

[@allgrainbrewer10]

Ben - no. They just end up with all the money, and a hot wife.......who ends up sleeping the pilot. IE the real badass

[@LT7Racing]

I got to literally touch one of these computers as an intern at NASA this summer....pretty awesome

[@JohnSmith-ox3gy]

But could it run crysis?

[@leozeo1900]

John Smith Dude I ran crysis on the apollo computer and got 9000 fps at max graphics settings

[@jarr4287]

Actually you can't even play half life 1 on the apollo computer.. actually you can't play anything

[@MohammedAli-tb7zc]

@@jarr4287 could you play worn on the Apollo computer?

[@lordmikethegreat]

@@jarr4287 You could play tic tac toe!!

[@ricroz07]

2 things about this story is so relatable for me. First, in early 80’s U.S Navy mainframe school, we students were given a problem to make a machine language program to take input from a teletype and respond back with the same output. As I recall, my initial program took almost 100 lines of code. Not satisfied, I worked day and night to get it down to about 20. This system only had 16k of core memory, so efficient programming skill was critical. I was quite proud of my accomplishment, although most of my classmates probably didn’t really see the point. Secondly, with a deeper knowledge of machine level programming, I was later able to solve a problem with U. S. Navy communications between ships and aircraft. Because of this I was awarded a Navy Achievement Medal. So that being said, I truly loved this video. Thank you Scott!

[@josephgroves3176]

"This is what we call 'lies to children'" +1 for Discworld reference:)

[@CC-bu2gv]

Well they didn't reprogram the main code, but I would think you could say they reprogrammed the memory. So I wouldn't call it a lie. Sure it wasn't very specific but not a lie.

[@RolandHutchinson]

They changed only data (which they could alter) without altering any program instructions (which they could not do). So calling it re-PROGRAMMING could well be construed as truth-stretching. It's "programming" in the same way as "programming" your set-top box to record a show is -- which probably would not have been called "programming" at the time, if such a thing as a set-top box (or even a VCR) had existed.

[@VerixLin]

I don't understand 98% of this video but I'm still thumbing up for good reasons.

[@johnt.kennedy3856]

Verix Lin You must be smarter than me....I’m at 99% clueless.

[@fribigy47]

@@johnt.kennedy3856 well it’s rocket science

[@kainhall]

computer science major here..... that programming is so....simple.....yet elegant and powerful

[@ramonalonso4177]

Scott, you take me back to my youth. I was a member of MITs Apollo project (the older I get, the better I was). Your are right, Don Eyles was one of the essential geniuses, second only to Hal Laning, who architected runing 6 or 7 layers of interrupt on that little computer. My claim to fame was that (early on) I proposed and did the early work on the core rope (hard to break code you cannot change), as well as the DSKY concept with Verb & Noun and the scheme for getting the angle and velocity increments without having to execute a subroutine. IBM labeled that scheme "cycle stealing," which I thought was brilliant. So many people were "essential!" What a priviledge (and luck) for me to have been part of that project! Ramón Alonso

[@Hyxtryx]

Cycle stealing... Was that the counters that may increment/decrement between every instruction?

[@ComputingCactus]

Nice to see that even in the 60s programmers commented their code silly.

[@sergeboisse]

Yeah, in the 80's I worked on a MITRA525 computer (a french computer, 14 bits words, very limited ferrite RAM, but nice real-time multitasking abilities) using the LTR2 langage, and, because I was running out of memory, I had to program a process to basically suicide itself by overwriting itself with temporary data, and at the same time ask the scheduler to restart it after the temporary data had been used by another process. My comment on the code : "resurrection is not an option". I think that this code, eventually transpiled in Ada, then in C++, is still running in the Flight data processing system of the french air traffic management system. The comment, however, might have disappeared and no one ever will be understanding this piece of code.

[@mikemac2888]

Thank you for using "hack" in a story that doesn't involve food or home DIY repairs. Subbed.

[@hexadecimal7300]

"hack " was around before computers.

[@adamkerman475]

@@hexadecimal7300 still an annoying way to use the word

[@hexadecimal7300]

@@adamkerman475 Sorry I do not see how. I guess it just depends what you have been hacking?

[@zacksstuff]

Scott Manley: Apple engineer by day, KZbin rocket scientist by night.

[@mihailazar2487]

He works at Apple ? ... No ... That's NOT possible

[@justanoman6497]

To be fair, apple have competent engineers, it's the suits that are the problem.

[@CMDRSweeper]

God I hope not! How can you work on stuff that are better suited for making booze than running software?

[@anthonypoltes7413]

@@justanoman6497 look up a channel called Louis rossman on KZbin and you will realize the engineers are not as competent as you think

[@justanoman6497]

@anthony poltes I don't think you understand how engineering works. There is never a fail proof complex product. The way engineering should work is find problem, solve problem. Many apple product problem are multi generational, and that's rooted in the suits. Further, stuff like rubber inserts would have never been the primary choice of a fix for engineers, chances are suits pressured for the least costly solution. And that's the problem with apple products. Not that the flaws should never existed, there are actually rather few "new" flaws, no more than any other computer products. But that flaws persist for an unreasonable amount of time and across generations. The only "engineering" flaw that I see in apple product is when something unimportant fails, it magically take down the whole system. Though I'm not entirely sure if that was an engineering thing or a product of anti-repair culture that the suits are passing downward. So that might be suits, might be engineers, I'm not sure. And yes, I've watched his videos, that's how I know the nature of the failures, as opposed to just "many" failures. The multi generational failure is, in fact, one of his chief complaints about apple products.

[@Madsy9]

And on the 2nd place on the list goes to the team behind the Remote Agent for the NASA Deep Space 1 mission. The probe stopped working due to a race condition bug. Fortunately, the people behind the Remote Agent had written most of their code in LISP (which was unheard of at the time for code meant for critical space missions). So the probe even had a LISP REPL, Debugger and everything. Using the NASA Deep Space Network, they managed to talk to the Remote Agent's LISP REPL to get a dump of the program state, find the culprit and fix the bug using the REPL debugger. Had the team listened to the skeptics and used C and/or assembly instead (as was tradition), the bug would have been catastrophic and unfixable.

[@kallewirsch2263]

This is not an issue of Lisp versus C. As long as one can write to the program memory, things can get fixed. Even if the whole thing is written in C which - granted - would need a step down to patching the program memory directly to insert a jump into a formerly unused memory space and build up the replacement code there. After all, the computer is not interested in the programming language you use. All it does is execute commands, which are nothing more then numbers in some memory space. It doesn't care by which process the numbers (=commands) got there.

[@davidwuhrer6704]

*kallewirsch2263* You'd need commands in your compiled from C interface that allow you to peek and poke around in memory. You can't just attach a debugger. And you certainly can't just set breakpoints, or the software will just stop responding without a way to continue. And depending on the software you may not even have the luxury of malloc. It can be done. It has been done for most deep space probes. Famously the Pioneer-10 has been reprogrammed after cosmic radiation burnt a bit to 1. They had to read out the data store word for word to find the bug. The LISP REPL allows you to inspect live code, and change it while it is running. For free. You have to admit that is a lot simpler and more convenient.

[@johnfrancisdoe1563]

David Wührer But consider the risk of a broken LISP interpreter.

[@i.gusarov]

I'm curious if they use Rust now to avoid bugs like this

[@mikewa2]

“I’m sorry Don, I’m afraid I can’t do that”

[@ravingidiot]

2:18 - Just a minor note here which really doesn't alter your point much, but is worth mentioning: it's true from the perspective of software, most CPUs define an octet (8 bits) as a byte and a byte is usually the smallest addressable quantity of memory from the CPU. However, while there are definitely still memories which have an 8-bit data bus (and even some smaller ones, like 1-bit or 4-bit), the data bus for most modern systems tends to be the size of the machine word. This is done largely to increase throughput. For comparison, the 72-pin SIMMs used on most 486-based PCs from the early 90s has a 32-bit word size, while DDR2 has a 64-bit one. Additionally, it's common for address buses on modern CPUs to omit some of the low-order bits (again, as seen in the 486) since they're not needed and that could be the difference between needing a larger package. This is why misaligned memory access usually either incurs a performance penalty or triggers a trap on newer CPUs; for a write, you'd need to access the memory (or at least the cache) four times to write it. From a programmer's perspective, this is abstracted away, so it really doesn't make much of a difference at the end of the day. All the CPU has to do is fetch the word, then select the appropriate byte from the word, so the programmer never sees this. I just thought I'd mention it because it is (on a very technical and low level) a tiny bit misleading. Otherwise, this is incredibly instructive. The AGC is a work of art, and we're lucky to have the source code available so that we can get in the minds of the engineers who wrote it.

[@davidwuhrer6704]

They didn't have 8-Bit bytes back then. They had words, and they liked them big. They usually were not powers of two. They used four to six bits for printable characters, and they had to hack them out of the words byte by byte. And they didn't always align to word boundaries. 8-Bit addressing makes all of that easier. But as someone who prefers RISC machines, there is no Intel magic that abstracts the byte-size addressing for me. (But I cheat and use high-level languages 😜)

[@johnfrancisdoe1563]

David Wührer 8-bit bytes predate the moon landing. The AGC memory was 16 bit including parity, just like ASCII is 8 bit including parity. However 15 bits is a nicely readable 5 digits in octal. And researchers recently revealed another hack involving the precise behavior of modern RAM interfaces to slowly read unreadable memory without setting off alarms.

[@ClockworkAvatar]

I still find it pretty amazing that we made it to the moon using hardware less powerful than some wrist watches.

[@scottmanley]

Well todays wrist watches are more powerful that the computers I coded on for 2 decades.

[@tricitiesair]

ClockworkAvatar Also the complete design of the spacecraft and boost vehicles was done on sliderules and human calculations.

[@sbalogh53]

ClockworkAvatar ... You don't need powerful computers for maths or control. Most of the power of today's computers is wasted on bloated software and GUI interfaces. We used to perform very complex operations on 8 bit computers with 8k memory back in the mid to late 1970's

[@Hyxtryx]

The Apollo Guidance Computer had 72K of ROM, 4K of RAM, was multitasking with priorities, could run an interpreted language as well as native machine code, and had a faster clock speed than the 32KHz of the digital watches it has been compared to. Plus there were two of them on the mission. That statement comparing to a digital watch, which I first heard in the 80s, is a lie.

[@johnfrancisdoe1563]

Hyxtryx Well, some wrist watches now exceed that and need frequent battery recharging. 1980s wrist watches did neither.

[@robertahearne423]

Great piece. As someone who has been awakened at 2 am to fix a production system, I am super impressed with Don Eyles.

[@jtveg]

That was awesome. *_"Computer Nerd Saves Apollo 14 Mission"_* read the headlines. 🤓 Computer nerds make today's world go round. 👨🏼‍💻

[@charlesparmele]

I am a USN submarine veteran and watching this reminds me of the old ballistic missile submarines navigation computers I worked on during the late 80s and early 90s. Have to love core memory and making sure the currents were adjusted just right to prevent errors.

[@cgo225]

Many of the senior managers, guidance technicians, and programmers on Mercury, Gemini and Apollo had worked on ballistic missile programs such as Polaris.

[@coronapapi]

Wow!! Thanks for such a thorough video! I really enjoyed the programming vocabulary (verb/noun/etc)

[@VorpalBunnysRevenge]

Videos like this are why I like the Apollo program so much. Going to the moon wasn't about the technology so much as it was about the people. Dedication to detail -- "I know what that is!" -- and the preparation necessary to make it all work even with faulty hardware (which is the only kind of hardware humans can produce).

[@VladTchompalov]

wow, rewrite the flight computer in flight... What could possibly go wrong? Thx for sharing this stuff

[@KaiHenningsen]

Hah! I once rewrote the keyboard entry routine ... using the keyboard. And I'm talking replacing machine code byte by byte here. (I don't quite remember what problem I was solving, but it worked.)

[@TOMGOOTEE]

At least you had a keyboard. All we had was a row of toggle switches and lights.

[@tythagoras5787]

You Altair users and your fancy toggle switches. I had nothing to compute with but a pencil. My teachers always told me that I wouldn't have a calculator when I had to do math in the real world. It's been decades since I was more than a few paces from a calculator or other computing instrument.

[@VAXHeadroom]

I had to hand-assemble a SPARC assembly instruction and poke it into a flight computer this week :)

[@Hyxtryx]

Didn't rewrite the flight computer. Toggled a couple flags and faked-out the computer so it looked like a different program was running to whatever periodic task read the abort switch. Still a great achievement. But there was no re-writing of flight software involved.

[@SpineGevity]

As a non-coder, non-expert - I was able to track with the logical thought progression all the way through. Thanks for simplifying. It's amazing how much depended on this event! That guy is a rock star!

[@tomsvenkesen2476]

Matthew Brown m

[@wfobeor]

Should have put more struts on in the computer

[@dann9208]

wfobeor its moar struts not more struts

[@WilliamDye-willdye]

Thank you for posting a detailed explanation. I've heard about hacks in the lunar program software many times, but this is by far the most detailed account of a particular issue. I'm a programmer specializing in life-critical software, plus I'm old enough to remember Apollo 14, so this sort of story is of great interest to me.

[@Cynthia_Cantrell]

Apollo 14: "Houston, we typed in the command codes. The computer now shows 'Err 404'." Houston: "NOOOOOO!!!!!!!!"

[@fribigy47]

“Houston, we have a problem”

[@rty1955]

As a proud Bethpage resident, thank you for calling it the Lunar Excursion Module or LEM. My best friends father was in charge of wiring for the original LEM. Over 85,000 people worked round the clock @ Grumman during that time as they had other projects (E2C, F14, etc) they were working on. They also made two identical LEMs for the first moon landing. One was transported by NASA in the Super Guppy to Fl, the other one stayed in the plant. Everyone that HaD anything to do with the LEM was to be at the plant when it was descending, on the moon surface and when ot lifted off. After the successful return of the astronauts, the 2nd LEM was pushed into the parking lot and sprayed with camo green. It stayed there for over a year. I used to pass it every day when I went to work. I always thought they should have given it to the Smithsonian before they painted it green. Today the plant in Bethpage is closed down, and although a few buildings remain (one is called Grumman studios) the property now has apartment building on it. So sad to see this piece of history treated like this. There are plaques placed around the property as a memorial to this great achievement. My house was 1/4 mile from the end of the runway. I saw the guppy come in to pick up the LEM. also would regularly see E2C, F14 and many other aircraft land there. I have photos of a pilot of an E2C waving at me. Yes I could see them very clearly in my backyard

[@Zoomer30]

PNGS and AGS. PNGS was the Primary Navigation and Guidance System. AGS was the simpler and lower power usage Abort Guidance System. That is important because they were able to use the AGS on Apollo 13, since it used less power it produced less heat and therefore used less water.

[@JamieBainbridge]

This is one of the best KZbin videos I've ever seen.

[@wa9kzy326]

What an incredible story! Had me on the edge of my seat the whole time. As far as remote tech support for software, I was available for engineering support when the field guys got in over their heads. Dreaming up a software patch for a live telecom switch, after being drug out of bed, is challenging. The good news was that it didn't happen that often. Thanks for the very inspiring tech support story.

[@TimothyChapman]

And today the best hacks seem to be just setting a few flags or changing a few numbers that are checked by the software. Don't need to reprogram the software when you can simply change the data that it's using.

[@JanJannink]

Always nice to catch up on your favorite hacks! I love 4-5 line long computer programs.

[@e.rivera4251]

..this bring me memories of my days in college..Fortran , Cobol, punched cards and floopy discs!

[@Guhonter]

I never thought this would be as interesting as it turned out to be, I'm very glad I stuck around. Great presentation, superb explanation, kudos!

[@pjwarez]

"I'm the technician that will be helping you today... have you tried turning off and back on again?"

[@crocellian2972]

Outstanding. Thank you. Ed has been my hero for 40 years. I have never heard this story. Carry on.

[@rentacowisgoogle]

Don might be the world's most ethical hacker

[@Mernom]

I'm pretty sure that there are actualy hackers who's job is to try to hack into their client's systems and point out any security flaw they find so that it could be fixed. So, hacking is NOT a universally bad act.

[@mduckernz]

Oh yes, these are called "penetration testers" (a rather... ambiguous... name..!) Then there are security bug bounties, which is a variant activity where you simply ask the internet at large to try break in, and report any flaws you found for payment on a per-case per-severity basis

[@Mernom]

I'm assuming that the database has a potential new security system, but no important data, right?

[@fnors2]

Could be old systems, legacy pieces, new systems, sensitive data. Anything, really. Also, not running the tests on the actual databases you want to protect is somewhat silly : the live services will definitely have more problems than the test services. Security is generally based on the weakest link, which almost always end up being the old code people keep on using, even if you have fancy new stuff stacked on it. The sad part is that many groups will get a penetration test done, identify the vulnerabilities, check the box of "We did a pen test" on their list and then proceed to not patch the problems and deliver a flawed product. :\

[@LavenderSystem69]

The whitest hat hacker

[@michaelclark737]

Don is my idol for this. This hack makes every work-around I've ever done pale into insignificance.

[@cuzinevil1]

I think Apollo 14 demonstrates how 'by the seat of our pants' the Apollo missions were. In truth, a flawless mission would have been an anomaly.

[@Balance2097]

Great video!!! Oh, and look at how long they left the engine going once they touched down... BEAUTIFUL streaks of dust plume splintering out from where the engine exhaust hit the surface. Awesome.

[@TheEvilmooseofdoom]

The engine shuts down before touch down.

[@stargazer7644]

As soon as the ground probes hanging below the lander feet touch the ground, the engine shuts off and the lander falls the last 6-10 feet to the surface. The engine doesn't run while it sits on the surface.

[@corylynn8739]

Imagine jumping around on the moon and then the module takes off on it's own

[@KeithCooper-Albuquerque]

As a former coder, I love this story! Thanks Scott!

[@ThZuao]

Hey Scott. Could you do a video on earth's artificial ring? Maybe even replicate it in KSP. I've been wanting to ask it for a while. Always forgot. It's Project West Ford. Before the age of satellites and the communication nodes we now call "internet", it was feared that an atomic war would disrupt radio communications by messing up the ionosphere. So they used a rocket to disperse some 20Kg of copper wire "needles", that acted like dipole antennas, through medium orbit, in order to create an artificial ring that could bounce back radio comms. It worked, and another two launches were intended to create full coverage through the planet. However, the project was scrapped mostly due to satellite communications becoming feasible, but also due to concerns over space debris. I think it's a really interesting idea, albeit obsolete. You can use the ionosphere to send radio signals to someone literally on the other side of the planet, but it's finicky. This ring of crap solved it.

[@frasermanley9903]

There something about spending a tonne of time studying then taking a break to listen to your bro continue to educate me.

[@finnmcmillan5698]

"Have you tried turning it off and on again?"

[@canuckcorsa]

As an ex-IT guy (now retired) this was THE greatest tech support call!

[@wardedthorn6523]

"Burn, baby, burn?" Sounds like a variable name I'd use lmao

[@dannygjk]

'Do Until HellFreezesOver' One of my faves I used to use.

[@johnfrancisdoe1563]

Dan Kelly The classic is GO TO HELL

[@QqJcrsStbt]

Motorola PowerPC register bit; Enable In-order Execution I/O, labelled the EIEIO bit.

[@Edt_dcs]

This is such an incredible video simply because a simple mind like me actually understood what he was talking about from start to finish, which indicates how good he is at explaining things.

[@kennethflorek8532]

The story of a programmer who saved billions of dollars. And that was back when billions of dollars used to be a lot of money!

[@MCOGroupNews]

Kenneth Florek I mean it's still a lot of money today, certainly more than I have

[@kennethflorek8532]

It's more than I earn in a week! Once upon a time, before you average youtubers were probably born, people might consider some one rich, without them being a billionaire. OTOH when you are spending other peoples money, it isn't quite the same. As a Senator once famously said, I think in the '70's, "A billion here. A billion there. Pretty soon it adds up to real money."

[@robertharris6092]

Kenneth Florek your talking about this thing called inflation. Dollars from the 70s are still worth little compared to say the 30s

[@Justwantahover]

I wonder if he got a raise.

[@psychadous9403]

First video of yours that I've watched. You have an amazing story telling voice and the subtle accent doesn't hurt :) Keep up the great work!

[@bersig]

Did they try just rebooting the computer first? That's the first thing the "help" desk always tells you to do. :D Seriously though, for some reason this video made me miss the days when one person could grok the entire contents of the computer at the bit level. I can still remember the hexadecimal address for the entry points to various subroutines in Apple ][ and VIC-20 ROMs. "Call -155" ring a bell for anyone? :) (Or -151 to do it quietly.. and of course 3D0G to get back.)

[@almostfm]

I know you asked it as a joke, but they'd already determined that the problem was in the switch itself. Had it been an actual fault in the computer or the programming, they probably would have cancelled the landing, because who knows what other faults there might be. Also, restarting the computer would have involved reinputting at lot of mission critical data, not the least of which would be the "state vector" (a matrix of six numbers that told the spacecraft where it was and it's velocity in three dimensions relative to (in this case) the Moon, and the "W-Matrix", which speeds up a number of calculations

[@nathanglevy]

This is probably one of my favorite videos from your channel, thanks for making this!

[@L0j1k]

"Applied Cryptography" is one of the books on my shelf, too!

[@chrisanthonissen17]

Great video! I'm also super impressed how informative the comments thread is. It seems unusual for the internet.

[@marvinkitfox3386]

And this show us just how rushed the Apollo mission were. They had the time to design the mission to work, but not the time to consider and prepare for all needed exception scenarios. When designing a complex, life-critical piece of equipment, one needs to consider all possible failure modes, and have them documented. Something as simple *and likely* as a stuck/intermittent/nonfunctional control switch should have had pre-documented info on its effect on mission, possible workarounds, etc. Note that in subsequent missions this functionality was built in, no problem, at least for a subset of the controls. The problem is, that doing this sort of analysis and preparation takes time. a LOT of time. And with the schedule concerns of the moon landing deadline looming, they only had time to get the systems working, not smooth out all the burrs. It is scary when your first actual test of a complex system, involves strapping 3 humans to a large explosive device and hurling them at the moon.

[@johngayceowens7236]

Marvin Kitfox Not anymore, now we can do these tests in a few months. Thank you simulation software.

[@Sandeee]

It had to be rushed. There was no way around. Russia would have reached moon first, and all of NASA's money on apollo missions would have gone down the drain.

[@Sandeee]

Skylord8000 lol true. I didn't check the number. My mistake.

[@Maibuwolf]

Exploration and research was never even remotely part of the main purpose. If that were the case we would have already sent humans to mars. Dont kid yourself. The scientist may have wanted to do those things but the people in charge of handing out the funding would have never even coughed up enough money to build a facility much less launch any missions without the threat of russians getting a massive military advantage if we did not do so.

[@Hyxtryx]

Marvin: First test? Why do people speak when they don't know what they're talking about? They flew a bunch of times, testing things out. They even flew to the moon twice before that, and just orbited, didn't land. Did you know that?

[@jetzeschaafsma1211]

This is a fantastic video, and really captures the feeling of cascading side effects of quick and dirty bug fixes.

[@stjepanbrkic3215]

4:32 I never expected for Scott to give me a flip :(

[@rickmccutchan4419]

Pretty neat. I'd never heard this before. Listening to the commands to octal brought me back in time. I started programming in 1964 and used all sort of low level languages and bases like octal. And to one of the commenters, yes commenting was especially important on languages like assembler or machine language as no one would be able to really follow logic without them. Of course if no comments you could write a logic map to help you but it would be difficult.

[@JamesCoyle95]

Is there a reason they couldn't just disconnect the button?

[@St0RM33]

thinking the same thing..

[@moritzkockritz5710]

The problem isn't *that* the button was blinking, but *why*

[@agarceran]

I asume they had no idea if it was the button or something else. Also I doubt they wanted to start disassembling parts of the main computer in flight.

[@RAFMnBgaming]

Probably because exposed wires and spaceships get on like a house on fire.

[@mduckernz]

A house on fire with an atmosphere of pure O2, yes ;)

[@Sprucey]

WOW, Just WOW... Some guys are just exceptional. Simply no other words for it.. Thank you so much for the history lesson.

[@Goroh35]

Why didn´t they just F5 before the descent?

[@AlchemiconSilver]

This savegame is on a mode that disallows that from happening. Also, the universe's F5 key is broken anyways. *Yes, I know you were joking, but still.*

[@dann9208]

Because there were no F5 button in the apollo cockpit (Idk what theyre called but where the controls are)

[@JoMiMi_h]

@@dann9208 _woooosh_

[@rmfeld]

They surely did have a different view on what was an acceptable risk in those days ... Great Video! I love this level of detail!

[@1.4142]

The original machine had a base plate of prefabulated amulite, surmounted by a malleable logarithmic casing in such a way that the two main spurving bearings were in a direct line with the panametric fan. The latter consisted simply of six hydrocoptic marzlevanes, so fitted to the ambifacient lunar waneshaft that side fumbling was effectively prevented. The main winding was of the normal lotus-o-deltoid type placed in panendermic semi-boloid slots in the stator, every seventh conductor being connected by a nonreversible tremmie pipe to the differential girdlespring on the "up" end of the grammeters.

[@dp0813]

This was like Apollo 13 for code Geeks LOL! Great job with the video and the explanation... Somehow I followed the whole video without a hitch and understood pretty much everything. I found the schematics especially useful

[@yeetskeet691]

Scott flipped me off at 4:28 ;-;

[@CostlyFiddle]

Well Done Sir! LOVE your videos about the Apollo missions & tech. KSP videos are awesome as well! Thanks for all your hard work!

[@skyr8449]

So, basically a build in GameShark in Apollo 14?

[@johnfrancisdoe1563]

Lord Lima Bean Actually in the whole series. There's also a socket for a full size debugger, but the debugger was too big to bring onboard.

[@johnoehrle5973]

I knew who don eyles was when I was working at draper lab, previously called the instrumentation lab, and knew who his boss was, Margaret Hamilton. Draper lab was (is?) a heady place to work. Really enjoyed your video. Incidentally, Apollo 12 was hit by lightning on the way to orbit. I was in the SCAMA room when Houston called and asked if they had a go for TLI, trans lunar injection. Houston was told to turn the guidance off because it was trying to torque itself out of gimb lock, then do a P51and P52 to align the platform using the sextant. One orbit later the platform alignment was checked, the alignment drift was negligible and we gave Houston a go for TLI. I was on the fence on that decision but calmer minds prevailed.

[@ohger1]

"I know we're not really running the abort sequence Dave...."

[@ayeyefookinw0tm859]

John-Del I’m afraid I can’t do the abort sequence Dave

[@CurtBrown420]

Having taken the course "From NAND to Tetris (part I)" I feel I have a bit more insight into what you're talking about here. I highly recommend it to anyone with an interest in how computers work. It's really amazing what they were doing in the early days of the space program!

[@holmrekr]

it cant be tech support, they didnt suggest restarting it

[@aquaticllamas28]

holmrekR Was it next to a microwave?

[@JeepTherapy]

But the automatic retractable cup holder wasn’t invented yet?!

[@TheExplosiveGuy]

Nor mention if the power cable was plugged in...

[@Lukelins1]

Do you have the device in front of you?

[@johnfrancisdoe1563]

Penguins Yes they probably used the restart (common on the AGC), were instructed when to face the computer versus other landing instruments and did have remote viewing of the DiSplay/KeYboard active at all times.

[@johncashwell1024]

Great video! Gotta love that "Antique Computer Programming"! Seriously though, the computer input keyboard used onboard the Command Module & Lunar Module (not sure about what was on the Service Module) is a fascinating bit of kit. It appears to have really simplified programming for the astronauts, especially if they had to input code during an emergency. One note, however, I swear this video is twice as long as the 11:22 that it is supposed to be😁!

[@Marios5556]

Imagine what a misstype could do in that situation.

[@jshepard152]

Welcome to Venus!

[@seraphina985]

Well after each number is keyed in the enter key needs to be pressed so you could implement a procedure like this: Pilot 1: Reads out number Pilot 2: Reads back and keys in number Pilot 1: Checks display and confirms number matches before saying "ENTER" Pilot 2: Presses enter key Ok it's still possible for two pilots to make an error but the possibility is greatly reduced when you have multiple crew members acting as checks and balances on each other. Course this has failed to prevent someone programming the wrong waypoint into the FMS on a modern airliner and directing the flight computer to fly the plane directly into a mountain (Which it promptly did exactly as programmed).

[@henrychan720]

I mean you are basically just wiring a byte at a time to the memory so if you wrote it to the wrong location, I guess you could just re-write a correct one?

[@rlu1956]

Best KZbin clip for me for the year...thanks for sharing it. Awesome research.

[@Suedocode]

If the computer inputs were all in octal, were the "8" and "9" keys just never used?

[@scottmanley]

+Suedocode some nouns let you specify input in decimal.

[@marvinkitfox3386]

Oh yes, lets complexify the problem even more. If you want to run the thing in Octal, fine. It makes it easier for the hardware, and not *that* bad for the operator. But to then allow a hybrid of octal and decimal, where very often the flavor of the input cannot be determined by the value? That's just asking for trouble.

[@Suedocode]

+Marvin Kitfox I think some nouns *only* allow for decimal, whereas others are *only* octal. While I agree that it's inconsistent, within the noun list at 4:27 I'd imagine Noun 4 would be annoying to type in octal lol.

[@Hyxtryx]

Decimal numbers always displayed with a + or - in front of them and had to be entered the same way. Octal always displayed without the sign. Re: "Octal makes it easier for the hardware". Octal as opposed to what? Decimal, where a byte could only hold a value up to 99 instead of 255? No computer works like that, aside from some early calculators and digital watches maybe.

[@RolandHutchinson]

Tons of computers have had binary-coded decimal arithmetic built-in to their CPUs either as their only mode of calculation in the old days ("business computers") or as an alternative to binary calculation. There are even some (relatively useless) BCD-oriented instructions on current x86 CPUs.

[@marcpearce9121]

I love this! I work as a developer and do on call out of hours support for a fairly large client. That is stressful enough. God help being in that situation for an apollo landing!!

[@waxore1142]

Is that the old Xtronaut board game behind you? lmao

[@hipwave]

ground control to major Tom, commencing countdwn engines on, check ignition and my God's love be with you

[@ComandanteJ]

Wouldnt it be easier to just disassemble the switch?

[@milanmottl8983]

Walter Boxhead: Hold my beer....

[@CrisTooR]

the problem wasn't the switch, it was the computer, taking the switch off wouldn't do anything

[@therealxunil2]

No

[@feynthefallen]

Well, ONE among the MANY things I wouldn't want to do while hurtling across the surface of the moon preparing to land on it, is disassemble my spacecraft. Admittedly, it would have been easier if there had been a manual inhibit/override for that kind of switch, like you have to close two switches at the same time. There's a lesson to be learned here: Never construct anything truly important with just one switch. Even a twin-contact switch with the two contacts put in series would probably have avoided the problem.

[@brettatton]

Except if you made everything imaginable or otherwise redundant the weight penalty would have been a problem....it happened once on one mission...they found a work around (which weighs nothing!)

[@narfarkya2890]

+Scott Manley good work mate, loved it. I thought I already knew a fair bit about Apollo but a funny thing happened on the way to KZbin; I learned even more about it thanks to people like you. Cheers.

[@didotb01]

I want your applied cryptography book

[@scottmanley]

It's a classic - Bruce Schneier is a legend amzn.to/2wrwSYx

[@AndySuk99]

Greatest tech-support call? Maybe but have to compare to John Aaron. Not just Apollo 13 but the earlier lightning strike ("switch to aux"). Loved this vid. Awesomely Nerdy. Thank you.

[@brucepulver8358]

Lies told, Apollo 13. How to hack the Carbon dioxide filters. Man enters room throws 20 pieces on to table, team of 20-30 start work.....NO! Nope! did not happen!. One man on his own, did the hole how to square peg- round hole. When asked did you feel cheated? he replied Hell no! I'm that good, even Hollywood replaced me with a team of 25 people.