Why Japan's Moon Lander Crashed Due to An Unbelievable Computer Bug

The investigation into why the Hakuto-R lander crashed into the moon last month after an otherwise perfect mission has revealed the answer: The software encountered unexpected terrain and didn't believe what the sensors were showing, so it started ignoring them.
iSpace Report
ispace-inc.com...
LROC images of the crash site:
lroc.sese.asu.e...
Follow me on Twitter for more updates:
/ djsnm
I have a discord server where I regularly turn up:
/ discord
If you really like what I do you can support me directly through Patreon
/ scottmanley

[@kjgoebel7098]

I'd like to see an episode of "Things KSP Doesn't Teach" about instrumentation. How air/spacecraft instruments work, their limitations and quirks, and how they can fail.

[@JohnWilliamNowak]

I'll second that. The Soviets had a number of uncrewed vehicle losses because they used ionic sensors to determine the orientation of the vehicle, which would fail on occasion. On the other hand, the gyroscopes aboard Apollo 13 held true despite being pushed well outside their comfort zone. Some sort of video about orientation sensors would be very enlightening.

[@BabyMakR]

Yes please!!! We need more of those videos please Scott.

[@ferdievanschalkwyk1669]

Another vote. I see it in formula racing where drivers are having to "fail" various sensors to address issues with the power train.

[@eekee6034]

Yep, me too. I think I'm aware of the issues already, but I'd like to know how different real sensors would be.

[@Spacedog49]

@@ferdievanschalkwyk1669 As a former Formula 500 driver, the fastest lap times were NOT the shortest distance lines around the track. A computer simulation takes the shortest distance, while the faster drivers took a slightly longer, but faster path that defied logic.

[@maurice_walker]

In their official debriefing, ispace actually admitted that it's primarily a (project / program) management issue, not an engineering issue. That gives me hope that they might actually learn something from this.

[@rspawn]

most underrated comment

[@curtislowe4577]

Life imitates art: a common problem in the Dilbert comic results in utter failure.

[@philkarn1761]

It's almost *ALWAYS* a project/program management issue, not an engineering issue. This was also true for Mars Polar Lander and for Mars Climate Orbiter (the one that famously mixed up imperial and metric units).

[@tomhenry897]

Don’t bet on it

[@SayAhh]

​@@Josh_728 Get with the program: in 2023, we measure things in bananas

[@robertbarron7660]

It's very interesting that this is almost the exact reverse of the famous 1201 alarm on Apollo 11. In that case the computer restarted and generated errors on the astronauts control panels. But because they knew that they were at the right altitude per the flight plan they had confidence that they were still flying correctly and Neil Armstrong brought the lander down safely.

[@warrenpierce5542]

Source of 1202 and 1201 alarms was traced to the rendezvous docking radar, used for rejoining the command/service module was inadvertently left on, at the same time the radar for landing, the only one needed for the decent phase was running. This overwhelmed the lunar module computer, but mission control knew it was still safe to land because of one man at Huston.

[@robertbarron7660]

@@warrenpierce5542 yes, when you go into the details then these are different cases. But in the abstract, in both cases the computer was confused because it got signals which were unexpected and didn't handle them well. In Apollo's case, the human was able to use additional information to recognize that the problem wasn't severe and in this case - there was no human.

[@larrybud]

@@warrenpierce5542 In Mike Collins' excellent book, he mentioned the 1201 and 1202 weren't exactly "well known" issues. Took a bit of "looking up" (quickly, albeit)

[@richardmogie9675]

That second antennae wasn:t inadvertently left on. I saw Buzz sheepishly confess, the engineers didn’t think the same way he did in an interview.

[@purnachandran87]

Just realized that manned missions are technologically easier (skill of pilot) than unmanned soft landings that are possible now due to the progress of software systems.

[@ReverendTed]

It continues to amaze me that we managed to safely land astronauts on the moon AND have them take off from the lunar surface and return home, several times. Obviously, having actual humans present makes a ton of difference, but the number of things that could have gone wrong but didn't is mind-boggling.

[@MarlinMay]

The brain is a wonderful flight computer. Lander: I'm going to land here. Human: Dummy, there's a rock the size of a McMansion there! Gimme manual control.

[@a4d9]

The first moon landing was saved by the astronauts: the automation on the lander was going to put it down in a field of big boulders.

[@unflexian]

think about it like this: humans have managed to control powered airplanes since the start of the 20th century, while autonomous aircraft have only just appeared in the last decade or two. humans are just that versatile

[@raifikarj6698]

​@@MarlinMay I am howling, when I pictured this in my head with astronaut Slapping their computer and called it dumb.

[@technocracy90]

One of the NASA research reports justified the cost and risk to send human astronauts to the Moon with the allegory says "Human brain is the most lightweight and easy-to-aqcuire real-time non-linear computer"

[@_Mentat]

My experience of being a software engineer is that the code has to be tested every time. It's amazing how often things that can't go wrong do go wrong.

[@hanskloss7726]

It is not a sw that changed but the parameters of the flight. You may of course argue that the sw was made for the particular landing zone which I do not buy. I may be mistaken as the video is the only source of my knowledge of the situation - sort of like this radar was. So you take a peek at the surface with radar and see this crater with it or rather a human having visual would have seen the crater - the landing module saw just a point on the surface which was 3km higher then the previous point it peeked at. I suspect what they would have needed to do is to have more points that radar is measuring especially from distance and make an average out of it or use some other technique to see where one is. When much lower this would also needed to be done to see if there is no big stone occupying part of the landing zone. I suppose this last thing was eliminated by assumption that the landing is going to be done on the flat empty surface by choice of the mission control. I suspect if they were landing on the water/liquid surface this radar error could only occur due to a massive tsunami - well no water surface and no tsunamis but hard landing. Interesting to know all this tho, aint it?

[@simonmultiverse6349]

Been there! Written lots of software... made some unbelievable bone-headed mistakes, which are all *BLINDINGLY OBVIOUS* in retrospect. "This change is SOOOOOOOOOO OBVIOUS that we don't need to test it" ... ha ha ha... this is when reality bites you on the backside, informing you that you definitely *DO* need to test it again.

[@simonmultiverse6349]

@@hanskloss7726 HA! Then you discover it's high tide instead of low tide... maybe you simulated it with mean sea level but a mile away someone opened the sluice gates and there was a large wave from the reservoir... etc.

[@roguedrones]

This moon lander crash is an example of space sabotage. Deliberate.

[@hanskloss7726]

@@simonmultiverse6349 low tide v. high tide does not cut it here - the surface is mostly flat still at least from a 5km perspective. The crater is a different story so you need to have many points possibly also a map? Not sure what is easier here but their method obviously failed. We know this is not a shame - we all have been there....

[@rhymereason3449]

It fascinates me that as you look at the history of disasters how many of them are ultimately caused by cutting corners to meet time pressures or budget targets. In this case you have to wonder (A) why the target zone was changed late in the game, and (B) why simulations with the new target zone weren't run. I would bet a dollar that engineers thought of it, but they were over-ruled because of time pressures or a budget target.

[@aarondavis8943]

Your question (A) is a great one. It could be that the new landing site could be reached with less expenditure of propellant or something like that. They thought it was a lower margin of error. Or was it the opposite? Was there a "better" more ambitious site with more interesting geography?

[@rhymereason3449]

@@aarondavis8943 It is interesting to speculate. IMHO "less expenditure of propellant" would fit into theory about disasters and cutting corners to meet budget targets. On the "better geography" thought... unless an asteroid suddenly impacted an area close to their original site, one would think that the geography question would have been settled long ago... the lunar surface is pretty well documented (at least the front side).

[@pierQRzt180]

Proverbs have a sort of statistical truth. "Haste makes waste" exist exactly due to that. The sad part is that seemingly we keep doing the same mistakes

[@rhymereason3449]

@@pierQRzt180 Yes it is sad to think about all the people who have lost their lives due to decisions on someone's part to save a few bucks by cutting corners. One of the latest examples appears to be that partial collapse of the apartment building in Davenport. Looking like the owner went with a cheaper contractor who would forego shoring up the building before proceeding.

[@Beregorn88]

And C) why there weren't redundant sistems with majority check before deciding to discard the most vital part of your data...

[@Nioub]

There was a similar bug in the LEM : if the module had flown above a circular-shaped crated of a certain size, the radar altimeter would have shut off all propulsion, probably leading to a crash. Fortunately the bug was never triggered (mainly because the onboard crew had taken over manual controls at this point) and was found decades after the landings.

[@alamrasyidi4097]

why are lunar manned missions not done anymore these days?

[@jessepollard7132]

@@alamrasyidi4097 Congress dropped funding, so NASA had no money for going to the moon (canceled the last planned 4 trips).

[@vast634]

@@alamrasyidi4097 No Soviets to beat

[@dr.cheeze5382]

​@@alamrasyidi4097 isn't nasa planning to go back? Starting with an (unmanned?) Mission sometime after 2024?

[@alamrasyidi4097]

@@dr.cheeze5382 so ive heard. but compared to the alternative of having to lose these spacecrafts to software error, i think "no soviet to beat" is a ridiculoua excuse. so i still really dont understand why lunar exploration has been strictly rover based these past few years...

[@ksbs2036]

About 30 years ago I had a single page photocopied from Computer World or some such industrial publication taped to the outside of my cubicle. On that page was listed the ten most expensive software defects (bugs). I was astounded when the most expensive defects caused hundreds of millions of dollars of loss. When you read the list the top five defects (again, multi million losses) you found out that they were all losses of spacecraft and/or their payload. Flight software is tremendously complex and a single error will cost you your whole vehicle and years of effort. Now that page would have to be scaled to near billions of loss I expect

[@a.p.2356]

Maybe not most expensive, but Therac-25 should be on that list somewhere. Ya know, because it ended up maiming and killing a bunch of people with intense doses of radiation.

[@RoryMacdonald-pfff]

There you go Scott - that’s an epic video right there. Top 10 most expensive Astro/Software defects.

[@o0alessandro0o]

@@a.p.2356 In a way, that is possibly the most expensive software bug ever; in another, it's quite cheap. Consider: we know for a fact that cars kill people, all the time, in every way, yet we do not ban cars. The value of a human being's life has been calculated, and apparently it's cheaper than you would expect. Electricity production has a cost measured in lives per TW/h. You can look it up. Biofuel has a cost of 12 people per TW/h. Solar is 0.44. Wind is 0.15, and new/clear is 0.07. The average American consumes 0.1-0.2 GW/h per year. In other words, over the course of your entire life you will likely kill less than one fiftieth of a person in order to keep the lights on. This does stack with the people you kill while driving, however - I'm talking about tyres particulate and excess death from pollution, not running somebody over. Ain't that grand?

[@travelbugse2829]

@@o0alessandro0o It's not easy to respond to that kind of information. I do know that training and regular checking of pilots contributes to a high level safety for commercial aviation (ignoring mechanical failures). For drivers, I reckon that similar processes should be followed. It would not be popular among the general public, but I have said for years that licenses should be graded, based on years of experience and how many training courses a driver takes. Governments balk at the idea, however, and go on putting up cameras and roadside radars, more draconian speed limits, but never addressing the fact that poor situational awareness, slow and inappropriate reactions, and limited skills are the biggest factors in car accident rates. But I'm going down a rabbit hole!

[@malbacato91]

Not strictly a bug, rather bad design; but implicit nullability - first introduced in ALGOL in 1965 and later copied into most programming languages - was famously coined by its creator as a billion dollar mistake. I think I read somewhere that at the time the estimate was quite accurate, but that was 2009 so by now it wouldn't be surprising if it is an order of magnitude too low.

[@miroslavhoudek7085]

In my personal experience, people insufficiently care about aerospace software. I worked in a software company that worked for ESA and we were always pretty much ignored (e.g. in all presentations of our local space agency). But when some other company made a screw for a satellite, it was plastered all over their presentations. There were literal delegations going to take a look at the space-screw-producing machine. Such an interesting visit, you see, to a hall with machining equipment, clean rooms - that's the "space stuff" in minds of people. Something you can touch and see. How do you brag about a company with people sitting at their PCs? Nobody cares. Even if these are the guys whose work ultimately decides whether these magical screws end-up doing something or are splattered over the moon. I don't care about the publicity but it's the mindset. Everyone focuses on the aluminum this and titanium that - and software is always the afterthought. We can change that anytime. We can even send an update to space ... so, why should we think about too hard? Bam!

[@deang5622]

Good point. And I think it is because it takes a higher level of intelligence and technical knowledge to understand software systems and the media and others can't understand it. You only have to look at any news article published by the main stream media, on television, in newspapers and you will see the errors the journalists make, the incorrect use of terminology, the lack of detail and you walk away realising the news article has told you almost nothing.

[@jayasuriyas2604]

oof

[@windywaz]

Boom! As a retired architect for space sensor payloads, I can say you are spot on. I watched management spend all sorts of money on convenience tooling but if SW wanted licenses for software production and testing tools, oh God, you got run through the gauntlet. So how many times must a company learn these lessons? Simple, once per program.

[@Mernom]

It's the same attitude all over the place. Games no longer ship out as completed projects... 'we can just patch it later'. Mamy other fields also do shit like this.

[@B_dev]

Software in general too

[@JeffreyCherry-v6i]

Another outstanding episode Scott! Being a software safety engineer for the last 39 years, I have to agree with previous comments that point out this is not a software bug, but more of a people problem during design, testing, management, etc. I believe the first Ariane 5 launch was a similar issue where the software worked perfectly per its specifications (from Ariane 4) and doomed the flight to failure. Like in this case, proper testing would have prevented the, expensive, tragedy. Also wanted to give a shout to "How To Destroy Wayward Rockets - Flight Termination Systems Explained". My 39 years were all spent on Range Safety Software with the last 13 years working on autonomous flight termination systems. That was another outstanding episode! Keep up the awesome work!

[@Icowom2]

Pop op o99⁹9th kiwi's😊

[@xGOKOPx]

It is a software bug though. People problem is that the bug wasn't caught

[@vast634]

Have you ever experience a flight termination system not working instantly, but 50 seconds late, as with the starship launch?

[@JeffreyCherry-v6i]

@@vast634 Depends on the type of Flight Termination System (FTS). For solid rocket motors, they use a shaped linear charge that opens the casing and exposes the fuel which burns up quickly in an impressive display. (I think Scott mentioned that in his previous video.) For chemical fuels, things are different. You have more choices. The basic idea is to stop thrusting the vehicle so it falls into an unpopulated area, such as a broad ocean area in the case of SpaceX. Based on the video of the flight, the FTS worked properly and detonated explosive devices that created holes in the fuel tanks. That reduced or stopped the fuel flow to the engines. The FTS did its job. After that, it's all physics. If the fuels are hypergolic, they will combust on contact and you get a near-instant explosion. Otherwise, you need combustible fuel, oxygen, and an ignition source. Guessing, it took about 40 seconds before the three elements came together in the right quantities in the case of SpaceX. An FTS doesn't need to create an explosion. Rather than connect to explosives, the FTS can connect to fuel valves that terminate fuel flow.

[@JeffreyCherry-v6i]

@@xGOKOPx I understand your perspective. My point is that the bug should have been avoided during design or implementation, and if not, then detected during development testing. Find and correct all the bugs before deployment. Since their development testing failed to react properly to "unexpected terrain" (kind of a silly term considering the moon's terrain is pretty stable), the people failed in the software development cycle and left in a failure mode (i.e., the bug) so it could be exposed during execution. The software did what it was designed to do so it worked properly. The people failed to account for something. The same thing happens with hardware but folks don't usually blame the hardware. The failure of Galloping Gertie wasn't blamed on the bridge. The people who designed and built it were blamed for not accounting for potential wind loading.

[@sharizabel2582]

I flew fighters for over 20 years. The Kalman filter was the bain of the navigation and bombing solution. It would actually discount most of the updates I would insert. It thought it knew more than I did … it didn’t.

[@Papershields001]

I feel such compassion for the Hakuto-R team. They are going to accomplish it!

[@serronserron1320]

I hope that they can make a new one and landed on the moon the next few years

[@emileriksson76]

I watched the landing live stream and I felt so bad for them. Their nervous faces really hurt me too. I bet they do it next tie!

[@abarratt8869]

They may not accomplish it. Very often such incidents reveal a whole load of issues that have been swept under the carpet, and the necessary organisational change required to address them all can easily break a small team / organisation. Even big companies can be killed by this. This is what is going on in Boeing right now. They caused the crashes of two 737MAXes and killed people. Since then they've tried to institute root and branch reform of how they run their business. Yet, they're still having problems. The most recent one was a fuselage manufacturing defect (they were building them wrong) that had gone unnoticed for approx 700 airframes (yep they're flying, possibly with Southwest today!). Fine, they've found it, repairs needed, not immediately dangerous, but cannot be ignored. Trouble is the manner of them finding it was accidental; someone was in the right place, at the right time and realised what was going wrong. The issue is that, if despite the introduction of a root and branch reform about how they approach quality (= safety, reliability) they're still finding major issues by chance, then the root and branch reforms are junk and are not working. They should be finding such problems as part of a systematic continuous improvement process, and they're not. So the bet-your-life question is, what else have they missed, given that they've essentially admitted that they've not been looking hard enough? It's similar with 787 (fuselage barrel joints), brand new 737MAXs with FOD and rodent damage, etc. This suggests to me that Boeing are in no way adequately reformed following the MAX crashes, the problem most likely being in the senior management who never understood it before and are still there today. It's worryingly possible that they're going to make another fatal mistake. Ok, the FAA is now (belatedly) keeping a much beadier eye on Boeing, but they can't see and check everything; certification engineers / inspectors are not there to do basic QC and basic QC improvement. The Hakuto team's best bet, if they're to try it again, is to just fix that one core issue and try again, and do as much simming as they can muster. Unlike Boeing, crashes are just disappointments and money.

[@99guspuppet8]

❤❤❤❤❤❤❤❤❤❤ Yes they will succeed…… After they spend a lot of someone else’s money……… Let’s all go to Sugar rock Candy Mountain

[@thePronto]

But they launched knowing that their testing was invalid. Kinda like practicing parachute landings in a field, then jumping over water. I hope they don't ask me for a donation, because polite refusal often offends.

[@hjalfi]

There's an argument to be made that if a sensor is critical enough that if it fails you're going to land on non-existent terrain 5km up, then you just assume it won't fail. If you handle failure gracefully but then don't have enough data to avoid crashing, what's the point of handling it gracefully? Of course, ideally you'd have a backup. Like another radar, or GPS, or a video camera capable of estimating height using machine vision and a map, so you can sanity check it. The next best thing is just have a map: the vehicle knows where it is, so if it knows the terrain it can estimate what the radar values _should_ be, so instead of going 'eek, a delta of 3km in ten seconds is clearly wrong' you go 'the radar has shown a delta of 3km in ten seconds, what does the map say the delta should be? Right, 3km, moving on'.

[@stoic.little]

You can have a video camera that is very good at finding the distance by using phase detect autofocus, same principle as a rangefinder.

[@driedurchin]

I work in flight software and you're right. At a certain point if a system is so critical and irreplaceable you just have to trust it won't fail because as you said, detecting the failure isn't helpful if your SOL.

[@Spillerrec]

There is an argument to be made that if a $90 million project can go up into smoke due to a single sensor failure you have an expectation that it could potentially fail, you should really have some sort of redundancy even if it is unlikely. Or some other form for backup plan. The question is if it was actually considered if this sensor could fail, or if it just used the same behavior failure detection and handling as any other sensor without further consideration.

[@dmacpher]

Such a bummer that a error correction filter with and edge case nailed them. Lots of amazing data and at least it’s a software fix!

[@sliceofbread2611]

Cliff case*

[@dmacpher]

🎢

[@thePronto]

Edge case? A crater on the moon? But it's not just a software fix is it? Or are we talking about a KSP do-over?

[@slcpunk2740]

Seems a pretty basic error, in what universe did they think they could figure the exact altitude without the radar? Even if it was broke too bad, damned if you do/don't.

[@dmacpher]

@@thePronto They moved their landing site to align with NASA South Pole targets super late in development (post validation). The threshold for culling/re-baselining seems to be the issue. The sudden change in relative altitude wasn’t expected from their simulations.

[@regolith1350]

Software may have been the proximate cause but you can argue the real problem was somewhere in the development and quality control procedures. How can you not re-run a full landing simulation after changing the landing location? It reminds me of Starliner's problems in 2019. The software glitch where the flight computer grabbed the wrong "time" was the proximate cause, but the real problem was Boeing never ran a full end-to-end launch simulation.

[@srinitaaigaura]

Actually these days so much of manufacturing and coding is outsourced that the management, hardware and software teams are no longer next to each other - quality control begins to suffer massively. The more people outsource stuff, the more the work gets into the hands of rookies paid on cheap wages, who then end up making rookie mistakes that then require even more time and energy to fix. Boeing turned from an engineering firm to a management firm and the rest is history - 787, 737 max, 777x, Starliner. And as more and more automation comes in there's less and less human intervention to take care of the times where the computers reach their limits.

[@すどにむ]

Feels like they might not have a great CI indeed, probably more like bunch of artifacts in git LFS type of management. But Starliner glitch might be slightly different topic IMO

[@BubblefishOfTrem]

I was also wondering how expensive such a simulation would be. If they aren't too expensive, I was wondering if you couldn't run landing simulations from randomized positions and flag anomalies from there. Not so much that you can just fling the lander at the moon arbitrarily, but more so you can find starting conditions which result in something weird. IDK, maybe we're getting into a space where "moon lander software testing" and later "asteroid lander software testing" might be a market, that would be amazing. With the costs of these missions, there might be some money on the line for a testing company - especially if they end up with a body of "known problematic situations" like the one from the video.

[@MrJdsenior]

How can you put a tank that has experienced both problems and damage in test into Apollo 13? Exactly like that, only different. Or get km and miles crossed up and smash a probe into Mars (IIRC), or ... ad infinitum. You can run all the simulations in the universe and still have problems, but not running ANY sims to cover a deviation in the program...yeah, that's just begging for it. I would think, in this day and age, that you could pretty much run that sim real time in parallel with the mission, for the problem they had there, knowing the path and surface profile, I'm guessing, and have it fire up a quick "do not ignore the damned properly functioning radar" command, or some such. It might even be good to have REAL TIME simulations running against the truth of the mission. Having done some aerospace hardware design, I'm guessing that there were schedulers and/or bean counters directly in the problematical loop. Or maybe idiotic MBA wielding managers that think they are engineers, or worse know BETTER than the engineers, because they know a few buzz words, and then maybe hold people's feet to the fire to get them to sign off on VERY cold Shuttle launches, or what have you. That's the sort of feedback you do NOT want in, say, a servo. :-/ Sometimes I look back and am glad I am retired, frankly. Some of it was fun, some of it SUCKED. Doc requirements come to mind as some of the latter. I had one junior documentation fiefdom wannabe tell me that the real output of a program was the documentation. When I finally quit laughing I told her that if she actually believed that she should go talk to some F16 pilot and ask them which they'd rather have with them on a mission, a working LANTIRN pod, or the documentation that describes it. She wasn't happy, because then a couple of people standing around laughed too. She wasn't a nice person (that's putting it mildly), or I wouldn't have said it that way. My bad, I guess.

[@i-love-space390]

Armchair quarterbacks are a dime a dozen. You can certainly crow if you ever land a vehicle on the moon or even achieve orbit. Perhaps we can talk about "how obvious" the solution was when we stop whining about how LONG it takes to build and fly a vehicle and how the contractors are "milking the American public" for so much money. I thank Providence every day for Kathy Lueders and NASA for riding herd on SpaceX to make the Dragon 2 safe. Everyone had lots of criticism for NASA for being conservative and "delaying" the first launch of the manned spacecraft. But all that effort kept the astronauts safe. (Also SpaceX had a real leg up on Boeing, because they had a working cargo spacecraft in Dragon 1 to build on. The last time Boeing designed a manned spacecraft was the 1970s and the Space Shuttle. All those engineers are long since retired.)

[@peterweston1356]

Makes the Apollo landings even more amazing. Considering the precision of sensors and computational resources, both to simulates and support landing.

[@perishmokrat8257]

Working as a Software Tester I often see the managers tend to take the risk to save some money vs malfunctioning SW especially when it has to deal with error handling.

[@Henglaar]

Which is a shame, really. The more expensive the project, the less management should feel like cutting corners on error handling and verification. Ah, well, what "should" happen in the real world doesn't agree closely with what actually happens in the real world.

[@IsMaski]

Unfortunate to see what led to the failure of this mission. But glad to see that they have found the issue. Really hoping they succeed on their next attempt. Thanks Scott for the comprehensive explanation on this!

[@MrPaxio]

they didnt find the issue, they made the issue

[@MonkeyJedi99]

Sounds like the software took the path of flat-Earth "science". What I see doesn't fit my preconceptions, ignore it!

[@togowack]

People need to wake up, controversy surrounded moon landings because there is stuff there. The issue / bug was in there on purpose. They will probably never let us see the real moon.

[@davidbeppler3032]

They did not find the issue. The issue was management. The software was fine. Software did not change the landing location, management did.

[@togowack]

@@davidbeppler3032 The whole things was planned it is every time with every country why do people not see this, every single machine that lands on the moon has issues - #1 because the surface is covered in glass domes and other hanging debris #2 to cover up such things from the public in a convincing way.

[@AllAmericanGuyExpert]

My Dad helped design the Apollo lunar landing software ... and curiously enough, it was never used due to a sensor overload ... the famous DSKY error 1202. When Neil Armstrong disabled my Dad's software for Apollo 11, that was the end of it. The LM landing program was always over-ridden by future LM pilots and the LM was landed manually. The fault was in a completely unrelated system ... I guess a lot of people wonder if it would have done its job. My dad says it was pretty robust and he never saw a simulation that it would have failed if given the chance to run to completion. It's a good thing Armstrong was a good pilot! My dad would go on to be famous for mockups, and then later, he worked on the avionics of the world's most capable fighter jet. He's getting old, but still with us. I wish he was more of a storyteller ... but the one he thought was the funniest (and most irrelevant) was meeting the president in the restroom at NASA ... as in, _um, nice day, isn't it Lyndon?_ as they conducted their business. I am guessing it was during LBJ's visit to Houston in 1968, the same time frame that my dad was working there.

[@AllAmericanGuyExpert]

You @PT-xi5rt didn't know that LBJ was president? Or that he used the bathroom like the rest of us?

[@dust1209]

This reminds me of an Alastair Reynolds novel where an automated system recorded the sudden vanishing of a planet but disregarded the data because the event was so far out of expected results that it assumed there was some kind of fault.

[@letsburn00]

It then accidentally creates a cult.

[@yogiwp_]

Which novel is this?

[@dust1209]

@@yogiwp_ Absolution Gap, it's the third book in the Revelation Space series which is kind of weird. If you're looking to check out the author, I'd recommend Pushing Ice!

[@ShoeTheGreyCat]

@@letsburn00 And also liquifying the poor guys wife stuck in the scrimshaw suit

[@letsburn00]

@@ShoeTheGreyCat I forgot about that bit. Given that series largely relates to characters that are functionally aging immortal, it's wild how easily they torture and kill each other.

[@henrymalone422]

Been watching you since 2015! You have helped keep me interested in space flight! Thank you for doing what you do Mr.Manley.

[@firefly4f4]

By, "unbelievable", I'm pretty sure you meant, "Completely realistic, very common scenario when the software is put in an untested environment." Note that I am saying this as a software developer myself. I actually just identified a scenario where our existing tests were thought to be sufficient, but then some surrounding parameters changed and a bug was found.

[@jarisundell8859]

As a software developer myself, I'm actually asking myself why those simulations were not set up to run like a CI.

[@firefly4f4]

​@@jarisundell8859 Good question. Seems like actually running the sim again once the final site was chosen should have caught this, maybe allowing them to upload the fix. For the record, CI is how the one I looked at was caught... prior to release 👍

[@danstenger1]

Scott is also a dev by trade, too, lol, he works at Apple.

[@cinquine1]

@@firefly4f4 I think it's a joke, since the bug happened because the computer didn't "believe" the radar

[@scottmanley]

By unbelievable, I mean the software stopped believing the radar

[@i_Kruti]

7:50 Yeah , the VIKRAM lander from CHANDRAYAAN-2 had lost communication and went out of control , but with improvements in software, damper etc , we are again ready for CHANDRAYAAN-3 to it in July according to official message......

[@glennpearson9348]

Excellent explanation, Scott. Thanks for putting it all together for us to easily digest. Nice Kerbal recreation, too!

[@thePronto]

A lunar lander encountered a crater and got confused. Total freak accident: one in a million. I can totally relate: today, I encountered a Starbucks in a strip mall.

[@AMeierhoefer]

Scott, I am surprised that you did not touch on redundancy. I was a fighter jet aviator and one of the things we always did was use multiple sensors to allow the software to compare and then estimate probability. If they has three Radar altimeters they could see the rate of change of the surface as the spacecraft travels. Even if each would have shown the cliff, probability calc would have told it that its is virtually impossible that all three are suddenly all bad. Redundancy would be one answer in my book.

[@thierrybriand2413]

Agree and also on my part, I always thought that radar altimeters were used « closer » to the surface.

[@drill_fiend1097]

Probably budget constrained.

[@AMeierhoefer]

@@drill_fiend1097 This is a commercial effort so they could have just gotten one normally used in aircraft. It's not NASA where they cost $750K each just because...

[@ns219000]

Japan, sorry for your loss, but thanks for the software design lesson. Rockets are hard and this is how we learn. Thanks for sharing this one, Scott!

[@Hagop64]

If it stopped to a speed of 0, then fell to a speed of 500 km/h then it would have had to fallen for ~86 seconds. Moon gravity acceleration = 1.62 m/s^2. That means it was in free fall for a distance of about 6.0 km. That's all based off of the "500 km/h" crash speed given.

[@scottmanley]

Actually, I figured out 500km/h based upon the amateur radar measurments of 88seconds of freefall.

[@Hagop64]

@@scottmanley Love how reliable basic physics equations are! With either bits of data it still comes up with the same results! If only the rest of landing on the moon were that simple.

[@travelbugse2829]

What I want to know is how that equates to a violent impact on earth. Do I divide by six, which comes to 83.3km/h or just under 52mph? That's bad enough for it to need airbags...

[@highdefinist9697]

@@travelbugse2829 You multiply by the square root of six - assuming there is no air resistance, so with air resistance you might end up with something not too different from 500 km/h for this type of vehicle.

[@Kromaatikse]

@@travelbugse2829 When it comes to the moment of impact, 500kph is 500kph. It's about Mach 0.5. You know those old war movies where they show fighters shot down and augering in? *That.*

[@chouseification]

hey Scott - thanks for the analysis. I remember this one (as well as the Israeli and Indian ones) and seeing the disbelief in the control room was sad. It is easy to tell who has a clue and who is a bureaucrat by their expressions, etc. :P

[@adarsh4764]

Hope there's no software issue when Nasa lands back on the moon!😂

[@chouseification]

@@adarsh4764 agreed - one would have thought that even a small lander would have a pretty robust navigation system these days, but obviously they met an edge condition they hadn't properly tested for... and a sad oversight too as nearly all landing trajectories will have the radar return affected by craters you're passing over. There are many of them after all, and although most are small, many are large/deep and you need to keep their profile in mind as you use the radar/laser/etc surface measurement. The state vector routine needs a sanity check to make sure the drift never disagrees from projected too much without it doing some form of reliable recheck.

[@joelcorley3478]

But what if the radar altimeter actually did fail around the time it passed over that crater? It sounds like it would have produced the same result. I think the only way to deal with this in the design is to have at least one redundant sensor for something this mission critical. Of course the problem with just one sensor is that you need to try figure out which one is actually the broken sensor. That's why there is often 3 sensors or 3 computer systems that are used in this kind of redundancy...

[@sonaxaton]

Sounds like a redundant sensor wouldn't have helped this particular issue though, because it would have just gotten the same confusing measurements of the cliff wall. I think they just need to thoroughly run simulations of the actual mission to catch edge cases like this early.

[@a4d9]

On a vehicle like this, without humans onboard, the space and weight requirements might be too costly compared to the risk of a failed sensor.

[@SashaNaronin]

@@sonaxaton exactly. Proper simulation campaign would've catched that.

[@Damien.D]

@@sonaxaton 3 redundant sensor and a voting system is the way to go. Worked flawlessly in many aeronautical things, from Concorde autopilot to missile guidance system.

[@travcollier]

The dead reckoning system combined with prior knowledge (a map of roughly what is expected) should have been enough of a redundant system. Seems like they should have included a reassessment/recovery routine to check if that apparent altimeter glitch (which wasn't a glitch of course) cleared and the instrument was giving reasonable data. This stuff is really tricky without a human in the loop.

[@Anacronian]

It's crazy to me that they didn't redo the simulations when a new landing site was chosen.

[@bobbun9630]

From this description it sounds like the software worked as intended based on the circumstances. It sounds more like they need to rethink the system level design to have more inputs that can be used to sanity check one another, and perhaps have a means for a one-time instrument glitch (at least in the design interpretation) to be "forgiven" if later sanity checks pass.

[@u1zha]

Yes, that makes sense, and the "forgiving" part is commonly solved by Kalman filtering, which Scott also mentioned. Here it sounds like Ispace overengineered a little bit, overeagerly dropping sensor data on the floor before giving the filter a chance.

[@Anvilshock]

Hardly a "bug" when it worked correctly for the data input it was programmed to handle. At best, it encountered data it _wasn't_ programmed to handle, which makes this more a missing feature.

[@mikehartsough489]

I was thinking same thing. Sounds like the software did exactly what it was supposed to do.

[@1224chrisng]

well, a bug is just unintended behaviour. The computer did exactly what you told it to do, just not what you wanted it to do

[@ddnguyen278]

Can't imagine why they didnt run simulations of this. It's not like the moons topology isn't known down to the meter. Stick it in Kerbal and run simulations.

[@RemyPorter]

@@ddnguyen278 Uh, the moon's topography *isn't* known down to the meter. Some areas of the moon are, but generating meaningful maps of the moon is actually quite hard and time consuming. There are folks whose entire job is to take lower res digital elevation maps and apply reasonable interpolations to generate higher fidelity maps than we actually have. Not saying they shouldn't have done more sims, but it's harder than it sounds.

[@davidwright7193]

Repeat after me “That’s not a bug it’s a feature”

[@wChris_]

Its amazing how Apollo didnt have such bugs, despite it being written in pure Assembly!

[@PMA65537]

They chose tamer landing sites.

[@phloxie]

@@PMA65537 apollo 15 likes to have word wth you

[@castafioreomg]

Apollo missions had some issues but they handled then well..The engineers couldn't even visit their families becoz of the work pressure

[@lyoha5028]

I wonder what all these people in mission control were doing during the landing. Were they analyzing the telemetry in real-time? I assume they were supposed to notice that the radar altimeter was considered faulty and disabled. If so, perhaps they could have reviewed its readings and realized that after passing the edge of the crater, the readings returned back to normal. In that case, they could have just manually reenabled the radar altimeter. Since it is not Mars, the signal delay is small enough to allow for manual corrections during the landing.

[@katho8472]

Word!

[@ooooneeee]

They lost telemetry. If they had a connection they could saved it.

[@pavanshetty9806]

There might also be delay in communication.

[@thetooginator153]

It would be interesting to try an optical parallax system to verify the radar readings. If both systems agree, then the data is correct. Cameras could be a few meters apart, so, the parallax would be measurable from pretty far away.

[@EnricoGolfettoMasella]

That’s a very creative solution! ✌🏼✌🏼Pretty sure would work!

[@xonx209]

If they don't agree, then what do you do?

[@4k8t]

@@xonx209 In sci-fi usually it would be three independent systems with two having to agree as to what they were seeing. A two system setup would require that both system have to agree and if one system cut out a sensor as malfunctioning and the other didn't, something would have to be present to break the disagreement deadlock.

[@riccardob9026]

To be honest (and a bit philosophical), I would not call this a "bug," in the sense that sometimes with bug you mean an error in the software that makes it behave differently from the behavior specified at design phase. In this case the software had to face a situation that was not expected, that is, a suddenly increase of altitude due to a deep crater. It was not an error introduced at implementation time (that is, when they wrote the software), but at design time. Like a bridge that breaks down, not because some error during the building, but because of a strong wind that was not considered at design time.

[@DrDeuteron]

I agree. This was planning error, or a failure to test error, or changing the landing into a regime that had not been tested, or all of the above. It's been know for a long time that radar altimeters can be spoofed by terrain, it is nothing new.

[@serronserron1320]

An engineering oversight

[@aspuzling]

As a software engineer I agree lol but that's not to say it is not also partly the responsibility of software engineers to raise potential bugs in the design.

[@chaz720]

Agreed, and came to write this. As a space systems engineer, this was a systems engineering failure, not a software bug.

[@bbgun061]

They should have tested their software with real data.

[@ytashu33]

Love this! Thanks you for reminding me of Kalman Filters, i studied those in my M. Tech., loved them but never thought i would ever hear of them again. I still remember how the "location estimation" part, based on current velocity and direction integrated over time (aka: dead reckoning) can provide smooth and accurate predictions over short durations, but errors tend to accumulate in a physics based predictor like this and needs to be augmented with an independent measurement (ie: the radar), even if the radar data is not accurate. Amazing to see how stuff like that led to this outcome. It is a tough one though... I wish you had shared your thoughts on how should a "faulty sensor" be detected then? I mean, you could say that a 3 Km sudden jump in the sensor output means the the sensor is probably broken, right? If not, how else would you do that and handle the case when the sensor actually is broken?

[@Beregorn88]

Redundant systems and majority check: if all three of your radar sensor reports a sudden altitude change, than that's what actually happened. What surprise me is that the sudden altitude change eventuality is never accounted for...

[@tertiaryobjective]

Like when you're walking down the stairs and miss that last step.

[@RogHawk]

Thank you, Scott! You answered questions I've had for the last few years about the landers crashing on the moon.

[@noahserio4182]

I’m surprised they didn’t have a redundant altimeter to verify the suspect altimeter reading against.

[@snwendland]

I seem to recall the University of Wyoming having a "Missile Guidance for Dummies" audio description of a guidance system for knowing where the missle is by knowing where it isn't - it seemed pretty rock solid. I have to wonder why this method hasn't been adapted for spacecraft yet.

[@frodo9649]

It substracts where it should be, from where it wasn't.

[@H-S.]

Exactly. It would be especially helpful in this case; if the lander knew where it isn't, it would not waste fuel by trying to land as if it was just above the surface. :)

[@u1zha]

I believe that's just a sentence for lulz, engineers expressing themselves purposefully obtuse. Kalman filters are exactly the "knowing" part, and a closed loop control system is exactly the "subtracting" part.

[@frodo9649]

@@u1zha kzbin.info/www/bejne/mIvIZn1uiLt2j7M This video is full of these sentences, that are close to how control loops work, but not quite, which I find quite funny, especially if you know how it works

[@simongeard4824]

@@u1zha Unfortunately, it also inspires a lot of morons to quote that line constantly on KZbin, perhaps under the mistaken impression that it makes them look smart.

[@kaineis]

I love the ksp2 animations you added. That was really nice to watch.

[@subhakantagmail]

Finally the software bug is fixed and the Vikram lander from Chandrayaan-3 landed safely on lunar surface by ISRO. Hope most of the space agencies share data among themselves so that space progress is accelerated faster, instead of each one reinventing the wheel. Knowledge for Humanity...👍

[@henrikibjensen3869]

Sorry, Humanity doesnt land on the Moon, nations do - or dont.

[@antonioloma2327]

If they tested by simulating landing on other spots but not on the selected one, then they didn't tested! This isn't a software bug but a project mangement issue (specifically testing). It's like if you "test" your computer program on your desktop but then deploy it in a server and the faster hardware makes apparent a race condition that borks the system. Testing is expensive, testing is hard, but not testing the actual flight plan is dumb.

[@RobertBlair]

Software engineering does not start at the keyboard, and end when it gets sent to a testing team that is somehow not software engineering. Engineers have a responsibility to work with testing crew, to validate the test scenarios. The teams failed to run enough variations of realistic input, so inputs outside the limited sets caused a fault. Specifically several bugs in the system as a whole 1 Spacecraft is unable to land without altimeter inputs. Relying on only inertial guidance cannot be accurate enough to land, due to inherent input noise. If the altimeter signal is discarded more than X seconds before touchdown, error margins cause failure rates approaching 100% 2. Guidance system (apparently) had no way to recover confidence in sensors 3. Guidance system would erroneously flag valid inputs altimeter as a broken sensor. 4. Testing was not done to cover new landing site (and yes, a senior engineer should have balked at the change)

[@AleXsSpaceXTalks]

Very good explanation and top video! I guess also the loss of the Mars Polar Lander was caused by a software issue, telling the landing thrusters to ignite too early, causing the probe to run out of fuel...

[@bertram-raven]

I would add optical recognition and stored high resolution images to the package. These would optically compare the expected position and orientation to the visible information and so call out anomalies. This entire apparatus would be as small as a Raspberry Pi, using off the shelf components. DJI drones use something similar in their RTB software. When the drone sets off, it takes photographs of its starting location and compares them to the downward facing camera live images when returning to land.

[@Topcoatdetail]

One of the reasons Chandrayan-2 failed because of the mapping. When the lander moved away from the photographed landing site it tried to over correct and failed.

[@mrpocock]

FYI if the landing location and approach is part of the software spec then a change to the landing site and approach is a change to the software spec and requires a full end-to-end revalidation of the software.

[@brentboswell1294]

Didn't Neal Armstrong have to do some on the fly recoding to overcome the 1202 error when the Eagle lunar module was getting overwhelmed with input? (Which was fixed on later Apollo missions through code fixes and turning off an un-needed radar as part of the checklist?). Seems like they could have used an altimeter, but on the moon the altimeter setting is always "00.00" 😅

[@Bill_Woo]

A moment of silence for the Mars programmer who couldn't handle math, metric and standard. Although, having some direct experience with atrocious egomaniacal team leaders and managers, including one in particular at JPL, I would suspect that the programmers and reviewers did exactly what they were told, while the team leader should get a "black mark in his permanent record" for being the one who was the true culprit.

[@robertst-laurent6452]

Mr. Manley, for the whole planet you are our 21st century Eugene Kranz. At 01:13 your video proves that we now have available: ‘A da Vinci World of Creativity at Home’ The video shows that they used a $170 Airspy R2 receiver (with a $620 LNC + antenna) with the mind blowing power of the software available for the Airspy, so for less than $900 USD you can have the same setup at home ! Your use of the Kerbal simulator, to help us better understand the sequence of events, is of jaw dropping beauty.

[@caturlifelive]

Thats why i love Scott Manley video, so detail

[@brianjrichman]

When flying on instruments, pilots are trained to trust what the instruments are showing them, not what the software in their heads is telling them "I can't see the horizon, so I think I'm upside down..." No... You are the right way up etc.

[@thePronto]

We too low...

[@brianjrichman]

@@thePronto We too high when we ran out of fuel... OOPS. Instruments didn't lie then?

[@thePronto]

@@brianjrichman bang ding ow.

[@oohhboy-funhouse]

Been through the training, you absolutely cannot trust your feelings. It varies for each person, for me, I felt I was leaning one left and right. I had to actively fight to focus on the instruments, draining your energy very quickly. Even if you are completely focused on the instruments, your arms will try to 'Level' the plane. You acclimate with each subsequent flight easier, eventually it's nothing. Instrument flying is still draining as you have to fuse all these sensors. In comparison, looking outside is about as stressful as driving.

[@brianjrichman]

@@thePronto

[@averystablegenius]

Yet another example for building a constellation of Lunar Position Satellites prior to attempting to land on the surface. Like GPS, but LPS for the Moon. As always, failure to invest in infrastructure always costs in the long run. Anyone wanting to create a commercial venture to put four satellites in Lunar orbit for licensing LPS services to other companies and governments, please reach out to me.

[@7tonsofsalt865]

sadly lunar gravity is very lumpy that means no stable orbits and iirc the moon doesn't have a big enough sphere of influence for stationary orbit

[@thePronto]

If we can't land on the moon 50+ years after it first happened we shouldn't be wasting money on tech that could be just as erroneous.

[@averystablegenius]

@@7tonsofsalt865 Good input, 7, but I wonder if the lumps can be compensated for, even dynamically through INS, in software... LRO has been orbiting productively for, what, 16 years?

[@averystablegenius]

@@thePronto What tech did you have in mind?

[@7tonsofsalt865]

@@averystablegenius the problem with the lumps is that they make spacecraft require a lot higher fuel usage than say earth orbit. even the lunar reconnaissance orbiter which is on one of the few "stable" orbits will eventually run out of fuel and with the cost of lunar injection flights i just cant see that being profitable maybe one we can launch and refuel them from the moon but well see

[@yahoolane]

My first feeling was that one of the issues is that they all seem to be young people, I know they are very very smart young people. But I don't see the 'old guy' in any of the pictures. An old guy who has experience with developers and engineers.

[@younameme2]

Perhaps a stupid question but why didn't it have redundancy on sensors?

[@linecraftman3907]

Probably weight limit

[@samuraidriver4x4]

Weight would be the first thing that comes to mind. More mass = more capable spacecraft needed = more costs.

[@abdullahkhalil9284]

It has. all those "radio Sensors" (I think it had 3, for collecting data from different directions) that were measuring the height were identified as "faulty" by the bug and It turned off the input from those sensors. Marking the sensor as faulty based on the output of the sensor and turning off all those sensors was the bug

[@oohhboy-funhouse]

$$$ and weight.

[@olasek7972]

find me a spacecraft that has/had redundant radar altimeter, even Apollo didn't have one

[@gefrix]

It looks like the software did exactly what it was configured to do, it was just improperly configured for the new landing route. Unless there is more information to this, that's not a bug, that was a misunderstood feature. Someone requested it, then forgot it exists.

[@jeffstaples347]

Dunno if you'll see this, but I'd be suuuuper interested to hear your thoughts on whether software engineers should have the same certification processes that physical engineers have.

[@Esablaka]

Fyi: they do have in some countries. One isn't allowed to develop software for medical gear for example here in Germany without certain qualifications. There are also legal requirements and standards that the software development teams working on critical or potentially dangerous software have to adhere to here both in regards to coding and testing, but also in regards to their overall software design, risk analysis and much much more.

[@jhonbus]

The kind of software you're working on makes an enormous difference to the consequences of getting it wrong, and I don't think it necessarily makes sense to try and develop a certification scheme that's simultaneously rigorous enough to assess people working on nuclear reactor control software while not failing 90% of candidates going into a career in casual game development...

[@torinor6703]

They do. Software engineer, computer engineer, computer scientist, etc they are all degrees that you can get studying in college. The issue is that you have all the 3 month long React crash course Jimmys that call themselves "software engineers" when they are software developers instead. So yeah, there are many different certificates for software engineers, its just that the name is often not respected. Btw Im not trying to shit on self taught or non engineer software devs. The same way people like Michael Faraday did amazing things in the name of scientific research without a degree, some software developers are incredible at what they do. I dont want to sound like Im implicitly blaming the lack of certificate enforcement as the reason why stuff like these software bugs exist. This could have very well been coded by a software engineer, who would be also not to blame because stuff like this has to be peer reviewed and tested by many people and on many levels. It was simply and incredibly unfortunate event

[@jeffstaples347]

A degree is not certification. And for physical engineers that assume accountability for the designs of their companies, Principal Engineers, their job titles do not come easily. What would be interesting to see is if there is any push to require engineer ACCOUNTABILITY, not just responsibility, as things are currently in the physical space.

[@CarFreeSegnitz]

This brings me back to my software engineering course. There are supposed to be several stages like requirements, specifications, design, implementation, testing, maintenance. And each stage is supposed to be evaluated and fed back to improve processes. The instructor stated that there had only ever been ONE software project to have ever followed the theory: Space Shuttle avionics.

[@jeffzaun1841]

This brings to my mind Neil Armstrong calling audibles seconds from Apollo 11s touch down, keeping his head about him as the computer re-booted then flying his contraption past a rough spot. Go humans

[@MattMcIrvin]

Speaking as a veteran in the software industry, no bug, no matter how absurd, is unbelievable to me.

[@Jeffrey_Friedl]

"Dead Reckoning" is the phrase you were looking for to describe how it tried to calculate its height after the radar was cut off.

[@Jonathan_Doe_]

Really makes you appreciate Margaret Hamilton’s achievements given the computing power of the time. Yes the Apollo landings were manned missions, but they were still very dependent on that code functioning correctly.

[@rayoflight62]

Thank you for all the detailed explanation! Greetings, Anthony

[@junaid-vc3js]

I was hoping to learn that a line of assembly code or c code caused the error, but blaming it on testing verification and validation makes the software coders think phew ‘ it wasn’t us gov ‘ - to be honest I just feel for the engineers involved

[@wolf7115]

Damn. As a software developer myself, this is totally something that I feel like I could have happen to me as well. I feel for the ispace team, and know they'll get it next time!

[@Henglaar]

Agreed. You would have had no way of knowing, or testing for, a way that valid real world data could trigger the failure detection routine. I'm thinking that they need to do a hardware design change: add a second sensor, or better, a second TYPE of sensor and cross check the two. If the data only goes wonky on one of the sensors, declare a failure. And radio home that the mission is probably fatally compromised, given the nature of the failure.

[@viCoN24]

Guess what? This is how it works. The problem here is the definition of what should be interpreted as error in the input. If you have incorrect assumption, another array of sensors will reach the same consensus about being out of order based on unexpected range of values read. Initial landing site was a plain so you wouldn't expect drastic changes in altitude. Compare that to the new landing site near a rim of a crater where rim was high enough to fit the definition of unexpected error for the assumed scenario of landing on a plain. It seems that everything worked correctly for the purpose it was designed and implemented for but someone decided to change critical part of assumptions and cut corners based on gut feeling instead of thorough simulations. I wonder if Japanese culture worked against this project where rising this issue was frowned upon by the higher-ups so nobody raised any concerns.

[@virt1one]

I don't normally have to deal with "what do I do if this important sensor fails?" scenarios. (in most of my things, all the data I have can be trusted 100%) So here I would envision "what do we do if the altimeter returns implausible data?" "Rely on other systems" seems like a good answer, but DEMANDS you also ask "how long can we rely on them?" The answer here is "not very long". Which brings up an important question "what do we do if we've been relying on this other system with decaying accuracy?" If THAT question had been asked, then they would have realized they needed to reconsider using the data from the altimeter. You solve this by assigning a "quality" to your sensors, and you go with the highest quality. If you get a wonky reading, you do something like drop the quality to 80% or 60% or 20% or whatever you feel is appropriate. (ONE weird reading might drop it from 95% to 70%, and successive apparent glitches might continue to subtract 5% each?) And when you're using inertial guidance, which is normally "recalibrated", that should have its Quality decay over time, AND reset to the quality of the recalibration source every time it's recalibrated. That's just how that's done. In this scenario, say the quality of the altimeter was changed from 95% to 60% when it glitched. OK. The inertial guidance would start ticking down from 95%. Hopefully it will cross below 60% soon, at which point you revert to using the altimeter because its quality is higher. This approach might have "re-enabled" the alitmeter and saved the lander.

[@abrahamd2k]

I wouldn't call it a software bug. The software performed as instructed. It was the changes in the planned landing sight at the last minute without any testing afterward. Someone f____d up.

[@Twenty-Seven]

I remember one of my computational methods professors said: "There are no such thing as software bugs, only human error" in one of our first lectures so that we would code more carefully and make sure everything has correct syntax before moving on to a new line. It'll save you a lot of time rather than overconfidently writing 100 more lines of code and then having to scroll through it all only to find that you coded "fr" instead of "for."

[@effedrien]

Most bugs are not caused by typos but by overlooking consequences somewhere else in the code, like creating a potential timing/synchronization issue. Or by wrongly interpreting the functional requirements, or by misreading the documentation from an external library and things like that. Typo bugs are rare, except in the ui but that is because certain programmers cannot write decent English sentences 😅

[@aakksshhaayy]

@@effedrien I think he was just giving a basic example.

[@stuartgray5877]

The Mars 98 lander failure was found to be due to a software bug AND a hardware "idiosyncrasy". First there was a Software bug that allowed the Flight Software to begin looking for "touchdown" from the landing leg sensors way too early in the descent. Next there was a hardware issue where the hall effect sensors on the landing legs would detect the magnet passing under the sensor during DEPLOYMENT of the legs NOT just during landing as they are meant to do. They would false trigger with about a 30% chance of sensing the deployment. The lander had three legs so it was almost a guarantee that at least one of the sensors would detect "touchdown" *during deployment* . But the FSW was not supposed to be watching for "touchdown" during deployment. Finally, a wiring defect and a failure to repeat tests was what caused the demise of the lander. When they did the "Pyro Shock" test of deploying the legs the very first time they were running the descent simulation. When they went to simulate "touchdown" the technicians lifted a leg and nothing happened. Tried it on all three legs, no detection of "touchdown". It turns out it was a wiring mistake on the hall effect sensors that was not caught yet. They rewired them, and it fixed the problem. The legs now could detect touchdown as expect. However, they chose to NOT RERUN the entire pryo shock test again because it was a test that is hard on the hardware and very expensive. So they launched it not knowing that it would detect "touchdown" way too early in the descent sequence and thought it was on the ground so NEVER FIRED ENGINES to complete the landing. This was discovered on the second lander during the pyro shock test because they had the sensors wired correctly that time.

[@kukuc96]

Hi Scott. Speaking of interpreting sensors: Do you think you could do a video about the topic of what sensors spacecraft are using for navigation? Maybe as "Things Kerbal Space Program doesn't teach". Because in ksp we magically know everything precisely, from orientation to position and speed. But how you measure that IRL could be an interesting topic.

[@Henglaar]

I like that idea too. My information on such things is decades out of date. I know they use star tracker cameras to take fixes on specific stars, at least in some cases. And I seem to recall the Apollo astronauts actually used a sextant to do the same thing manually and fed the data into the computer. In other cases, like satellites, they may not bother with navigation fixes at all, as they're monitored from the ground anyway.

[@davidboyle1902]

Sounds a tad like what would have happened to Apollo 11 had Neil not been ‘the computer’. Their planned landing sight was a disaster waiting to happen and was why Neil had to take control of Eagle and fly it clear of a boulder field. Nice presentation and a wake-up call to the folks who think computers are infallible.

[@BoroBootBoy]

Not really a software error. More like a programming error.

[@spacenomad5484]

Most likely design/specification error

[@han5vk]

Programming = creating software. What the fuck is the distinction in your mind?

[@eddievhfan1984]

Would there have been enough memory/CPU time in the lander to include a semi-coarse terrain map to compare against the RADAR altimeter's results? For Apollo 15-17, they modified the LM guidance software to include a 1D polynomial function map of the terrain under the LM on its planned approach path, since they were headed into areas that were less flat than previous missions, and didn't want the landing RADAR generating mistaken position info.

[@juliusfucik4011]

I think that would have been so easy to include that they probably did include it. I think it is a lack of testing of the software. You could easily simulate a million landings and find these bugs.

[@chrisstone2742]

You think there would be considering the advancements we have made in modern electronics. Either this is a case of newer isn't better or the software engineers really didn't think things through since no human lives were at stake.

[@trs4u]

I was wondering a few days ago why this kind of instrument hasn't appeared on expensive munitions fired in the direction of people who understand how satellites work. A few real-world things seem to be showing signs of having gone too long since they last faced really hard tests. Hopefully next time we walk on the moon we won't stop doing it.

[@chrisstone2742]

@@trs4u In some ways technology have made society has a whole less intelligent IMHO. Why send someone to school to become a master welder when you can just have a computer and robotic arm do the same job.

[@trs4u]

@@chrisstone2742 That might be stretching the topic a bit, even if I would instantly agree with you! "If I ruled the world" I'd try to address that. I've grown accustomed to my global dictatorship being quite improbable.

[@synergy021]

Is there a requirement that all titles must be clickbait and include one of these words: Unbelievable, Shocking, Terrifying? No the reason wasn't unbelievable, it's actually quite believable and simply just an oversight.

[@scottmanley]

It’s unbelievable because the navigation software stopped believing the altimeter.

[@synergy021]

@@scottmanley Lol, good save. Wasn't really directed at your video per say, just that's the KZbin titling by youtubers trend these days. Although yours is actually technically accurate hah 🙂

[@kiereluurs1243]

What was the 'REAL TRUTH?!!'

[@acasualviewer5861]

As a software developer I know there's no such thing as an unbelievable software bug. Whatever can happen, will happen.. you have to be incredibly diligent to remove as many bugs as possible, and still there will be bugs.

[@teslatrooper]

okay but what if we just don't write the bugs in the first place, ever thought about that? j/k

[@acasualviewer5861]

@@teslatrooper haha.. tell me you don't know anything about software development without telling you don't know anything about software development.

[@Navak_]

As a software engineer it is so incredibly frustrating how edge cases are not taken seriously. Unlikely scenarios are often treated as if they do not exist. In fact the term "edge case" is used to dismiss anything the team doesn't want to work on--if you say "well this is an edge case" it's synonymous with saying "we won't fix this." It's truly shocking to me. My company's software is used by millions of people per day; a bug with a 0.3% occurrence is going to affect more than ten thousand people per day. And it's just treated as if it were nothing. Yeah we aren't landing on the Moon, sure this isn't Boeing's MCAS software which has people's very lives in its hands, but still. We're still a fucking company trying to make money, aren't we? You think having a really stupid and frustrating bug affecting ten thousand users per day isn't going to affect our product's perceived quality and our company's reputation? Like what if we ran a TV ad and we misspelled words in it... you'd want to fix that, right? Even though it's "just a small issue?" And that is less disruptive, I'd argue, than a crash, or a freeze, or a hang, or wiping the user's input, or navigating to the wrong screen, or having buttons that don't respond, etc. Especially in a new feature. We rush out these new features, unpolished, because we don't want to "waste time" on "edge cases," and what does it teach our users? It teaches them to be suspicious if not resentful of new features we release because they are likely to be buggy and have a bad user experience. I am convinced the only reason no one cares is not because of cost but because software is complicated and annoying to think about and people would rather just release the product and pretend there's no issue rather than sit and think about this odd bug that only happens in some very particular case.

[@TheTennesseeTornado]

I always have to laugh out loud when I hear that someone programmed a "Just go ahead and ignore that other thing we programmed, even though you were programmed to pay attention to the thing we are telling you to ignore" code. It always works out really well.

[@JohnSmith-fz1ih]

I don't see the problem. Hardware failure happens... so writing code to detect when such a failure happens and falling back to another method is the right thing to do. This wasn't a problem with the concept of checking sensors and ignoring them if they are faulty... this was a problem with a minor bug in the implementation that wasn't caught due to lack of testing.

[@tbengineering7066]

@@JohnSmith-fz1ih I have to agree with John on this. Sensors do fail and they do so all the time, especially in extreme environments. How you account for such a situation is a constant debate between engineers. Do you fall back on another sensor? Do you assume it is faulty? Do you fail current state? Do you fail to a different state? Do you run calculations and try to verify if the sensor is correct or not? The answer will vary depending on circumstances, situations, and underlying memory storage. I was asked this question yesterday by a coworker as to what state should we fail our pressure control valve if the sensor fails. We discussed and came to the conclusion that it depends. We could fail open to make sure everyone is safe even if the outcome isn't desirable, or we could fail to current state and rely upon another sensor and another valve up stream to properly determine the pressure. Both situations have their drawbacks and benefits. All in all this isn't a dumb logic situation, but a simple oversight situation that failed to account for a possible condition the lander could have been in.

[@richardmattocks]

I’m visualising a cartoon-style fall. The sort of fall you see when a character runs off a cliff and is fine until they realise… it “lands” (and hovers) but then the jets stop and “yikes” as it plummets.

[@kendokaaa]

Interestingly, this is the kind of problem that could also occur if you make a kOS (or kRPC) landing program in KSP if you use instruments and not the game's data

[@paulphillips9548]

Hello Scott Love your posts and enthusiasm for all things space! In the last several attempts to get small landers to the moon, there is one common thread...no real time marker to tell the lander where it is above the surface. They have used simulations to create software that uses extrapolation of location from multiple inputs to guide them to the surface. The unfortunate reality is that all this software has to have limits that have over-max cut-outs on them. When these limits were passed during the landing sequence, it has caused loss of "Ground Truth" if you will. A tried and true method that is used by bush pilots for determining exactly where the ground level is when landing in flat light conditions on snow, where contours are impossible to see, is to toss a contrasting coloured wooden cross with 1 meter long arms out of the plane onto the snow to give them a reference point of known size to let them figure where the ground is. My thought is to have the lunar lander equipped with several small optically reflective markers that are equipped with radio beacons that are ejected laterally from the lander at an arbitrary height above the surface. These markers would then free-fall to the surface and create physical markers of the location of the lunar surface. The markers would have to be light enough and robust enough to be dropped from say, 5 kilometers above the lunar surface and survive the impact acceleration. Given the rocket powered ground penetrators with telemetry suites in them that I have seen developed, that really is a no-brainer. These markers would provide both visual and radio references that the lander could rely on, in order to make a perfect landing using both optical triangulation and doppler radio feedback. Your thoughts?

[@oo0OAO0oo]

That's really interesting, but not really a software bug. It worked like it should. But the software engineers need to write additional code for those cases, or even upgrade the hardware to prevent something like this.

[@Observer_Effect]

"Unanticipated" is a better word than unbeLIEvable. It happened, it's beLIEvable, and now we know it's a variable that should have been seen.

[@scottmanley]

It's unbelievable because the software stopped believing the landing radar.

[@dipakyadav3355]

Your wish fulfilled Scot .India is on the 🌙🌚

[@HaloPiter]

My experience in KSP tells me that best way to tell how fast you're gonna meet the surface is watching your shadow. I wonder if that idea will be used some day in real life

[@revwarnut]

The problem with that is that your shadow may be miles away from your current position. And of course it would only work if landing where the surface is getting some sunlight.

[@8minecrafter8]

A Kalman filter-based estimator forms predictions of the sensor measurements given the current state and differences those against the actual measurements to form "innovations," which is where a large discrepancy might result in the measurements being rejected so as not to corrupt the recursive state estimate. I wonder how the predicted altitude measurements were being generated- from a terrain map/database or just some basic constant altitude model whose process noise wasn't large enough to see the massive altitude discontinuity due to passing over a cliff edge as plausible?

[@ShirishJadav162]

Accelerometers are hard to work with to integrate and figure out distance travelled. As accelerometer have noise which increases the error in estimation of distance and this leads to wrong distance estimation.. that's why you need a real distance measuring device like lasers or something

[@piotrd.4850]

Software is different in several key aspects: first, it is tremendously easy to add complexity. Weights nothing, is might be actually cheaper / quicker and is not patently obvious as excessive components on board or too much material on chassis. Also, no implementation is going to fix bad design and vice versa.

[@deang5622]

Kalman filtering isn't sensor fusion. It's an algorithm to reduce the noise on the signal from a single sensor.

[@scottmanley]

It can do that, and more

[@SIBUK]

IMO it was an utterly stupid decision to program the software to assume that the altimeter was faulty and to start ignoring it from that point onwards. How could you ever possibly land without knowing your altitude? Can you imagine landing a plane with no eyes and no idea what your altitude was? It would be impossible. The chance that the altimeter was working correctly is a million times greater than the chance of landing safely without knowing your altitude.

[@ekt3233]

Honestly it probably shouldn't require a simulation run and software update to adjust to a new landing site. The problem was they built a system with a single point of failure. If the software already accounts for the failure of the radar altimeter they should have also planned ahead and built in some laser altimeters that sample multiple surface points looking away from the craft's bottom center. At the very least they could have had a backup radar altimeter to crosscheck readings, or have been sending back live data to the command center and let a human verify the readings and manually accept the "false" readings.

[@KirstyTube]

As a QA test engineer i appreciate this video approximately 99% Makes you wonder why it didn't just keep dropping at maximum safe landing velocity. Maybe until the landing legs impacted or something. Im sure lessons will be learned, more QA implemented etc

[@joshmurphy2766]

This is why standards such as DO-178, DO-254, and ARP-4754 (and associated specifications) exist for aircraft. These are very intense standards for development, but even if your company was not required to certify with an aviation authority, following something like these in the development of a spacecraft would reduce the possibility of errors such as these. It's very possible there are similar standards that were followed, but sometimes I wonder if these errors occur because the critical error was not discovered due to budget/time constraints sacrificing a critical portion of the development lifecycle.

[@WoefulMinion]

This happens a lot in other areas of science and technology, too. People over-estimate how much they can know about extremely complex systems and make assumptions without realizing they are doing that. It's easy to forget that software depends on fully understanding problems and anticipating any issues that may arise. That's why I think people over-estimate AI. They don't know enough to distinguish clever mimicry from original thought.

[@ColinBroderickMaths]

I should show this to my non-software colleague who says "just write a Kalman filter" for every little sensor problem, as if it's trivially easy.

[@perwestermark8920]

Kalman filters is mentioned in the video but it isn't so much sensor fusion as handling noisy sensor data.

[@c.ishikawa6346]

Thank you for the lucid explanation. Japanese media did not seem to convey the essence of why the failure occurred, yet (?). I understand the problem with the software. I often insert "this can't happen." processing in my code. Tough, it was not remedied somehow by secondary sensor or some sort of re-calibration effort. I wish the project a success next time.

[@matthewjames7513]

makes you appreciate the perseverance rover landing through the 7 minutes of terror even more.

[@KubotaManDan]

They should include a drone deployment prior to landing to descend to the landing spot 1st while sending back critical data to the lander software

[@rajamicitrenti1374]

The growly male vocal from their backing band that was doing the "Gimme gimme" part does have a few songs where it's more prominent, but as far as I can remember, the main gals do not go growl vocals ever. I think my favorite song of thers is Akatsuki. It's more of a metal ballad song, and it's a solo piece for only Su-Metal.

[@penguinista]

I give them credit for being willing to face the problem. Too many large organizations become unable to look at errors honestly.

[@LMarti13]

3:43 if you want to just get to the error