Why Did Facebook Go Down? - Computerphile

Just what was it that took Facebook, Instagram & WhatsApp offline on 4th October 2021? - Dr Steve Bagley investigates!
previously titled "Facebook's Day Off"
Facebook announcement on the outage: bit.ly/C_FB-Outage
The visualisation software Steve was using was BGP play, downloaded from the RIPE website.
/ computerphile
/ computer_phile
This video was filmed and edited by Sean Riley.
Computer Science at the University of Nottingham: bit.ly/nottscomputer
Computerphile is a sister project to Brady Haran's Numberphile. More at www.bradyharan.com

[@HerrHeisenheim]

What's a phone book you ask? Well, it's like DNS, but for people and their telephone numbers.

[@caniggiaful]

Couldn't compile. Please check for circular references.

[@EarthboundApocalypse]

@@caniggiaful - Please reinstall compiler.

[@sodiboo]

How does a phonebook resolve ambiguous usernames? I know DNS only works on domains that already have to be unique, but player usernames don't.

[@Phroggster]

@@sodiboo Generally alphabetically, order ascending, but it might depend on your location I guess.

[@sodiboo]

@@Phroggster How.. what? How can two identical names be sorted alphabetically? That's still ambiguous and only defines the order they appear, not a strategy to select the desired number

[@thedrusus]

The world briefly became a better place

[@Slay_No_More]

Imagine if all social media just snapped out of existence.

[@JasminUwU]

@@Slay_No_More that would be bad because I would lose all my friends

[@test4274]

>look at me how cool I am and now go back to reddit, discord, Twitter, you subhuman soylent npc

[@lukaspinoti107]

@@test4274 using 4chan style formatting, saying "soylent" and "npc". kinda cring

[@altmaster3288]

@@JasminUwU Social media friends aren't real friends.

[@danielcarlossmd]

it's like Stackoverflow going down and their devs not having where to search for the fix

[@MM-zw8sm]

😂😂

[@HenryTitor]

Stackoverflow going down is the most terrifying thing one can say

[@theKonamacona]

So many jobs would be lost on this day

[@JonatasAdoM]

@@HenryTitor It would take all overflows with them.

[@JonatasAdoM]

Reminds me about that Onion news where Congress wants to pass a law but they don't remember how to do it anymore. They even vote to call a retired member for help. At least I think it goes like this.

[@jonathanjacobson7012]

Ironically, Facebook's mission statement is best fulfilled when Facebook is disconnected from the Internet.

[@Dutch3DMaster]

A Dutch group advocating for internet privacy send out a tweet during the outage: "Wow! Facebook is adhering to practically all privacy-statements in the GPDR for about 5 hours already!" :P

[@Dutch3DMaster]

@@glas4849 Bits of Freedom ;).

[@gerdsfargen6687]

Whats their MS again? Making the world a better place ?

[@Dutch3DMaster]

@@gerdsfargen6687 I thought it was "Connecting people" which ironically was impossible :P.

[@LaNoireDetruit]

@@Dutch3DMaster That was Nokia.

[@robertoricardoruben]

The phone book names are the same that the teacher calls in "Ferris Bueller's day off". Brilliant

[@ferdievanschalkwyk1669]

Love the Apple magic mouse in its natural position, upside down, asking you to recharge it.

[@TiagoJoaoSilva]

Savage (but true)

[@JNCressey]

aEsThEtIc

[@mrnarason]

Apple brilliant engineering so you can't charge and use the mouse at the same time

[@brianransom16]

@@mrnarason Great at making software, HW not so much.

[@ThisNameIsBanned]

One the worst designed hardware in history.

[@Muthwill]

I love how you're talking about DNS, IPs, ISP and several technical terms but need to explain what a phonebook is

[@MrRyanroberson1]

What's a phone book, grandpa? "Well, an old philosopher put it like this: it was the great book of doxing. It told you the names, numbers, and sometimes locations of everyone in town."

[@MattRose30000]

From today's standards it's quite baffling that people would just put their names, phone numbers and adresses into a book that's given to everyone... But I guess social engineering wasn't nearly as advanced back then.

[@bpj1805]

@@MattRose30000 What should rather be baffling is that we tolerate people who harass others, who bring their petty squabbles into people's personal spaces. Without those people, doxing would be an impotent act.

[@kilroy1964]

Phone books also make great makeshift boosters!

[@trempton4106]

It is also very interesting that because you have a phone number technically you could be disturbd in your everyday life by everybody else on the world that has a phone number.

[@kilroy1964]

@@trempton4106 And this is a tactic actually used by authoritarian mobs these days.

[@allesarfint]

It was beautiful, but as everything beautiful it didn't last long.

[@retmotiv]

I love the song of your pfp

[@VivekYadav-ds8oz]

A thing isn't beautiful because it lasts. - Vision

[@WarDogYtPersonal]

Facebook played hide and seek with everybody

[@ArgentavisMagnificens]

Yes, and that's my secret for eternal life

[@virmirus]

I dunno... Stars are beautiful.

[@LoPhatKao]

imagine a world where it took days rather than hours to fix personally, i think this timeline lost

[@steveclem7873]

Benson&Hedges!

[@ragnkja]

@@steveclem7873 Is that meant as a reference to _Cuckoo's Egg?_

[@CFSworks]

@@ragnkja My first thought as well

[@4lecsg]

"The major difference between a thing that might go wrong and a thing that cannot possibly go wrong is that when a thing that cannot possibly go wrong goes wrong it usually turns out to be impossible to get at and repair." Douglas Adams

[@steveclem7873]

wrong=thawteEN2zAA!

[@kanjakan]

That's actually pretty true, yeah. When you believe that something cannot possibly go wrong, you usually don't even try to have any failsafe for it.

[@well_as_an_expert_id_say]

My mind immediately went to the Titanic and the number of lifeboats on board. They're just taking up space anyways, the titanic could never be sunk

[@EgonFreeman]

That's pretty on-point. This is why the key first step in disaster management planning is to reliably identify _what can actually break,_ and how. You first have to accept that something could break, so that you design the thing with service accessibility in the first place.

[@EllaJameson]

Today I learned that with the BGP protocol, you have to constantly scream that you still exist, or everyone will assume you died.

[@NeonNotch]

@Dyanosis it’s not that deep, enjoy the joke

[@JonatasAdoM]

@@NeonNotch It's neither a joke or a meme, yea I'm going there. It is more like a fun fact and I'm annoyed of reading "it's a joke" on most threads, even when it has nothing to do with an actual joke!

[@EgonFreeman]

@@NeonNotch It actually _is_ that deep, because your brain is doing that as well! If it doesn't get feedback from your body's senses, it'll start increasing the sensitivity to _provoke_ a reaction.

[@fmdj]

We need to start explaining things the other way around: a phone book is like a DNS for phone numbers.

[@KaiHenningsen]

Actually, a phone book is more like the HOSTS.TXT file mailed out or downloaded from SRI-NIC - the thing that caused DNS to be invented because it didn't work so well, one list of all hosts. The only remains left of that are the local /etc/hosts files (I believe they still have the same syntax).

[@fmdj]

@@KaiHenningsen interesting, didn't know about that

[@grahameida7163]

One thing I learned the hard way on this, whenever working on a remote router always set a timed reboot on the router before you make the configuration changes. So if the worst happens you just wait for the reboot into the saved configuration and not the running one. instead of jumping in your car to the remote location.

[@golebiewsky]

JunOS and confirming commits FTW

[@srproductions8798]

This would take too long for all to reboot and facebook would be down for 5 to 10 mins

[@benneboii8117]

@@srproductions8798 which is alot less than ~6 hours.

[@srproductions8798]

@@benneboii8117 true but thats 10 mins st first..in total it would be 2 hours evetually...because once it restarts the timer shifts by some seconds...plus it would need to restart in a specific order otherwise an error would occur

[@sodiboo]

@@srproductions8798 what? if it succeeded you'd cancel the timed reboot - wdym 2 hours eventually? how?

[@AttilaSVK]

This reminds when I worked on tech support at an antivirus company. The IT admin of a mining company called us that he had accidentally pushed the wrong configuration out all of their computers, including the mission critical ones down in the mine. I cannot recall what his config was meant to look like, but the wrong one was set up to run a total scan of all hard drives every 10 minutes. Since a single scan took significantly longer than 10 minutes to complete, many parallel scans were running, slowing the system down even more and every 10 minutes a new scan job would start. We had to use a special, last resort tool, which would kill the antivirus kernel, but getting hold of it was hard (it was given out only with a special permission). My advice for that IT admin was to try new antivirus configurations on his own PC first (or get a sacrificial one for tests), then push out the config to 10 machines (preferably of tech savvy people). If there are still no problems, push out the config to 20-50 more computers, and so on, but never ever do it to the entire network.

[@Dutch3DMaster]

That's not really possible with something like the internet, there might be a small propagation delay after making the configuration change, but it's pretty much a self-learning thing from there (hence why some experts consider this the weakest link of what makes the internet work, because self learning protocols can also cause the downtime of one very important link to possibly overload another, much weaker one).

[@hillaryclinton2415]

So..you are saying pushout the vaxx for a small group first?

[@Phourc]

@@hillaryclinton2415 Isn't that just clinical trials?

[@Crazmuss]

You suppose to have test environment actually. Probably even several test environments.

[@Dutch3DMaster]

@@hillaryclinton2415 No. You can't compare stuff like this with vaccination campaigns in any way... Once a change is made in the internet and the configuration in this case has been made permanent (there's a Cisco term for it that I just can't recall...), the changes will be quick, slightly delayed maybe, but quick enough from you being able to stop the damage being done.

[@kwanarchive]

Whoever slipped in that Bueller Bueller Bueller joke in deserves a raise.

[@marklonergan3898]

It may have been a throwaway question, but i was talking to my 11 year old nephew a couple of weeks ago and i mentioned a phonebook and he didnt have a clue. His best guess was a diary that you write your names and numbers into but he couldn't get his head around the idea that we used to get a catalogue of everyones phone number posted out to us every year.

[@metacob]

I don't think I've used a phone book in this century at all... To be honest, your nephew has a point, it is weird! Nowadays you wouldn't just put your name and number in a public place.

[@soul0360]

@@metacob Actually, at least in my country. Everyone who doesn't specifically opt out via their phone company. Has their name and number advertised on the internet, in searchable registries/"phone books". These same registries are what's used if you have an app installed, that gives you the name of an incoming number, that is not in your phones internal contact list. So I'd imagine, most people, including you, regularly use the equivalent of a phone book.

[@SineN0mine3]

You should have explained that we mostly used phone books as paper weights and door stops, looking up phone numbers in them was just a handy bonus.

[@metacob]

@@SineN0mine3 Also: feats of strength. You're not really strong unless you can rip a phone book in half.

[@metacob]

@@soul0360 that's scary, what about phone scams and doxxing?

[@gasdive]

A phone book is like a DNS server that's updated every 12 months.

[@yoshikagekira7600]

lol

[@steveclem7873]

quinkxEDjawjYooNoLizArx,,,fuzzywuz pritiprintapuntzzz

[@Mr.Leeroy]

you mean like 31104000 TTL?

[@jonathanjacobson7012]

Although it can't hold doors open

[@RustyBrakes]

So could one make an analogy that it's like a company deleting their address from their website, revoking it from Google maps and any other database? The roads could still take you there, but knowledge of where it is quickly dies out

[@AMTunLimited]

More like they took down all of the signs on the building. Then the Google maps car drives by, noticed that they took it down, and deleted the entry themselves, then Uber and Lyft noticed that Google Maps deleted it and removed it from their routes and so on

[@EgonFreeman]

@@AMTunLimited This is actually a very nice explanation for the discussed mode of failure, because even if _you_ happen to know it's still there, you can't get directions on Google, or a lift there because Uber doesn't care what _you_ know. :)

[@AlanTheBeast100]

I like that FB designed things to lock themselves out of FB.

[@EvilNeuro]

What?

[@Soken50]

@@EvilNeuro Facebook's DNS architecture was briefly a "keys locked in the car and I'm outside" kind of situation until they managed to sort it out

[@EvilNeuro]

@@Soken50 oh my bad.

[@michaelmwasela5249]

@@Soken50 SO did they break the windows? Or they had to let some guys into the server rooms without key cards? lol

[@Soken50]

@@michaelmwasela5249 I don't have the details but I imagine they found a way to communicate without using the Internet so the people inside could get the credentials from those outside and unbrick their very heavy and expensive paperweight

[@FlorianEagox]

it's kinda wild to me how people LOST THEIR MINDs over this. Back only a few years ago, stuff would go down and you'd just say, "idk, shit's down right now, I'll try again later"

[@archivethearchives]

I read a lot of people explaining that in a lot of countries, whatsapp is a very common primary source of communication through phones because texting is not cheap in a lot of places. I am not sure about the full context but with a pandemic happening I know remote communication is pretty essential to staying in touch.

[@FlorianEagox]

@@archivethearchives ah, okay that makes sense then. For folks that depend on it, it's totally reasonable.

[@Fataha22]

@@archivethearchives can confirm Here 1 gb data plan equal 4 sms

[@gabrielcoelho1623]

@@archivethearchives this is pretty much it. I live in Brazil and I can tell you: NO ONE here uses SMS or iMessage or whatever. The only texts you'll receive are from your phone carrier, your bank and those phone confirmation texts when you sign up for something. Day to day conversations with friends, school work, even whole businesses revolve around Whatsapp over here. When it's 100% down like that, it's truly bad. It's not just because of the pandemic either, it's always been this way. Well, not always, but I'm pretty sure we've been using for at least family and friends communication since before the acquisition by Facebook in the early 2010s.

[@animewow311]

@@archivethearchives It's not only cost. It's cultural. Where I live SMS is not particularly expensive, but the idea of using it for communication is very unnatural. Like, I don't think people even consider it. Whatsapp is simply the regular mean of communication.

[@locust76]

I don’t understand how Facebook didn’t have an out-of-band back door in case their ASN completely vanished from routing tables everywhere. I mean, these guys helped write the book on data center infrastructure Best Practices

[@casperes0912]

Read several of their research papers in my work on creating CRDT systems. They do indeed tend to know what they're doing

[@lyoselli]

they use BGP all the way to their top-of-the-rack switches. could be that they screwed up their internal network too. if that's the case, though, then why don't they have an out-of-band management network based on other hardware than their main net, though. I don't get it.

[@ferdievanschalkwyk1669]

I am guessing their management net has more traffic, redundancy and complexity than most people production nets. For all we know, they borked the firmware on all the routers.

[@jaylewis9876]

Perhaps they decided there might be a need to disconnect from the world. Like Switzerland having explosives on the bridges into the Valley as a way to make invasion hard (predating aircraft). Having alternate back doors might be bad.

[@jonathanmitchell9779]

Concur. The notion that a trillion dollar corp doesn't have a separate physical network in place for remote management, in case of disaster, sabotage, etc, seems completely absurd to me. Giving them a pass on this less than 24hrs after because it 'must have been a bgp config mistake' is equally ridiculous. It will be very interesting to see what actually happened.

[@flamencoprof]

The national network I have worked on since 1975 used to have a thing called "the Shout-down". It consisted of analogue audio circuits over copper only, between major control rooms or what-have-you. Everyone had to call in every morning to check its integrity. It was intended to circumvent network failures and keep communication alive robustly between these places. (As a young man then it reminded me of those old marine speaking tubes from the early 20th century; Phweet! "Half astern!" "Aye-aye, Captain! Half astern it is!") I knew the rot had set in when about a decade ago some genius decided that it was old-fashioned copper tech and they moved it to a VOIP connection.

[@absalomdraconis]

Sorry to clue you in, but here's a dirty little secret: if those were lines leased from the telephone company, then it _wasn't_ an analogue direct connect anymore, just a digital simulation. And for any "national network", well, unless they _were_ a telephone company then it was pretty much guaranteed to be a lease.

[@Dutch3DMaster]

@@absalomdraconis Pretty sure they were analog, my country has been quite at the forefront in converting stuff to digital because of ADSL coming up at the end of the 90's and I think the first fiber optic cable between phone hubs was layed down somewhere mid-80's, so flanencoprof was definitely talking about analogue-over-copper, we even used a direct 1-to-1 copper connection up untill 2015 at the local radio station to connect to our transmitter site, a type of connection then already phased out by the national network operator in 2005.

[@steveclem7873]

@@absalomdraconis CEN2REunWhileHowelzaRT10hexBintvEMJurmz!

[@Novacification]

When I was studying to be a software engineer we had a semester on networks. We were assigned network equipment and tasked with setting up a network with security, services and so on. It was common for students to update the router configuration resulting in cutting off the connection that they were using to configure the router and having to get help to reset things. Almost everyone did this (including our group) and it was like a rite of passage.

[@gustaf0902]

Been there, done that

[@babybirdhome]

Mine was creating a firewall DENY rule for a single IP with a /0 instead of /32. OMG that was an awful day!

[@Dutch3DMaster]

That sounds like Cisco lessons, we had to do that as well. I liked the practical side of the lessons a lot, and even managed to choose a router once that after all the troubleshooting commands in the book according to me was deemed broken on one of it's interfaces, something later also confirmed by someone from the IT department :P.

[@tednewkumet]

Somewhere out there there are now two people one with a C64 the other an Atari 800XL with dial up modems. Their whole job is to call into the secret Facebook BBS interface to jump start the network the next time this happens.

[@NickiRusin]

all we have to do is kidnap these men

[@eekee6034]

LOL! Best comment I've seen today. 🤣

[@peter2327]

Nice idea :) But at least here in Germany, analog landlines have been replaced with IPonly DSL & VoIP, so modems are useless.

[@Tudorgeable]

@@peter2327 this is why we can't have nice things.

[@peter2327]

@@Tudorgeable for example ENiGMA½ BBS uses openssh as a "modem" Other than that there are other means of concealed and private communication.

[@OCPyrit]

It's remarkeable that one update affected all their servers. I imagined their EU and NA networks for example would be kind of seperated.

[@Megaranator]

Could be that EU network was dependent on the NA network.

[@llearch]

The issue is that the update in question nobbled (amongst other things) the authoritative DNS servers. So while everything else might be separated, if DNS goes down, EU could well be up and running happily, but unable to be contacted due to the US-side DNS servers being down. Redundancies for this would be having a separate set of DNS servers over in the EU, but that's likely not happened for other reasons - GDPR and similar things might have impacted how FB run their internal network. Or they might have been working on it, but it's not yet functional. Or they have it, but the secondaries require the primaries to be online. Or they got both with one shotgun blast. We really don't know, for all that the news is claiming.

[@benjaminmiddaugh2729]

A fairly major hosted application provider recently had an outage caused by an update module on their Gentoo Linux-based systems deleting the approved software package list. This caused all systems that received the update to remove all the software installed on the server, which made them effectively useless (and difficult to fix). They got it fixed, but it took half a day.

[@olivier2553]

What was not said here is that BGP operates not on routers, but on what is called autonomous systems (AS) and it seems FB has only one (or two AS) even if they have multiple data centers across the world. So if you mess up with BGP, your AS becomes unreachable, all your datacenters are unreachable at the same time.

[@autohmae]

@@olivier2553 it looks like the internal network was also broken, most likely it stopped talking to the outside world because it didn't have an internal network anymore.

[@alangraham6164]

A minor correction is that iBGP is not a separate protocol for use internally but just part of BGP that is called Internal BGP (iBGP) when the peering between devices is within the same AS (autonomous system). An AS is defined by its AS number and if they are the same then that’s iBGP.

[@abg44]

So there can be separate AS with the same number give by the IANA ?

[@KaiHenningsen]

@@abg44 Given that the number _is_ what defines an AS, the question doesn't make sense.

[@michaelmwasela5249]

@@abg44 YOur not allowed to think that way, no no no!

[@abg44]

@@KaiHenningsen That was my fault reading the question wrong after pulling a night shift lol. My mistake

[@abg44]

@@michaelmwasela5249 woah woah !!

[@rabidbigdog]

'A JCB through a fibre-optic' - such an understatement and incredible how often it happens.

[@steveclem7873]

goldtfeesch hazza no katraxpilloz!![sparc no alfees!]

[@farhanmahalludin]

People thought that their internet went kaput during Facebook's outage, even my ISP had to clarify that it was Facebook's fault and not their network's. Just shows how large Facebook is.

[@sujimayne]

I heard some kids on a bus talking about it and saying "Yeah, I felt so disconnected", "It's like my whole internet was down", "I was so bored, didn't know what to do" and such. The only thing I noticed was WhatsApp, and so I just used Viber, Signal and normal messaging. It's important to always hsve backups, alternatives and not keep all your eggs in one basket.

[@Dutch3DMaster]

In some countries or very remote areas, Facebook provided an actual internet connection and it was people's gateway to a part of the internet (usually sites Facebook allows to be visited, not everything, but still). In my country where this is not the case, I was mainly annoyed by news outlets reporting on this as a problem with the internet, or as if with this not working my society was coming to a standstill, and talking about withdrawal symptoms... I had hoped at least news outlets be a little more sensible in recognizing there is so much more to do outside Facebook, one of them even went "Now it was Facebook, but let's think about the risk of Google going down for longer than this."

[@boiwaif]

@@sujimayne For some people the internet is facebook

[@steveclem7873]

lloyk has2be reeezykilld spam roytasszz!!?

[@Roxor128]

Meanwhile, I never even noticed, because I don't use Facebook, or anything they own.

[@freezombie]

What I found really interesting is how long it took to get back. You're absolutely right that there's no way to fully prevent something like this from happening (and let's assume they did everything they could), but if you can believe the rumours about keycards, locksmiths, and angle grinders, they probably could have designed their physical security in a way that makes recovery in the event of a catastrophic failure a lot easier than it was.

[@olivier2553]

You have to factor the pandemic situation: in normal situation, they would have ppl inside the datacenters 24/7, but with WFH everyone was locked out.

[@VivekYadav-ds8oz]

@@olivier2553 bro imagine being locked out in your corporation's data centre. That would be a hellhole for sure.

[@eigenl]

Ease of use and secure are always on the opposite ends of the same scale. If you make it easy to recover, you make it easy to attack as well.

[@olivier2553]

@@VivekYadav-ds8oz I am the one having the key to override the system :)

[@freezombie]

@@eigenl yes, of course. But the rumour is that the key card system used to open the doors also went down with the rest of it, which suggests to me that the security system may not have been sufficiently independent/out-of-band.

[@cheesecakedoublepeanutbutt6511]

Whoever did that, thank you for making this world a better place, even just for several hours.

[@8koi245]

I heard it was an intern lol

[@Hxcker471]

@@8koi245 🤣🤣🤣🤣

[@dianeconrardy829]

Why be down on Facebook?

[@88Xlmk]

It's like the South Park episode when the Internet went down, but for Facebook. :D Edit: I remember a story from the husband of my sister. He is a Cisco architect working for a very big and well know company at the time. One day he came back home laughing, because one of his colleagues made a mistake when entering IP addresses. Apparently he copy/pasted the wrong address and caused a power outage in a city in China.

[@Danny-hj2qg]

I also recall phonebooks having addresses and advertisements for local companies.

[@autohmae]

Didn't you mean Yellow Pages ?

[@eekee6034]

@@autohmae In Britain, Yellow Pages was a separate organization to British Telecom and published separate books. I'm not sure I remember exactly how it worked, but I think businesses got listed in a separate section at the back of the phone book. I do remember they were first allowed to buy adverts (or at least more prominent listing) in the phone book. It was a little bit scandalous because British Telecom was a national institution, not some for-profit company like Yellow Pages.

[@steveclem7873]

@@autohmae 2ohm\son\lcorls\aaahYuhzzzRedOgrezLawlurch4HMedgCheepz2FatoyENtrollaz

[@ragnkja]

@@eekee6034 Was the business section (not Yellow Pages) on pink paper, like it was in Norway the last couple of decades before they stopped printing phone books?

[@eekee6034]

@@ragnkja Now you mention it, I think it was. Pink pages in a blue book.

[@ForSquirel]

It honestly couldn't have happened to a better company.

[@aknkrstozkn]

google.

[@kanjakan]

@@aknkrstozkn That will probably legitimately break the internet though.

[@ToastGreeting]

@@kanjakan just use bing

[@bruhSaintJohn]

@@ToastGreeting eeeeeewwwwwww dude!

[@stephenmatura1086]

"Let your fingers do the walking." A phrase more apt nowadays than when it used to advertise Yellow Pages.

[@Fogmeister]

Oh! I never realised why that was the logo on the yellow pages! Makes sense now!

[@renragged]

Hahaha. Soooo true!

[@exponentmantissa5598]

Everybody has a robust resilient network until the day it goes down. Same with security, everyone has a secure network until it gets hacked. One thing I have see in IT big time is that they develops all these processes, procedures and policies to deal with emergencies and disasters but rarely do they ever test it.

[@Duplicitousthoughtformentity]

I don’t know if it has a name but I call that the Preparedness Paradox. People always feel so bright and proud of their contingency plans, but indeed, they never test them… like buying a generator for a hurricane, but not stocking up on gasoline.

[@exponentmantissa5598]

@@Duplicitousthoughtformentity Yup. I know a guy that worked for a major telco. They had a loss of power and their UPS systems got drained. What was supposed to happen was that they had a generator that was supposed to kick in. The generator had been tested from time to time but no one ever did an entire system test. What they discovered was that the switchgear blew up the second they tried to switch to backup power. The hardware that failed was worth 2K but the loss due to violations of SLAs and other costs was in the tens of millions. But worse yet they discovered that many of their systems had been running for years and trying to restart everything led to yet another round or failures and race conditions.

[@JonatasAdoM]

@@Duplicitousthoughtformentity No, not even not stacking up on gasoline. *Literally turning ON the generator to check if the thing actually works!*

[@EgonFreeman]

An emergency procedure (like a backup) that's not tested _does not qualify as an emergency procedure._ At least for me. If it's not tested, _assume it doesn't work_ and plan accordingly.

[@pgriggs2112]

Their badge access failed because they had no route to their service provider. A local authenticator or “break glass” process would work. Physical structures have “Fireman” mode, allowing access to First Responders.

[@FlorianEagox]

Ooh Freeman mode? Crowbar time!

[@hanelyp1]

A local cache of authorized access, in case the central database can't be reached.

[@rhettorical]

@@hanelyp1 Unless the internal network was also down, then the keypads wouldn't work either.

[@absalomdraconis]

@@rhettorical : The point of a "break glass" route is resilience _in case of_ such failures.

[@hanelyp1]

@@rhettorical Keypad wired direct to a computer, which normally queries a master database but has a local cache, and wired directly to the door lock. Of course the local computer and wires to the lock need to be physically secured.

[@JennaGetsCreative]

The phone book entry "beuler, f" is awesome 😂

[@ikagura]

Save Ferris

[@aland7236]

This reminded me of an event that happened here in the US back in something like 2014. An engineer at Verizon made some bad commit to a routing configuration and brought down a substantial portion of the internet in the States. That was a fun day. On October 4 2021 the sun shined a bit brighter.

[@AndyChamberlainMusic]

I'm 20 and clearly remember going through phone books as a kid

[@DevilbyMoonlight]

BGP updates very sloooowly... and is full of wonderful little quirks and traps for the unwary, which made it so much fun learning nearly 20 years ago....

[@MeneGR]

I have learned the hard way: "Reload in 5". If in 5 minutes I'm not able to cancel the reload, that's my lifeline.

[@typo691]

What does this mean?

[@therealnemini]

​@@typo691 On Cisco routers "reload" means the current configuration is not saved and the router rebooted. You can schedule the reload by using the "in" parameter plus a number of minutes. First call "reload in X", then change the configuration. If you locked yourself out, the router will reboot after however many minutes you specified using the previous configuration, and you can get back in.

[@typo691]

@@therealnemini Oh, very cool thanks

[@steveclem7873]

@@typo691 livapoolizbombaDaddyzinvadingBatolSpaizeeNCraveArse

[@dsaints2344]

The scariest thing for me, during the outage, it made me appreciate open-source, decentralized software even more, because, I get the advantages of having a centralized infrastructure allowing better monitoring of the services of the various products of a company, allowing better communications with each other (integration)... But this is the edge case scenario that reminds us to not build everything under the same roots

[@gorillaau]

The same issue arises with cloud computing. It's someone else computer.

[@deoabhijit5935]

It's more like blockchain

[@EgonFreeman]

@@gorillaau And one should _never_ forget that. The "Cloud" is an abstraction, and as Joel Spolsky teaches - they eventually _all_ leak in some way, _no exceptions._

[@dunebasher1971]

Loving all the retro computer goodness in the background. I'm seeing an early IBM PC, Atari ST monitor, Apple II and BBC Micro. Sure there's other stuff too.

[@jacquesmorrisseau7420]

Love the retro hardware in the back! For my part, I always kept a physical key to our company building because I did not trust the access system. We are now in a new building completely run by keytag accesses. I guess i'm going to add a physical key to my request list to the new building operator and thank Facebook for giving arguments in favor of this.

[@steveclem7873]

coyENroWStayTinGLassEZ! GraphiotGRAFTedWardsQUINX!!

[@mikesully110]

lol do you really think corporate will listen to you? are you one of the bosses? in my place they would ask me why am I so special that I deserve a physical key when everyone else make does with cards?

[@jacquesmorrisseau7420]

@@mikesully110 I am the IT director, my boss is the CEO. Curiously enough whenever someone in the company do not agree with me, their bandwidth get mysteriously throttled...

[@mikesully110]

@@jacquesmorrisseau7420 lol I'm the IT director at my place too but honestly my main concern is, when does my shift end. Bring on UBI if we can't have UBI bring on a 4 day workweek that's all I will say. 1950's optimism promised us 10am - 2pm by now.

[@devnol]

11:00 And this, my friends, is why we always keep two cups and a stretched wire as a communication backup.

[@jesusoliveira2]

"The thing that's probably happened here is that someone has had a bad day at work, they've pushed something up to their servers and to their switches and things, and it just didn't work." Now, that was detailed, unexpected and fact-based! Someone then thought of disconnecting the Molex cable but it didn't improve things much.

[@isyt1]

Another amazing video. I love that computing is such a huge topic that most experts only know about their wee part of it, except Steve seems to know everything about everything lol

[@support2587]

Reminds me of the AT&T long distance failure. Cascaded due to a code issue.

[@CTimmerman]

Standard insertion of nonstandard code.

[@steveclem7873]

egg smekz?...

[@fostercathead]

We're destroying evidence just as fast as we can, and expect our services to be restored as soon as possible. We apologize for the inconvenience ...

[@moistmike4150]

Epic comment. : D

[@theunknown4209]

Yup

[@HUYI1]

Evidence for what, what were they hiding I wonder...

[@dread69420]

That is so true. Federated approaches are much more resilient to these sorts of events.

[@Daniel-us1dl]

Thank you Mr Philip Seymour Hoffman for your outstanding explanation!

[@krzysztofkwietniewski9100]

At least *NSA* "Prism" network was still statically *connected* .

[@bhatkrishnakishor]

They probably run on adjacent server rack, under the same roof 😉

[@eekee6034]

I wonder if it would still be up if Google went down. 🤔🤣

[@eekee6034]

@@glas4849 Possible, but I'm not sure what the motive would be. I've been thinking of social-engineering type hacks because there's this situation with China, which I'm told has been going on since the early 90s, and Russia too, and also it seems entirely reasonable to believe the statement that industrial espionage and sabotage are 100% normal now, but you never see anything overt. You know how there's always someone to explain why it's necessary to mess up the entire project for a 1% performance gain (or other excuse)? Some of those people are undoubtedly a bit crazy, but what if some of them were paid to do that by a competitor or a foreign power?

[@t34d1um]

this is amazing how clear you can explain it that clearly ! I don't know much about tech and still managed to understand the majority of the video :D

[@Dutch3DMaster]

I honestly explained it to someone saying something like this: "It's like I'm saying "Hey everyone, I'm not living at house A anymore, but at house B! And while all people who received word about this send their information to house B, house B is like "I don't know who this is, return to sender" and while things start to go wrong (I'm not getting particular pieces of mail anymore) after a while all addresses that want to send stuff to me have now learned to sent everything house B. This time however, House A was also the place where it stored credentials for physical access control, which now was clueless where to find the right data to allow entry of House A".

[@PhysioDetective]

Fascinating. Thank you On a different note, your laptop on the edge of the desk was stressing me out!

[@duncancampbell9742]

Really well explained - thanks. Having recently retired after years of data networking, my first thought was for the poor guy that pushed out the change ... and whether he (and his boss) will still be employed next week. I hope they (fb) learn from this and maybe segregate their networks a bit more ... especially (if the media stories are to be believed) the physical security and building access networks so staff can at least have a chance to start diagnosis

[@DjResR]

It was quite interesting to see how much of facebook and instagram is buffered locally on the smartphone, I didn't spot it being offline until starting to watch a video and it failed mid-way._

[@EarthboundApocalypse]

Wait until you watch how much data FB gathers from you when you allow them to access your internal storage, so you'll see whats very scary when using these apps.

[@hil449]

@@EarthboundApocalypse so what? I have nothing to hide lol

[@AuroraAce.]

@@hil449 always one with that weak argument

[@EarthboundApocalypse]

@@hil449 - Its all about privacy, not about having or not something to hide. I wouldn't like to be profiled just by my web surfing history, or when I'm shopping.

[@EarthboundApocalypse]

@@AuroraAce. - the only thing weak thing in this world is your mind for not realizing violations to your main rights. I suppose you are of those who can't hire a lawyer when a problem arrives, because you don't believe in laws, principles and rights. You may think you comply all your obligations as well, but behave like a 'Karen' when no one known is watching you.

[@shinoobie1549]

What are the odds of a once in al lifetime catastrophic failure happening exactly when the company was embroiled in a scandal?

[@VascoCC95]

what scandal?

[@sageinit]

@@VascoCC95 60 minutes

[@NeedForMadnessSVK]

​@@VascoCC95 Two things in the last 72 hours. There was a data breach and 1.5 billion users data got stolen and a whistleblower came out with internal documents and research from facebook, that show facebook, built to maximize retention time and interactions, promotes hate, polarizing ideas, depression, eating disorders and misinformation. And all important people at facebook know this, but they wont change anything because it would make them less money.

[@RevRussSmith]

The odds of a once in a lifetime catastrophic failure happening at some arbitrary time are exactly the same as the odds of happening at some other arbitrary time unless you also assume some sort of causal link. That doesn't appear to be the case. If it were malicious in intent, someone could have done far more harm to the FB brand by releasing more information of the type that just came out, not by taking FB down. If it were an attempt to do something in FB's favor the event would have been a lot shorter and stopped as soon as they realized the effect it was having on stock value. From watching the network fail, the most reasonable explanation is the one in this video. It was a catastrophic coding error that caused FB to disconnect itself from the Internet.

[@shinoobie1549]

@@RevRussSmith cool

[@TheJaguar1983]

We had something like this happen to the Australian internet a few years back. The internet provider Dodo (who I worked for years prior) sent out an update that forced all traffic in the country through Dodo's routers, slowing down the entire country's internet.

[@deus_ex_machina_]

Did anyone even notice the difference?

[@tenseikenzx-3559]

Probably not, our internet could be a lot better

[@TheJaguar1983]

@@deus_ex_machina_ Yeah, everything was ⅓ to ½ speed.

[@deus_ex_machina_]

@@TheJaguar1983 It was banter about Australia's stereotypically slow internet speeds. The joke is that it was already so slow that no one even noticed that it had become even slower.

[@Spasmomen]

Thanks for the great explanation! I loved that wire diagram.

[@sasuke2910]

It's funny how at the end they seem to be saying it's impossible for Facebook to create something that doesn't have a single point of failure.

[@sageinit]

Technically correct given that we don't actually have an Internet, we only have a catenet.

[@nathanb011]

No matter how many servers that facebook makes, if the DNS doesn't tell you where they are, you can't do anything about it.

[@lordcirth]

With a sufficiently broad definition of "point of failure", yeah. Making physical hardware redundant, easy with their budget. Preventing any one admin from making large changes, hard, but possible. Preventing one bug in the distributed software from taking down their network? How?

[@brendawilliams8062]

Most things aren’t perfectly made.

[@goeiecool9999]

I think he was promoting the use of services that are not centrally controlled by one entity. Aka federation.

[@baseddepartment1306]

They had to reboot to deploy the skynet version

[@as-qh1qq]

The clarity of the explaination

[@JCTsFascinatingHobbies]

Cool, an Acorn RiscPC, just sat in the background!! One of my favourite machines of all time. I own a StrongARM SA110 upgraded RiscPC 600, which I use, occasionally.

[@mu11668B]

Pushing your new code to production that you felt like "it was just some minor errors that don't have be re-tested" be like:

[@steveclem7873]

HaRdazeKnight[Mov awe no?]...

[@SKyrim190]

2:43 "You see...a phonebook was like a physical DNS server, where you had this book that you would look up the person's telephone number by their name"

[@bcn1gh7h4wk]

IT person: "Mark, our servers are down!" The Zucker: "Have you tried turning them off and on again?"

[@JonatasAdoM]

We need to turn it on to let us in so we can shut it off!

[@mLyonJE]

There's a hint of Facebook's arrogance here. "They wrote the book on best practice", people say. They're world leaders and have extensive backup systems, apparently. And so on. The thing is, their perception was that they could put all their eggs -- external, internal, backup, everything systems all in one notional basket. I run the tiniest of networks and even I think to put some systems "out of band", so if the unforeseen occurs and kills my primary systems, I have a little alternative that I can reach/access/use. Note my use of the word UNFORESEEN. There's your "best practice" -- write the book, follow the book, and know there is more BEYOND said book.

[@steveclem7873]

finkDratBiYuhMoofiaRitENz2?

[@tanmaypanadi1414]

@@glas4849 nope they didn't get hack

[@pl4nty]

Have you read their blog posts/research papers? The affected system was their backbone - the cost of running a global OOB network just for config failure is far greater than the 7 figures they lost due to downtime. Far more effective to have strong auditing controls (like AWS, Google, Microsoft) which failed in this case. The real issue with their response was the lengthy downtime, which was blamed on security controls.

[@OlivierChambon1]

What is the name of this book?

[@boyan3001]

Email is exact example I was thinking about during whole video! I think it would be nice topic for Computerphile to discuss: open protocol based services and systems, with various implementation such as email is, vs. proprietary and centralized ones like popular instant messengers (from ICQ, MSN and Skype to modern such as WhatsApp, Viber, FB Messenger etc.) or whole platforms like Facebook, Instagram are.

[@autohmae]

So basically a video about: centralized, federated and decentralized.

[@somenathsinha210]

Anyone know what the BGP path visualization software was? That looked pretty amazing!

[@miss_sapphire]

Same!

[@namedeleted5329]

Hey, almost 2 million subscribers, great job, computerphile!😀👍

[@honestgoat]

Almost 2 million. Congrats guys.

[@cheguevara8500]

The guy who pushed the wrong config got probably very hard time after he realized what happened. I wouldn’t like to be in his shoes. Poor guy.

[@quintrankid8045]

What would you do to him if you were the boss of all bosses at the company?

[@autohmae]

I wonder if they use a git review process before pushing changes, that would mean it was checked by one or more other people before the update went out on the network to reconfigure the network.

[@autohmae]

@@quintrankid8045 at first ask what happened.

[@globalincident694]

@@autohmae They must use some sort of review process, they're not idiots.

[@autohmae]

@@globalincident694 So Facebook released more information, it wasn't a configuration change, they were running a command on all routers to request some information but the command did something else by accident.

[@DavidKutzler]

This incident is a good argument against the IoT. If a single-point failure can knock out your phone system and lock you out of your own building, it's a real vulnerability.

[@thefirehawk1495]

I could actually feel the positivity returning to people around me and people engaging more with eachother. Can't we just end Facebook?

[@kapa1611]

👍 here is something funny i heard: when the arab spring happened, the shutdown of social media (by the regimes that wanted to undercut the protests) accelerated them. apparently because people started talking to each other more in person xD

[@SephirothEGS]

That map of nodes with the flashing connections reminded me of that scene in Jurassic Park when they look at the map full of failed security systems and they realize that they're screwed. I imagine the people at Facebook had much of the same "Oh crap..." feeling.

[@levyroth]

For a short while, the world was simply normal. Wish FB stayed offline for good.

[@DeosPraetorian]

And it would be replaced by something else

[@chadfreestyle4371]

I feel like a cave man who literally walked out of a cave homeless knowing nothing about computer's but I'm learning and I find this really interesting. Thanks.

[@boratsagdiev6486]

It’s never too late to start coding.

[@steveclem7873]

aumheThyKatatNvohMat? oofeeeliangszzz....soapyDOpeMEanzCenTrueyanks?!!apeKink...oofNohTelefonKordaaz,,,

[@EgonFreeman]

Many people don't even _realise_ how much is going on "behind the scenes". Many _professionals_ don't even realise. And it's nobody's fault - the systems we construct are _so complex_ that there often isn't one person who "understands it all" and can solve any and every emergency... It's just _impossible_ to understand _all_ of it. Even this - the event that killed Facebook for _a few hours_ - is just the tip of the iceberg that IT people deal with day-in and day-out. So don't feel bad for not grasping this - you're in the overwhelming majority.

[@unvergebeneid]

"A bad day at work" is quite the understatement when you lost your boss seven _billion_ dollars.

[@addictedtoflames]

Tbh if your boss can lose 7 billion dollars in a 6 hour outage he can probably afford to lose 7 billion dollars

[@unvergebeneid]

@@addictedtoflames I would agree but that is of little comfort to the employee in question if their boss doesn't share that view. Also, that figure is just putting into a single number the amount of hassle you caused your thousands of coworkers and hundreds of millions of users worldwide. If the thought of so many people hating you right now didn't crush your soul then you're made of sterner stuff 😅

[@WarDogYtPersonal]

Someone lost their job at Facebook last night 🤔ᴴᴹᴹ 🤣😂🤣😂🤣 for a 7 billion f up

[@WarDogYtPersonal]

It's like taking your boss's car out and crushing it and saying to your boss no I didn't take the car hmmm why what happened

[@seneca983]

@@unvergebeneid "seven billion dollars" Where does this figure come from. Facebook's annual revenue is under $100 billion. It sounds a bit weird if a one day outage could cost such a large fraction of an entire year's revenue.

[@MsRope93]

One video that best explained what really happen with WB. Kudos @computerphile

[@AleksyGrabovski]

Every person I know felt better without facebook, they realized that there is so much free time :)

[@Konyad]

Every person you know might have social media issues

[@nomisrealfun9710]

WhatsApp is kind of needed for a lot of people's jobs though

[@Konyad]

@@nomisrealfun9710 this doesn't relate to op's comment. What kind of job need Whatsapp?

[@SuperSohaizai]

@@Konyad normal communication for non business people. For fast messages you don't go for gmail or something like, you go for messaging, like WhatsApp. Even business people uses it too afaik

[@Akshaytg]

Unfriend them, they are not the people you want to be with. If they spend too much on facebook and then say "i got better" when it was down.

[@irispounsberry7917]

"Facebook know what they're doing when it comes to networking", "they're not some fly by night operation"... which is why there's been chatter about them scrubbing some things from their network in the wake of the negative press they got the previous day. That being said, it definitely sounds like a regular corporate screwup to route everything through their own servers, depending on that constant update/refresh cycle, to secure everything in the IoTs rather than rely on any analog system.

[@moshet842]

Your comment makes no sense.

[@dr.angerous]

@@moshet842 wtf shut up

[@abg44]

@@moshet842 No, you just don't understand L2/L3 routing

[@irispounsberry7917]

@@moshet842 There have been guesses that FB needed to scrub something from their system and their logs in light of the stuff said in the news recently. In order to do that, it is speculated that they took their own system off the internet (in broad terms) to do it properly - they know their tech and how to engineer a "mistake" to give themselves the time on the servers to alter stuff deep in the code without outside observation for any future investigation. However, they are a regular corporation, likely with regular managers that know little about what the people they manage actually do. So, people who are not tech minded (like regular managers) would be inclined to route everything electronic through their own secure system, because they trust it more than other security methods. This lends credence to FB's given explanation of what happened.

[@moshet842]

@@abg44 Networking is literally what I do.

[@DeoMachina]

The thumbnail for this is pure genius

[@TheDankTiel]

4:38 That one machine thats connected to Facebook was like: "50,000 machines use to live here... Now its a ghost town"

[@sageinit]

Networks of Networks are just Catenets (conCATEnated networks). The term 'Internet' originally referred to capabilities beyond those of a Catenet-ones which the Catenet which we erroneously call 'Internet' lacks, because they sadly messed up the protocol design back in the 70s/80s. If it weren't the way it is, this sort of thing couldn't have happened. See the (unfortunately, due to his severe-but quite understandable-frustrations with the matter, occasionally very polemic ) writings of John Day on what that implies.

[@Belioyt]

Who is John Day?

[@absalomdraconis]

Haven't read his books, but from reading an interview he did on the subject, it probably wouldn't have helped. In particular, the _location_ of the problem would have moved from a support protocol like BGP to one of the applied layers of a generic protocol like RINA, with someone misdirecting e.g. nodes to no longer exist where they actually existed. John Day's solutions are not actually solutions to this particular problem, _at all._

[@sageinit]

@@absalomdraconis the fundamental multihoming problem wouldn't exist on a proper Internet, which would mean with the right setup (which isn't to say Facebook would have that!), /if/ such a problem were to occur (and, as you correctly said, it could still), the /recovery/ via out of band management would have been a lot easier & faster, possibly even fully automatedly, and even transparently

[@Peewee0413]

Nice description of the issue.... The Facebook Song : Scrub scrub scrub your servers, scrub them till they're clean. If you do it well enough, you'll avoid the guillotine.

[@philstuf]

I have NO idea why there are ANY downvotes to this video. It's accurate, it's informative, and it is entertaining...

[@nicknesler]

I would love to be in those retro meetings.

[@mikkelstb]

So everyone tried to go via Telia’s router? I guess that explained that Telia had network problems at the same time…

[@nullzero9224]

Wondering what happened to those people who were responsible for this “misconfiguration” mistake !

[@NeonNotch]

@@lostintheinternet2814 shut up

[@imho2278]

@@lostintheinternet2814 and bonuses.

[@blinky840]

My core Juniper router has a custom rollback timer. When making BGP or any other major changes I always use this.

[@Mar_Ten]

Thanks for this intresting video. I knew DNS but needed a bit more context to this!

[@dimasveliz6745]

I love this channel both because of the content and because of the desk of so many awesomely smart people looking messy as mine :)

[@JCBOOMog]

What program did he use to seed the network paths?

[@alnoobus1045]

I'm going through the comments looking for the same thing. Hah.

[@JCBOOMog]

@James Saunders post the link with spaces

[@alnoobus1045]

@James Saunders Thanks!

[@misterhoeflak]

Very interesting and clarified many things for me, thank you!

[@MatthewPiercey]

Brilliant n00b-friendly explanation and hilarious thumbnail 😅. This made me think of some kind of thriller movie, where a technician gets trapped in a massive datacenter in an outage like this. So they gotta go all Gordon Freeman to get through the complex's security systems - to get the servers back online in time, before a bomb goes off or something idk 😂

[@hoebare]

Isn't the DNS component a bit of a red herring? Surely if the Facebook hosts are unreachable due to BGP announcements (or lack thereof) then it doesn't matter whether DNS is functioning.

[@melind82]

I don't think we know the extent of the BGP outage. It could be that the vast majority of facebook's servers were still online in various datacenters around the world but none could be reached because DNS was down.

[@NeonNotch]

@@melind82 nope, BGP was definitely not advertising so intermediary networks wouldn’t even know where to send traffic to

[@melind82]

@@NeonNotch My point is facebook's networks are very distributed.

[@yarik12341]

I took a networking class last year and what he is talking about made a lot of sense. Thanks cs degree

[@steveclem7873]

declasse intelejinx railROded ritz cyr compress22 hellier joneseys ai chapshax java class? bowlEN2bird teechez lloyfaz,wikamainz? KhanSciASS!...

[@omdevs]

Awesome explaination!

[@donnamarie3617]

Excellent review thanks.