Why was Facebook down for five hours?

[@sykskysyk]

Facebook going down was a small price to pay for getting this incredibly informative video from you, thanks Ben!!

[@froller]

Let's bring down some other large network in some weird way to get next video! ...just kidding.

[@lilstubthumb]

I would pay for Facebook to be down.

[@froller]

You forgot to add "forever". :)

[@lwizzit]

It’s a win/win

[@francesco45]

Not for Mark ahaha

[@xXRedTheDragonXx]

"I'm connected to a router on AT&Ts network that I happen to have access to" As a former NOC tech for an ISP this was absolutely hilarious to hear you say this so casually. They don't just give anyone console access to random routers!

[@yves.dantas]

I think he still works as a nw engineer. If im not wrong he even worked for juniper

[@dddd6606]

This is better explanation of DNS than the one I had at university, thank you.

[@dm3on]

That's because he is self thought, not sure if Ben has high school diploma.

[@Ripsaw460]

@@dm3on "self thought"

[@Ripsaw460]

@The Poison Donut not only that but for even a person that isn't very knowledgeable in this field, he makes it very interesting to watch and all his videos are very easy to follow and i almost never find myself lost trying to figure out what is going on, ben is the best.

[@Nick-lx4fo]

@@Ripsaw460 oH nO, a slight spelling mistake that totally obfuscates the meaning of his message, what will I do?

[@Ripsaw460]

@@Nick-lx4fo you dont have to do anything, just thought it was funny.

[@tecno2053]

Im a network engineer for a medium size ISP (primarily enteprise fiber). This has become my goto video to explain to people what I do. I work with BGP and the like daily.

[@penguin1714]

I'm pretty sure Ben Eater is actually a physical manifestation of knowledge.

[@12crenshaw]

He not only knows everything, he also have time to live, breathe and share his knowledge

[@charstringetje]

All you need is a default route and the ability to read, and you can acquire knowledge too. Exciting times.

[@youkofoxy]

That sounds like he is a named version of eaters from Digimon cuber sleuth. Eaters are only interested in collecting and sharing information across it's hive mind.

[@martinkuliza]

Ben Eater is an A.I. this physical form that we see is a computer manifestation of his real self, Ben Eater is like the M.C.P. on Tron Part 1 LOL

[@cyber_8778]

@Penguin Hey I remember you! You commented on my video once :)

[@laykefindley6604]

Watching one person setup global router configurations is seriously invoking wizardry levels of mastery. I am truly humbled to watch a work of fine art like this and gives me goosebumps thinking how much power one can wield with the right knowledge and access to modern society.

[@autohmae]

Now multiply this times 70 000 or so, because that is the number of AS-numbers in use in the world, thus close to the number of independent Internet networks who connect with everyone else.

[@---nw9qu]

One of the best explanations of dns, routing, and BGP that I have seen.

[@askthisscientician6040]

That had to be the best explanation of this I've seen. Not dumbed down, no hand-waving explanations, but carefully and effectively explained with a live demonstration to boot. Thank you.

[@infinitenex8165]

Man, networking is so convoluted to me. Its a miracle that the global internet works as well as it does on a daily basis.

[@GummieI]

The number 1 reason it works so well really is the redundancy. Notice how multiple time through this video he keeps talking about dozens if not hundreds of this and that. And that is what makes things work so well, everything is connected in hundreds of ways, so if a route or 2 goes down for whatever reason, who cares, there are still hundreds of routes left to take instead.

[@LMB222]

There are further tools that make automatization much easier. You set up a few things in the beginning, and then sit back and relax.

[@jdpruente]

I tell people that the entire internet only works with gentleman's agreements and handshakes. And that's when I'm being nice about phrasing it.

[@edenrose2374]

@@GummieI except its rarely the case with preferential routing. ISP's sign contracts with third parties to reduce overhead and operating costs by having them operate long-distance runs. - This gives these single-points of failure a Scapegoat if it fails. US and UK are the worst offenders of this, literally blocking or banning competitors networks at hops that would route through their network rather than around it. This means that regardless of speed or outage state, a singular ISP can ensure the bulk of their traffic goes through and is on their and partner networks. This gets worse when we start talking undersea cables. Exit Nodes (backbone) connections to externally connected networks are entirely priority based. When it comes to routing overseas, these contractors get priority. Why? rather than operating their backbone at a flat-cost for a nation, and telling the Goverment to cover the cost of operation so all may benefit... They instead choose to charge rate-based, with tiered flat based contracts available to ISP's. - This means that, your provider has to both pick a rate limit(and quality[jitter] limit) for its clients and limits routing to the connecting country to ONLY this backbone. | Sometimes they have failover's to competitors backbone lines, but since its rate-based there is caps client and SEVERE data-rate penalties. Amazon is the worst offender of this, offering basement priced backbone lines that are PAINFULLY slow(except to /some/ of their services) and this results in much of the worlds internet being measurably slower as a result... Australia has the worlds slowest internet exclusively due to no one hosting services inside australia (draconian encryption laws), and relying exclusively on Amazon and Telstra's dedicated Undersea cables to route outside of Australia(to the UK, the longest possible run). Since nearly everything is hosted in the US or germany. This means reaching these places is usually several hundred hops. - Turning the Fibre network that makes up Australia's rather robust internet infrastructure into a cesspool of garbage slow performance.

[@vintyprod]

@@edenrose2374 this is actually super interesting. thanks for sharing.

[@critical_always]

Nobody can explain like Ben does. Everything is always at exactly the right pace and detail. I kinda feel smarter learning from this man.

[@BobHolowenko]

Hi Ben. I am a network engineer for an ISP and this was a VERY well done video that talked about the "house of cards" effect of the outage. Great work!

[@casualyoutuber6239]

I just enrolled in a CCNA/CCNP course being offered at my University and yes I am as vanilla as they come as long as Networking is concerned but this video was so crystal clear that I may have just revised effectively about 16 credit hours in a single 30minute video!

[@riz94107]

I've been an internet professional since 1994, and i knew all this already, but your explanations are SO GOOD that i watched it just for fun. Great job!

[@L0j1k]

Yea I spent years banging away on dn42 but this is BY FAR the best and most succint and insight-dense "class" on DNS and BGP I've ever seen.

[@TheRukisama]

He does give excellent explanations of things, he lays it out better than I've ever been able to a non-IT person.

[@mattboardman2148]

The subnet mask description was so succinct that I'm using it internally when explaining networking to both our techs and clients.

[@Mine8maniak]

not much changed since 90s, i mean the protocols. right?

[@EdGriebel]

Same. I took a grad networking systems course, Ben’s explanation of BGP is *so* much better than the course, I wish I had this then

[@memoryvice398]

The sound of Ben's keyboard is totally ASMR for a guy like me. Brings back lovely memories of my first PC back in the 90's.

[@NickiRusin]

yeah the clackity clack clack just sounds super comfy

[@javabeanz8549]

Goes back further for me, at the keyboards for an IMSAI and a Polymorphics in the late 1970's, and the college SOROC terminals attached to a PR1ME mini in the mid 1980's.

[@izzurzuhri]

what keyboard/switch he use?

[@lordkell1986]

@@izzurzuhri probably built it himself from breadboards and electrical switches. Each key is the size of a foot and he jumps on them

[@kaitlyn__L]

@@izzurzuhri sounds like clicky Alps to me, might be wrong though, a few other switches “ring” the same way

[@jdatlas4668]

To quote a random Tweet: "it's hard to even explain to a layperson how badly you have to fuck up to get BGP and DNS both as part of the same problem. Like, you need multiple diagrams."

[@nickwallette6201]

LOL -- that was my exact reaction when I heard about this. Like, what happened to break DNS and BGP, that took 6 hours to recover from? That is one COLOSSAL screw-up. With such huge stakes, too. It did make me suspicious what _really_ happened.

[@johnm2012]

It took them 5 hours and 50 minutes to find someone with a physical key to let them into the building.

[@NavidIsANoob]

@@johnm2012 That's false. It DID take them that long to get someone with the necessary protocol expertise to fly in and fix the mess. It's silly to assume that no one on site would have a key to the building. It's reasonable to assume that a corporation like Facebook understaffs their infrastructure and doesn't retain experts to maintain that infrastructure.

[@zacker150]

On site, they use badge readers to access the building. Those badge readers ran on Facebook's network.

[@NavidIsANoob]

@@zacker150 Again, this is false. Why would a badge lock system run on Facebook's global network, as opposed to an internal, segmented network? It makes no sense.

[@thewelder3538]

You are one of the most truly educating people on KZbin. Unlike so many others that are just so superficial with their details, mainly because they don't really understand the subject matter, you still down and give the real information. Now I understand DNS and how it works and it was just nice to see it properly explained. Would have been a bit nicer to have had an explanation about the DNS protocol, but this video is still an amazing explanation. Great work.

[@gnsoares_]

I just had a class about this today and I understood it so much better in this video. Thank you so much for your work Ben

[@realdragon]

It says a lot about me when I still don't understand it

[@SteveJones172pilot]

That's kind of what I was going to say - He boiled down probably 2 or 3 days of a CCNP class into 20 minutes or so.. Of course the "real" class would have confused you with all the other options you could put on those bgp routing statements, but it does make me think about maybe this video should be a prerequisite for diving into those sections of a network course. It's really good to see the basics working first!

[@To-mos]

It's a shame the education system is failing people, I didn't start truly learning until I dropped out of college. It's bizarre how much they charge for schools considering the word is Greek for "free time".

[@einarabelc5]

That's called applying knowledge. Next time, explain it to someone else. Try the Feynman method.

[@indeepjable]

Isnt The Current Educational System Still For Factory Jobs?

[@nebula_wolf3132]

I must say, you're one of the best teachers out there, not only is your explanation simple and easy to understand, but it also lends itself to allowing people to learn on their own, rather than shoving information into our faces (thanks school). Whatever you do keep teaching, you're amazing.

[@lightdark00]

At the end should have been a cutaway to a team of breadboard computers all networked together, that made all this possible. 😊

[@TNTsundar]

Cutaway scene like from the movie “The Prestige” with an eerie music that shows a lot of those breadboards, ICs, wires and then Ben using the monitor and keyboard connected to it.

[@BertGrink]

Maybe that´ll be the topic of an upcoming video?

[@K-o-R]

@@TNTsundar You mean the breadboards that generate the AI known as Ben Eater? 🙃

[@Aighthandle]

The actual servers aren’t that different, just a lot of modularized sheet metal chassises in big housing units with so, so many hard drives

[@mjsvitek]

Next video: "I built 30 breadboard computers so I could make my own internet... The Eaternet."

[@metalpunk]

I watched this when it was a fresher video, but I'm taking an introductory networking course this semester. While we haven't covered BGP yet, we _have_ talked about DNS, and so this actually made sense this time. Fascinating stuff, even for someone who doesn't really use Facebook

[@FunnelCakeRyan]

"A router on AT&T's network that I happen to have access to" WHAT A FLEX :D

[@esepecesito]

He works there...

[@jirensan828]

Not really a flex. That server is public and available for anyone to use to validate their BGP routes

[@TomStorey96]

There's a few of them spread around, usually referred to as a "looking glass".

[@tiporari]

Lol I like the "my address has changed by now" disclaimer. Don't poke me please. You are just poking my mouth breathing neighbor.

[@ratbag359]

@@tiporari It's bad enough with bots scanning ports and trying to dump payloads into the open ports they find yet alone the comment section of youtube seeing what holes or services are avalible.

[@zacharybearden9950]

Awesome visuals. As a fellow route/switch guy - I can't pass up a video with BGP in the title. About 7 years ago when I was starting my journey in networking, I watched nearly all of your network videos. The fundamental concepts that series covered set me on the right path.

[@ahensley]

Wow! That seems like a lot of work to set up the demo you did, but then again I don't happen to have access to a router connected to AT&T's live network 😏

[@bitterlemonboy]

How does he even have access to that?

[@SirLightfire]

@@bitterlemonboy my guess is that his full time job is a network engineer at AT&T?

[@Jeff-ss6qt]

Maybe he rents a server there or installed his own? He has his own website that likely gets a lot of traffic.

[@SirLightfire]

@@Jeff-ss6qt but that wouldn't give him access to a top level route table for an AT&T router To even be able to login to something like that is a huge deal

[@scott3489]

AT&T and several other ISPs host public route servers and/or a BGP looking glass where you can view their BGP routes. Their around to assist with troubleshooting multi-carrier internet routing issues. Maybe that’s what he’s connected to? But yeah, the demo he built is incredible.

[@jonathansteward1]

Ben this is an amazing video, as a network engineer in the industry I always like to listen along to these kind of videos to refresh my knowledge and remind myself of ways in which to explain various content and this was perfect! I was really thinking of making a video on the topic but this does the topic justice and I don't think I could add anything else to the topic! Keep up the great work!

[@_Stin_]

Now, THIS is how you explain things!! Foreword, introduction, background, examples and real-world demonstration... The best explanation for how the DNS architecture works (and how the rest of the Internet works lol)

[@bobbyb42]

I'm studying for the Network+ right now, and this video is the best explanation on these topics I have found anywhere. I could go on and on about how great your videos are, but mainly I just wanted you to know that I appreciate you sharing your knowledge with us and putting in the effort to make it easy to understand and fun to watch.

[@F4LDT-Alain]

Fantastic. I came for some more details on the outage than given in Facebook's statement and I got the best BGP tutorial I've ever seen. I'm more of a system guy myself, so local routing and DNS don't have much mystery left for me. But internet routing and especially BGP has always been kind of black magic to me. Thanks to you, I've got a grab of all this and the whole picture of that outage makes sense now.😁 Great video. Your simulated networks (and routers, I guess, apart from the AT&T one?) are impressive.

[@TheUnofficialMaker]

I'm just gonna keep it as Black Magic!

[@andrewohanian5132]

I am in awe at how many views a video, that is in large part a Juniper router configuration demo, has. This has got to be the most viewed video that details Juniper router BGP configuration. Amazingly well done.

[@meeDamian]

You are undeniably one of the greatest content creators on this platform.

[@therealchayd]

I've never managed to get my head around BGP until now (despite working with network engineers whose explanations just become white noise). Massive thanks!

[@aelolul]

Thank you for this deep dive! I really appreciate it. I'm a computer professional, but my knowledge of networking is limited to the ip command, modifying /etc/network/interfaces, and cursing at systemd. I hadn't even heard of BGP before this kerfuffle. I watched a few other videos on the topic since the outage, but this one is far and away the best I've see. I feel like I actually understand what's going on. Thank you again.

[@12-343]

Same. Best I can do is IWD and copy paste commands

[@xtdycxtfuv9353]

@@12-343 same LOL. Networking is really hard :(

[@Will_of_Iron]

Subscribing to your channel is one of the best decisions I've ever made in my life. I didn't even know anything about these and you explained it so smoothly. I guess I'm blessed to have a teacher like you. Keep up the good work!

[@GeoffreyThornton-TheWinGuru]

Ben, I just wanted to say how much I enjoy watching your videos. I've learned breadboarding techniques, logic design, microcode and processor design, and hardware bit error detection and correction (among others). Then, when you create a video that is my wheelhouse, you do so in such a concise and informative fashion that I'd be hard pressed to do the same. You cover the fundamentals of Internet name resolution and route exchange in 30 minutes in what would normally be an intense week to month long seminar. I'm impressed with the level of detail you provide, along with the time it took to create a lab environment to demonstrate the operation of BGP routing. I had not noticed the Facebook outage myself, but at the start of your video, I suspected it might have been a result of BGP route dampening due to network interface flapping, but that would have been for the entire AS and not just specific subnets. Your insight and use of BGP looking glass servers to demonstrate how specific routes were missing is remarkable. I don't have any particular insight as to how this occurred either, but perhaps Facebook is using an internal route reflector or confederation to handle routing Anycast traffic for DNS and ICMP. I don't have any experience with Facebook's network infrastructure, but this outage should hopefully help Facebook and other networks design and maintain more resilient networks. Thanks again for all your time and effort in creating these videos. You have always been a favorite content creator of mine, and this latest video just adds to my adoration. I could tell you were running JunOS, but I'm curious if you were using GNS3 or some other simulator. In any case, kudos for your time and devotion to create such an informative video!

[@hariranormal5584]

isnt the commit command a thing only on junos?

[@GeoffreyThornton-TheWinGuru]

@@hariranormal5584 It's a thing for JunOS and Palo Alto firewalls (PanOS). Juniper devices also let you do a confirmed commit where if you lock yourself out by doing something dumb, it will automatically rollback to the previous configuration. Cisco devices typically commit changes immediately and so can cause issues if commands are entered incorrectly or in the wrong order. The only recourse then is to reboot the device and let the "startup-config" replace the running config. That's why you typically see commands like "copy running-config startup-config" or old school "write mem" to finalize a configuration.

[@nicholaswest9146]

Your way of explaining things is incredible! I've worked in web hosting for about 6 years and now I really understand DNS :)

[@castles990]

You're working for 6 in years in web hosting and you didnt know what dns was?

[@ilyadaemon]

@@castles990 for web hosting dns is a table like /etc/hosts :)

[@kaitlyn__L]

@@ilyadaemon plus you don’t need to know anything after the point where your web server network connects to your ISP!

[@ndupontnet]

That outage just happened seconds after the update of my local Pi-Hole DNS instances, that was quite a head-scratching experience. Thanks a lot for those explanations.

[@caleballen4721]

You know a person is a good teacher if they can communicate clearly about DNS

[@SubspaceEmber]

I actually had to do DNS config for my website a few days ago, but wasn’t really sure what I was looking at, but watching this made that make a lot of sense. You can actually almost see how all this grew from just connecting a few computers to each other!

[@sneedsfeed757]

Nice to see someone practically explaining dns lookup.Even I have never tried to do it

[@hatecrewblaze]

'An AT&T router I happen to have access to' like it's the most casual thing to have hahaha (coming from someone working in tech support at one of germanies biggest ISPs) love the way you present and ur humor :)

[@junkmail4613]

Ben, I'm no scholar at this, but again (as I say 'bout most everything you do) you'vedone a superb job of detailing intricate(unfamiliar) connections in a clear and lucid manner. Thanks!

[@qpSubZeroqp]

Agreed, thank you!

[@BeepBeep2_]

You condensed the most important bits of my whole bachelor's degree into 30 minutes. Bravo.

[@dantheman4700]

Can you please do a video incorporating NAT. You honestly have explained subnets, dns, and ip better than anyone Ive ever seen.

[@Hugh_I]

that would fit into the explanation of this video quite well. I'll try, probably not as clear as Ben could do it: His router has two addresses, one public IP address that is routable in the internet (i.e. his ISP advertises routes to that address via BGP) and one on his local network, that is not. Only his router knows what to do with addresses on that local network. Now when he sends a packet from a host on his local network to the internet, it gets send to the router (the default gateway). What a NAT router does is now in addition to just passing the packet on to his ISP's router, is to slightly modify it. It replaces the source address (i.e. the 'return' address that the host on the receiving end should send its answer back to) with his own public IP address. It also stores the info of where it actually came from in a table in memory. That way it looks to the host on the other end like the request was sent from the router's public IP address, not the local IP address of the host in Ben's network. Any reply will be sent back to that router's public address, that everyone knows how to route to. Once the router receives a reply, it'll look up in his table where the request actually came from and rewrites the destination address for that packet back to the original local IP. Now it can route that reply packet via the local network, and sends it to that local address. This allows one to use a big number of local addresses but still access the internet, even though the ISP gave you only one single publicly routable IP address. As Ben mentioned, IPv4 address space is limited (to about 4 Billion addresses), so NAT is an essential crutch to keep IPv4 working today, since that address space would otherwise be to small to address every device on the planet.

[@javabeanz8549]

@@Hugh_I not bad, a little bit simple, since the NAT typically ends up rewriting the port number as well as the IP address, but that's a minor point compared to the rest of your explanation.

[@kaitlyn__L]

@@Hugh_I my favourite is when there’s two or three layers of NAT, so port-forwarding breaks because it’s only aware of its own internal NAT table, and not the layers beyond itself.

[@mattiviljanen8109]

I had to come back to comment. It's unbelievable how well you explained BGP. We touched the subject at school and I couldn't get it. Now I did. Not that I'll ever need it, for reasons, but I did get it. Thanks!

[@Kaepsele337]

I really like your style of explaining stuff completely. That really helps me with understanding, most other people would gloss over "details" like the root DNS servers.

[@HopeRunsDeep22]

I have been studying for my Network+ Exam and am taking it on Monday. This video was so awesome to be able to relate some of what I am learning and actually see it in a real life situation to really get a better understanding of how it works rather than just reading about it. Awesome video! Thank you!

[@AndreaTerenziani]

everybody gangsta till Ben pulls out the heckin AT&T router

[@ExoticMerle]

@Maxime St-Louis , Ben has ssh access to a router on the AT&T network, and he blurred out its IP address in the video. That is not public.

[@henrychoo4361]

@Maxime St-Louis do u have access to an AT&T router? Not every Tom, Dick and Harry has it

[@HaraldSangvik]

@@ExoticMerle Wasn't that just his home router?

[@krmr]

@@HaraldSangvik No, he specifically said he's connected to an AT&T router happens to have access to and blurred out the hostname.

[@thewhitefalcon8539]

@@HaraldSangvik No, it's part of the actual for real AT&T network, the one that may suddenly stop working and cost millions of dollars if you type a wrong command

[@mahanlamee]

You have no idea how good it feels to see a video from you uploaded less than a day ago

[@rymaples]

I feel more ignorant every time I watch one of his videos.

[@HandyFox333]

Same here.

[@rainbowbunchie8237]

Dig your feet in and don't give up.

[@opendstudio7141]

What is Facebook? The only one I remember is one from the movie ARMY of DARKNESS or Ash v.s. The Evil Dead. 👻🙃

[@ZeroUm_]

Haha, and this one is one of the most beginner-level videos he has around.

[@NoahK216]

@@ZeroUm_ I'm big into his breadboard PC series but not into networking. They're very different, this may be beginner but still wack for someone not familiar

[@BringbackTBC]

Sent this video to my networking professor to show to our class. This is the best explanation ive ever seen.

[@orchidejczyk]

plot twist: it was not simulated network, Ben just was recording while repairing facebook xddd he just needed few days to do montage : D also without recording he would repair it much faster! thanks for vid Ben!

[@anti7gn]

Ben took down Facebook just to create an explanation video.

[@SimpleLangSolution]

Thank you "Uncle Ben". I've never had any formal IT education, and snippets like these are what make IT as a career the best because it kinda justifies me not going through getting a college degree in IT.

[@AtheistReligionIsCancer]

Ben is always extremely consice and to the point. Very consistent too at that. Look at how he sets up his breaboards, it is simply pure art.

[@samiurkhan]

If I were to guess, they have something roughly inspired by OpenFlow SDN internally to propogate config changes. Configs store things like DOS protection thresholds, routes to key services (likely algorithmically generated and pushed based on an understanding of machines and distances between them). I suspect a config change they are talking about damaged the route generation.

[@myname377]

nohh. they just disconneced Mark Zuckerberg 10 years old laptop :)

[@samiurkhan]

@@myname377 fb hire this man right now

[@nickwallette6201]

LOL -- yup. The SDN controller was running on Phil's workstation, because that's the IP address it was configured to use when they were first configuring the dev POC. And unbeknownst to anyone, including Phil, it never got changed. So, when Phil shut down his workstation to do an OS update, just two minutes before a scheduled schema replication, the entire controller cluster went offline and started dropping routes. You know... or something like that. :-) If I've seen it once, I've seen it a thousand times.

[@kaitlyn__L]

@@nickwallette6201 damn you’ve genuinely seen that before? That’s depressing. I’ve had stuff like that happen with my NAS and various desktops, but to imagine professionals in a (public) company doing it… actually no, that makes perfect sense. The feet tend toward the well-trodden path and all that.

[@rbertoli]

TL,DR: All of DNS servers in same CIDR block. For some unknown reason, routers stopped announcing these blocks and the DNS servers became unreacheable, making impossible to access all Facebook services. Excellent video! Thanks, Ben!

[@YandiBanyu]

My god, all this knowledge for free! Thank you Ben!

[@tsraikage]

you know the video will be as informative as it possibly can when the tutor is building a GPU on a breadboard on his free time

[@deathsheir2035]

I learn a lot from your videos. May have to watch videos multiple times to get a firmer grasp on what you're saying, but my god have I learned a lot. Also, I don't think Facebook is telling the truth about why it was down for 5 hours. Then again, I never fully trusted facebook from the beginning.

[@emeraldthunder]

Everthings so much more understandable when you explain it.

[@nathantron]

Great job explaining this by the way. It's extremely challenging to explain this in a simple way for everyone to grasp, but you manage to do it perfectly. Great work with the lab stuff too.

[@Backyardinstallers]

Ben, just found your channel and I have to say this is refreshing to see a DIY learning channel starting from the beginning... you keep learning strong... thank you for your time...

[@Levy1111]

Wonderful explanation, thank you. Could you in some future video show how did you prepare mock networks?

[@MrAntiKnowledge]

holy shit, that's much more in depth explaination that I got in Uni. The prof gave the impression, that it's just a single request and answer type deal. At no point did he discuss that the first server will only point you in the right direction, but not directy to your destination.

[@hansdietrich83]

So just like anything else in CS, the internet works on abstraction. It's a bunch of small networks, grouped into networks that are grouped into networks, ... Until you get to the ISP Level

[@smitias_8474]

Came for speculation about Facebook outage but instead found outstanding explanation on topic I was always curious about yet was too afraid to touch at all. Much respect.

[@sophiemilton5939]

"Why was Facebook down for five hours?" A better question is why was ot ever up in the first place.

[@TheLamefeed]

Agreed, the time Facebook was down probably got families to actually talk with each other, and hopefully get a better relationship. I hope all these unsocial networks would cease to exist.

[@henrydorsett6076]

Technical question != interweb-social question But agreed in general :)

[@UncleKennysPlace]

@@TheLamefeed Actually, it made people crazy.

[@ELYESSS]

@@TheLamefeed back in my days we used sms instead of facetime

[@friendly0]

@@TheLamefeed pornhub usage increased massively 😂

[@ochanlee4414]

Bridging Gap Protocol :) Basically the reason for the outage was a human error that was propagated to >1000s of Facebook's backbone BGP routers through the massive use of network automation and management from a single pane of glass! It goes to show that automation doesn't get rid of human error but lets it propagate at a large scale...Don't forget the basics and think thrice before you hit that push button :) Excellent BGP and DNS overview Ben!

[@mustafaark4762]

Again, a great video. Never disappoints man!

[@EdgarStgo]

Hats off @Ben Eater, you are a gifted man, the way you flow explaining all this is so smooth. I used to work as a Network Engineer but reached up to OSPF from the client-side, not ISP. I just learned a ton of useful aspects about BGP and DNS from the ISP Internet perspective. Cheers!

[@bassman87]

when you said AS number 7018 I immediately said AT&T. Ive peered with AT&T so many times now that their AS number is permanently engrained in my head. haha.

[@hariranormal5584]

I do know 174 is cogent and 6939 is HE, and 1299 is telia, too much interest on the hosting community so that's just stuck on my head xD

[@BobHolowenko]

​@@hariranormal5584 Years of working in carrier networks and looking at BGP tables has me reading these like maps...It's funny how numbers stick inside the human brain.

[@JonathanRockway]

I decided to look up who AS 1 is, and it's Level 3 these days. Ironically, they are not AS 3; that's MIT. They should broker a trade ;) (I used to work at 111 8th Ave, and Level 3 proudly had space on the 3rd floor. Always enjoyed that.)

[@minirop]

@@JonathanRockway then they move to England and everything breaks x)

[@meme-hj5rs]

@@minirop is it because of break-xit ?

[@Mattboy300]

I have to say that the way you explain things in all of your videos by showing all the steps and distilling things into simple to understand chunks is really fantastic. If you were my college professor, you'd be my favorite.

[@danieladelodun9547]

14:20 but do they know you have access? 🤔 imagine getting rce on an at&t router just make a youtube explainer

[@rebmcr]

He could have a read-only account for auditing security, nothing he did on the live box needed rce.

[@mjodr]

I used to work with a guy like Ben Eater at GE. Made my head hurt sometimes diving into networking this far. Great video, but now I'm having flashbacks about that, haha. I'll never be that good at understanding it, and I've been sorta in the industry for like 24 years now. Cool to watch somebody totally on the ball like this. Now that I've seen a deep dive technical analysis (post-mortem) I'm even more confused than when I started. Yeah, how did their network go that bad? There's no way when I was logged into our Cisco routers I could have done that much damage with a single command like they said.

[@inspirated]

Thousands of outlets and channels covering the Facebook outage: zero knowledge and a clusterfuck of buzzwords. Ben Eater covers the outage and ends up creating a video that (informally) explains DNS & BGP better than almost everyone else out there. This is in fact a perfect video now to show to someone who asks "how does the internet work".

[@angelgutierrez8303]

Ben has a gift to make things simple! Should keep making videos..🎉

[@sim_only]

The level of intelligence, understanding and willingness to educate shown by Mr. Eater is genuinely off the scale. He should get into robotics (undoubtedly the next big thing) and change the world.

[@loiphin]

I have been a CCIE for the past 20 years and still very much enjoyed your video :) Thanks Ben!

[@jaredteaches894]

For context, Ben Eater's expertise is actual computer networking. He speaks about his amazing journey on his podcast. He is absolutely brilliant and his story is inspiring.

[@mattym8]

Yeah, this is the subject he started out learning and his pro experience. The rest is just for fun.

[@alimibrahem8120]

Thank you so much Eng. Ben for this pro explination, I think you have to do another one for what happen yesterday.

[@DaedalusYoung]

The fact that I missed the entire event says a lot about how much I value Facebook.

[@paulstelian97]

And WhatsApp and Instagram as well.

[@whynotanyting]

Only reason I knew about it is because my mom was flustered she couldn't reach me on Whatsapp... ugh

[@ornessarhithfaeron3576]

Only reason I know it is because I don't have the apps on my phone but I was checking if my professor answered my question in the private Facebook group we have for a lesson

[@MmeHyraelle]

My mom was crazy about accessing her cadle and resetted the internet modem because even if she understands its the first diagnostic step, she does not understand why and when to do it. You still had internet, mom. Just not those websites.

[@dieselgeezer18]

i didn't even notice it. My friends were talking about it and thats how i learnt about it

[@MotoRideswJohn]

Wait. An AT&T router that you "happen to have access to"? More information required. LOL. Great video, Ben. I've worked with many different routing protocols over the years: RIP, OSPF, EIGRP, etc., but I've never had the opportunity to play with any BGP environments. Good stuff.

[@JustinMayfield]

Seriously, we need to know the answer here! That is not something normal people have access to.

[@foxonrails4140]

This guy went from looking at a routing table in his LAN to a BGP table on Tier 1 ISP's network within 10 seconds. Damn.

[@ShainAndrews]

Nope...

[@joeo6378]

Ben's keyboard sounds like an old school buckling spring type. Has that ringing after every key press but still a nice 'chunky' sound when pressed.

[@__8120]

"So I'm in the console on one of AT&T's routers that I just happen to have access to" I love that he just casually glosses over that and none of us even bat an eye. "Yeah ofc he has access to it, why wouldn't he, nothing to see here"

[@warrenpang]

As a sidenote, this DNS demonstration tells me why we always get "NON-authoritative answer" from nslookup. Thanks for telling me how to trace the DNS all the way.

[@lightblue254]

Did not notice this was 30 minutes, wow, you really made it so easy to understand, thank you :)

[@funkykong9001]

Thanks!

[@jarred7721]

Damn this was really cool, I'm glad you took the time to explain this in such detail.

[@bassman87]

So just speculating here, but I would assume that a company the size of Facebook wouldn't be manually configuring each peering router manually, Im sure they have build in some level of automation though some network orchestrator application. If they are saying that a misconfiguration caused the outage, then I would assume that admin was using the orchestrator app to make those changes, which would simultaneously push those updates to all the peering routers at once. If this is the case, there is a lesson here to the potential dangers of network orchestration and automation. We can't just have the same mindset that we normally do with manual configuration where we rely on redundant systems to catch us when we make mistakes. The networking industry is pushing hard the ideas of network automation and orchestration, and personally I think its a positive direction to head but we need to recognize the power a centralized orchestrator has. As the saying goes, with great power, comes great responsibility.

[@jenda386]

That's when DevOps becomes DevOops.

[@devnol]

I read somewhere that the Facebook reps said there was a bug in their orchestrator and it would've caught the issue but oh well

[@bassman87]

@@devnol As with most of these large scale outages and attacks we are seeing, its not just one thing, but a break down of multiple things. Like Ben alluded to, the story as to why it took then 5 hours to resort service is more telling of the breakdown in their own internal policies. The fact that they got locked out of their own datacenters shows their reliance on a few systems and protocols that when broken cause catastrophic issues across multiple systems. I'm sure Facebook will be changing policy and systems to mitigate this point of failure in the future.

[@grzegorzkowalski1507]

I love how you just casually answered my questions about the internet that have been absolutely boggling me my whole life and that I didn't know how to ask for

[@iKrizNL]

Hey Ben what software do you use to simulate the routers and peering? Great explanation btw, pitty we don't get more insights from Facebook

[@sschueller]

I would like to know as well.

[@Themoonisachees]

I assume since he has access to an at&t router that he works in the industry, which would mean he has access to disk images of the router operating system. Then it's just a matter of putting it in several VMs and connect them together, using the VM provider of your choice.

[@TomStorey96]

My guess would be either GNS3 or some vSRXs running in VMware or similar. Logical systems are a possibility, but the interface names suggest one of the first two.

[@geordish]

He is using a Juniper vMX virtual router.

[@basedbulgarian511]

Cisco Packet Tracer and GNS3 are the most commonly used ones

[@theohallenius8882]

If I ever have to explain to someone what DNS is and how it works or how internet works, I'll just link to this video, it was very well made! Well done!

[@codingstudent3163]

You are Inspiring, You are my Knowledge Angle, Bless me. Praneeth from India.

[@khealer]

Angle?

[@wol2231]

@@khealer I think they meant "angel".

[@jamesinonati7608]

@@khealer Don't be surprised if tomorrow there is a temple dedicated to Ben in India.

[@clonkex]

"knowledge angle" ahahaha

[@WobblycogsUk]

I've been meaning to get around to trying to understand the basics of BGP for ages and now I do, thanks.

[@KarenTookTheKids364]

This was way different to Alex Jones explanation 🤣

[@KarenTookTheKids364]

@Toy Bingo

[@nickwallette6201]

Huh. You're right. I didn't hear Ben use the phrases "liberals" or "agenda" even once.

[@fus-ro-dah]

This channel pisses me off, because it demonstrates how easily schools could have demonstrated what DNS actually is, instead of having us memorize a bunch of unrelatable BS. Many thanks Ben. You're the teacher I never had.

[@yokmp1]

I saw the thumbnail and thought 'yes, lets get that LAN built' Still was an informative Video even if like every major Tech-channel made one of these over the last days.

[@Coloneljesus]

Another very clear and well done video. Looks like I'll be working with BGP at work very soon, so this came at an opportune time!

[@DanielCharry1025]

This is 1M subs bait, and I'm loving it!

[@tgriff007]

This is how you do explainer videos. Terrific job.

[@divyansharora6788]

I'm just done till 6:30 but this video will go in history, for sure

[@waldolemmer]

I've finally watched all of your videos from beginning to end.