What Caused the Largest IT Outage Ever?

[@Winchumbo]

Ty for reminding me it’s Friday 🎉

[@mackenzieclarkson8322]

It is not a Friday

[@coldcanida]

@@mackenzieclarkson8322it was when they posted the comment

[@adamMovieshd00]

Y are you spreading mis information..... It is Friday ​@@mackenzieclarkson8322

[@NichoNick]

@@mackenzieclarkson8322 Bruh

[@vaGin0]

Ellis should get the trivia point every time he sweeps the crew and they all get the question wrong

[@Did_U_Know..]

He’d win every season 🧐

[@StoriesByDighe]

The questions would get a lot harder lol

[@bojome3751]

Big fan but this meta ad gives me chills every time. Such a contrast to any other ad.

[@iMikeTech811]

One insane thing about the Crowdstrike issue that you don't hear a ton about is Correctional facilities. I work in a federal jail and when this issue hit, my facility lost all computers AND all cameras. we went like 18 hours without camera access. Scary situation to be in.

[@iMikeTech811]

@@girlgreenivy Oh HELL no! Well, I assume not but some officers are idiots and flap their gums a little too much! As far as I know, they weren’t aware.

[@Pottapatri21]

David saying "Uhhhhhhhhhhhhh, I don't know" at 45:31 should totally be made into a soundbite.

[@strawhatsmanager]

absolutely 😭

[@bethanyharrison3310]

Lol yes that's good

[@thanos879]

I still remember Y2K even though I was only a few years old. We coincidentally had a power outage and I heard some random dude outside screaming "Y2K!!!!!Y2K!!!!!". It's forever etched in my brain 😂

[@theglowcloud2215]

And that man's name was Dale Gribble

[@soundninja99]

If anyone wants a less technical explanation to the crowd-strike issue than Dave's, I recommend the computerphile video on the topic. It's two CS professors explaining it to a non-technical layman. Theo - t3 also has a great video called "diving into the embarrasing engineering behind CrowdStrike" that breaks down crowdstrike's statement. It's more geared towards people with dev backgrounds that aren't versed in security, but I'm pretty sure it should be perfectly understandable by any layman. But it's very ranty, so you've been warned.Also to comment on a couple of the things you guys were wondering about: You're completely right that they tried to find a way to circumvent Microsoft's validation methods by pushing code to kernel level software without pushing code to kernel level software. In their legalese, they're trying to justify this by saying it's just binaries labeled as content and not code (which is true, but they're not adressing the fact that it's just as bad). Yes it's allowed by microsoft, but it's the same way loopholes in the law are perfectly legal. It's against the spirit of the rules. About not testing on production. Software validation is never perfect, as these tests are written by engineers, often the same engineers that write the code that the tests validate, and you can't always catch everything. That's why staggered release would be better. Essentially, you send the update out to a very limited number of machines, preferably your own machines. Let's say 100 test units somewhere that the devs can check on. Then 1000 client machines. Then 5000 more... etc so that you have much more time to make sure nothing breaks as disastrously as this.

[@dit100]

The bigger issue was that they didn't target the deployment. It just went everywhere all at once. Even after testing, you normally push things out in waves so as to not have issues like this. Also the reboot 15 times was because of when the BSOD occurred. the computer was booted for a really short amount of time.After 15 reboots it was possible the CrowdStrike fix would reach the machine while it was online before it died and allow it be fixed. Another big issue people had were that their bit locker keys were stored on machines that couldn't boot so IT had to fix infrastructure in order to get the keys to even try to fix other computers.

[@dang3304]

Releasing on a Friday is fine, especially for anti-malware companies that are 24/7 operations and need to react quickly, but they didn't do any manual testing, didn't have working automated testing, and didn't do canary / pilot releases. These are the basics, they aren't hard to do, they don't take long, yet CrowdStrike failed to do any of them.

[@dit100]

@@dang3304 no argument from me there. Also why I refer to them as Clown Strike.

[@dextert20]

@@dang3304most antimalware companies release signatures everyday but avoid engine updates on weekends. Most also release by timezone or region. 20 yrs ago, trend micro released a bad update but affected mostly japan and was fixed immediately before other countries get it.

[@dextert20]

​@@dang3304most cybersecurity companies release signatures regularly but avoid releasing engine updates on weekends. most also release by region or timezone not all at once. dogfooding couldve helped them big time. deploy it first on your own network before pushing it to the world

[@U2haveanicedayty2]

​@@dit100Sorry maybe it should not be on your Shoulder when should not be from beginning but Thanks for explaining.💯✌️

[@jaganathannatchimuthu4317]

because of all the actually potentially apparently allegedly stuffs in this podcast, this could potentially be the biggest and the greatest tech conspiracy theory podcast. Welcome!

[@NichoNick]

Potentially apparently allegedly this may possible be a comment on allegedly and potentially the maybe biggest conspiracy theory podcast.

[@xayushx5327]

I didn't know *Majin Buu* was into tech. Now Goku is literally dumber than Buu

[@just_mdd4]

💯😝

[@hi55us2]

Ellis explaining the Hall Effect lol

[@Pjty1]

As a keyboard enthusiast I’m glad that you mentioned Wooting as the pioneer in hall effect switches and analog input for keyboards. I think Wooting as a company is still way too underrated, even tough (as the underdog) they single handedly changed the entire gaming keyboard game and and were the innovative spearhead for years (and still are)

[@wtfnahmean]

41:11 If apple's gonna make a flip phone they HAVE to provide real functionality on the cover screen. For instance Samsung cover screen, by default, is quite limited. However the good lock app and the "experimental lab" option gives users a lot of flexibility and allow all apps to run on cover screen, even if they aren't officially supported. I highly doubt Apple would give us that option. A flip screen with a bad cover screen is kinda pointless IMO

[@cocutou]

George Kurtz, the CEO of Crowdstrike was a previous CTO of McAfee in 2009. Back then, McAfee also BSOD a bunch of Windows XP computers by pushing out a software update that deleted Windows system files. Let's just say, this guy has a history of causing worldwide computer outages and didn't learn a single thing.

[@JoshBrahm-o5x]

Now I need David to do a deep dive episode on Y2K and why it didn’t go down the way everyone feared it would!

[@simranramchandani1248]

YESSSS

[@DougWolfe]

It didn’t go down the way they thought because thousands of programmers made sure it didn’t. I work in IT so my gang was in the office at midnight on New Years Eve in case our 600 PC office needed intervention. It didn’t so I popped the cork on some wine for the gang and we all had a drink. We were all bummed when so many “leaders” later said the scare was for nothing and they didn’t need to have spent money. Fuckers had no clue disasters didn’t happen because we all worked hard to make sure it didn’t.

[@nexusyang4832]

Just wait until 2038 rolls around…. That’s the new Y2K…

[@sean_vikoren]

Nice nerd out on the 'hall effect', otherwise known as; magnetism and electricity are two side of a very strange coin. It's also how to cheat at the 'i'm not touching you' game.

[@meta_N0X]

It was 10AM in the morning and I was in a Teams call when suddenly I got the BSOD. I spent the next 3 hours trying to reach out to the tech support and finally when I got connected I was informed that it was a global outage. Over 60% of the systems from my company were down. And to solve this you either had to take your machine to the Support centre or connect with Tech support... Which took another 2 hours on Sunday to fix.

[@upstarter]

Re: Razer keyboard. This conversation is as old as time in technology. For example, I DJ for over 24 years. You can imagine going from vinyl to digital media how much the equipment has changed. “Back in the day“ DJs had to manually beat match songs so they mixed without sounding like a train wreck. Now, or I should say quite a while ago a feature was introduced called “SYNC“ to wear instead of using the pitch feeder and manually having to listen to the track to judge how fast or slow it is, you press the button and it locked every track loaded to the correct beat per minute. TJ is all around. The world had strong opinions about it, and now a decade or so later it’s as ubiquitous as a technic turntable used to be. Move with innovation or get left behind by it.

[@coreydubia2881]

Was literally using a DC33 when I heard the second trivia question. Nailed it!

[@BreakawayB]

That “Steeplechase” joke made me LOL so hard 😂

[@sachinraj126]

Drawing QR codes out? 😂 I see what you did there Marques. Subtle and effective 😂

[@vivaanreddy8771]

UPI?

[@chasingdogs]

I appreciate the Dave's channel mention. I also had looked it up and his video was pretty info dense and educational.

[@mattjgraham]

It's finally the weekend in the UK, guys! First week back from holiday in Greece. But always time for the best tech podcast! Waveform, here we go!

[@nikeadidas9555]

It would be honestly cool to have David Plummer be a guest on this podcast. Guy has a lot of insight since he’s an OG dev for Windows

[@godbeforeme]

I wish I had my own personal Ellis to answer my own random tech questions in such a concise and instantaneous manner.

[@just_mdd4]

Escobar Phone mentioned! 😝

[@moisheschnitzler1262]

Now u understand why....

[@wtfnahmean]

29:00 People like to post on reddit when they see infrastructure, military and other critical machines use old ass OS from 30 years ago, this is why. The longer they prove to be reliable the better they are for the "job". If it's not broken never fix it

[@Eesemoore]

Ellis casually growing a moustache. And nobody is talking about it.

[@SellyS.]

Dave also mentioned there was apparently some regulation that prevented them from implementing an API that wouldn't prevented this sort of thing from happening...

[@TiagoFerreiraHazger]

About the Keyboard. This looks very close to what Fighting games had with hitboxes. Hitboxes allowed players to press 2 oposite directions at the same time, so if the game wasnt expecting this we could have some perfect blocks or charging back while moving forward. To fix this issue after the hardware that allowed this was invented they implemented SOCD ( Simultaneous Opposing Cardinal Directions). I believe that we will have the same for the FPS in the future. If the game wants all the players to have access to this feature they could code that the moment we press the oposite direction they ignore the direction until we lift the key (keyup). In the FGC (Fighting game community) controllers that dont have SOCD are not allowed on some games, other games like SF6 and Tekken 7 deal with SOCD at the game level so its ok to use a controller that dont have the SOCD on software level. But games live MvC3 (that where released before hitboxes where created) dont deal with this so the organization ban some controllers from it. More information on this bans in tournaments can be seen at the evo 2024 Rules

[@harshakj2946]

Was waiting for this ep 😁😁

[@likoetsvp]

So glad you all mentioned Dave’s Garage. I thought he explained it better than anyone else.

[@7zark785]

Definitely better than these guys.

[@Jet-Jots]

I'm so glad Ellis is giving some love to the Zenbook Fold as I think these dual screen laptops is where the laptop industry should be heading. I NEED to hear your thoughts on the Zenbook Duo 2024, please. I'm eyeing it as my next laptop and I need more people to see how good it is (or tell me how bad it is.)

[@skygaz3r]

May I recommend.. comparison review by KZbinr Just Josh? He’s really one of the most thorough KZbin reviewers around, under-hyped too

[@Jet-Jots]

@@skygaz3r i saw his comparison video with the yogabook and the x1 thing. Did he have a standalone vid for the Zenbook Duo 2024?

[@skygaz3r]

@@Jet-Jots unfortunately, no. That comparison video does have a section dedicated to the Zenbook Duo 2024 though!

[@adriendesentenac]

From what I can see, Southwest is not in fact using Windows 3.1 (that would be crazy), they're just not using Crowdstrike.

[@DreamMaker_23]

I was just having this conversation with my wife's godmother. She thought my shoes looked like crocs but they don't because theyre yezzy slides. I gave an example of people saying oh you have a new iPad when it's a Samsung tab, they think everything is an iPad when in fact it's just a tablet.

[@AndroidG101]

Like when I want to use my samsung pay and iam asked if iam using apple pay😐

[@hyperactiv8]

Crocs look better. Yeezys are ugly AF.

[@DreamMaker_23]

@@hyperactiv8 crocs are the ugliest shoes in history, just disgusting 🤮

[@MylesKillis]

The term is called coinage

[@momofmanda]

Calling all tablets iPads is same as calling all tissues Kleenex or calling all bandages Band-Aids.

[@toadlguy]

The one event that wasn't potentially, allegedly happening this week - it happened this week - was Meta releasing Llama 3.1 405B, the first frontier open source LLM model, and you didn't cover it because... you are advertising Meta AI? This seems to be a conflict you shouldn't be having.

[@kreigerbailey3550]

Portland, JAMAICA in the Building!!! 🇯🇲🇯🇲🇯🇲🇯🇲🇯🇲🇯🇲

[@dewyakana1543]

Thanks guys, this was fun.

[@RemoGtz]

Ellis and the Moustached Cat segment 👌🏽✨

[@garrettord]

Watching this while outside surveying on the Leica. hell yeah

[@MikeAbili]

28:35 I thought the OS that Southwest was running being old was debunked and it turned out that Southwest was just not using CrowdStrike but one of Crowdstrike’s competitors.

[@wafflenomad]

Y’all mentioned power users with the Fold vs. the Flip so in my brain wouldn’t Apple make a Fold with the Pro name and the Flip would be part of the standard iPhone line. Great episode and it always makes my Friday!

[@ItCouldBeAnything.]

40:01 the term you’re looking for is “Proprietary Eponym”! Cheers boys

[@Shadowstaarr]

For what it's worth, as I remember the CR-48 was their introduction to a Chromebook idea. It wasn't sold, but they were distributed by some sort of lottery (I remember hearing that if you had purchased like some stickers from a previous Google event you'd receive one). I had filled out a survey to try to get one, but they wouldn't tell you if were getting one or not so I'd keep track of whether a UPS truck was in my neighborhood in hopes i was getting one.

[@jeffeisley]

3:44 "or just give the ring instead" almost spit my Soylent on my keyboard 😂

[@Brunoxid0]

I love when Ellis gets to explain something super nerdy.

[@ARMYOFBRICKS]

Been waiting on this all week😭🙏🏽

[@cantstopthebeat17]

Love, love, love having a full breakout to Ellis nerding out over the Hall Effect explanation 🤓

[@darth_batman8190]

I work for a MSP and last Friday was a fun day.... So much fun. We manage a lot of doctors practices and this affected Dragon Medical One and so doctors were forced to actually type of their notes. LOL

[@Shocker99]

1:08:00 Tell me you're reading a script without telling me you're reading a script

[@malikfaisal416]

I prefer when people explain technical stuff to read from a script rather than just mumbling off the head.

[@Shocker99]

@@malikfaisal416 He got it wrong though - so what's the difference? I'd prefer a simple summary and for the person to point viewers to a source that actually knows what they're talking about.

[@brunolopesmello1986]

Best music intro and outro of youtube so far 🚀👏👏🤩

[@pace.hunter]

Today is Sys Admins day, we deserve appreciation for what happen in the past week!

[@davidthomas5562]

FYI (of WVFRM) , I believe it was 1985 that IBM announced their ability to create a cylindrical carbon nanotube that they anticipated at the time for creating flexible displays. I think you would be greatly interested in seeing what has been conceived since from that development.

[@yoduh4077]

Seeing Andrew stumble into a point during trivia warms my heart. Lmao

[@harales7849]

39:45 You mean a Flip-Phone? Haha. Great Episode

[@clairesilvermoon]

Adam's got new tats? They're so vibrant!

[@argentlupin]

The CrowdStrike Microsoft crash I only found out about when someone I called told me their computers were down because of this. I myself was dealing with my xfinity wifi hotspot that was not in my trailer but some where close enough for a medium signal disappeared for long stretches then could not connect because it could not assign an IP address. Concerning last weeks trivia that David correctly won as I have previously talked about the Chromebook Pixel is tied to the Nexus so I believe that invalidates it as not directly made by Google, Maybe the Apple fold/flip phone will become like the scifi Gene Roddenberry's Earth Final Conflict communication tablet cameras.

[@mitchbeachfun]

Love you guys!

[@BoluAjayifilms]

Question: Have you guys thought about doing these podcasts live?

[@esaedvik]

Runners have a tool too that gives them an edge that got banned; Nike had the carbon fiber plates in their shoes that increased the bounce force, making runners faster. Nike got banned for it and I think Adidas did the same later and got banned. Nike Vaporfly 4% was the base shoe, the banned was Alphafly with two plates, I think.

[@aravkilak]

Hey, Waveform, i just love your production standards, they're literally the best, I just wanted to point out that the thumbnail is a little incorrect, the Crowdstrike outage shows a blue "Recovery" screen and not the sad face BSOD that the thumbnail has.

[@caleblatreille8224]

If Andrew ends up doing a Studio video about this, PLEASE use extreme closeup cutaway graphics to show what you're talking about. I get the basics of the mechanics but visualization would really help here.

[@EndoAshura]

Andrew casually pulling and pushing the laptop lid back and forth almost gave me a panic attack 30:42

[@Doomerscroller25]

I finished being on call Thursday morning, the day before this happened. Perfect timing

[@WhiteEvo6]

The falcon driver is certified, updates aren’t as the process to get verification is lengthy, this gets around it. It’s crazy, as raised what’s the point of certification if you can bypass it. Also the driver has an option enabled to prevent essentially “last known good” turned off. This would have prevented the boot loop by booting with the driver disabled as it caused an issue Worked on fixing this for 13 hours at my workplace. Entire engineer and desktop support worked the entire weekend calling every single user and guiding them how to fix their laptops, get their Bitlocker encryption recovery key after deleting the Crowdstrike files. Thanks Crowdstrike for the $25 AUD Uber eats voucher for 13 hours working for free that really made up for it.

[@DarsenSantana]

Marquez got so serious at the end during trivia

[@jsax75]

CR48 was absolutely a Dev kit that was never available to the public. Not a consumer product. Thank you.

[@JlandelMoncada1]

A system restore also did work. You just have to pick a restore point before the Crowdstrike Update. Then when the pc eventually boots back up, crowdstrike automatically updates to the proper working update.

[@Pedropulgar]

I love how Andrews’s default reference to an FPS game is CSS ❤

[@unkowndata2338]

At my last job I used crowdstrike, building apps against their API, and setting up falcon sensors to forward data. It's a really cool security tool. Like one of the fun things it does is forwarding all of your process info to a hub with the sensors like Splunk/Grafana/whatever.

[@lxliner]

45:31 David's "aaaaaaaaaaa I don't know" to the soundboard pack!

[@ulyssemanceron]

The main reason why you push things in wave is because your test environment can never fully reproduce a prod environment. For instance, the video mentions apple rolling out software in wave, its because everyone have different apps installed, settings setup, etc. A slow rollout may catch bugs early in the wild, which can then be used to stop the rollout, issue a bug fix, and release that bug fix with the rollout. This isn't really considering "testing in prod" and is mostly used by large company to prevent large scale outage.

[@jakobsmith5656]

The craziest part of the Hall effect sensors is that this tech has been around for quite awhile. I know the Sega Dreamcast used these sensors in their controllers back in 1998 when it was released.

[@ChaseCares]

Dave's Garage was sued by Washingtons AG for alleged violations of the Consumer Protection Act. Reading through the report it seems like to me he made software that was extremely difficult to remove and when you purchased the software it added hi extra things to your order without really notifying you. honestly it's stuff that most companies do nowadays, he was just ahead of the game.

[@DougWolfe]

So happy. I knew the answer to the DC01 question. Had to research vacuums so vividly recalled that product moniker. First trivia question I’ve ever gotten right. Who says my sad little life isn’t working out. And it’s Friday.

[@sh8736]

Ditto I knew it too and often don’t get the trivia questions even when the answer is given. I reviewed a later model of the Dyson on Amazon Uk in the days when folks could like or dislike the reviews 🙂

[@MohamedHassan-r9i]

I can confirm that Samsung is a partner of the 2024 Olympics

[@fassoulia]

Love the idea for an hour long question! 🤣

[@skygaz3r]

Wait, why is no one commenting on the super cool tattoo on Adam’s arm?! Way to go, Adam!

[@anyotherdude]

Waymo gives rides from/to the airport in Phoenix

[@RyanMacabre]

Appreciate the EEAAO call out

[@matthewbruggeman7835]

An sat test for the final point would be so funny

[@omarrodriguez1929]

David satong how "we all remember Y2K! And if not you can look it up" makes me feel young as I wasent even close to being born yet. Looked it up and the year 2000 feels way before my time.

[@mon699]

Why did this feel so short

[@6donta]

Marques would make a perfect politician

[@soberanisfam1323]

That’s not a compliment 😂

[@EthanReynolds3090]

@@soberanisfam1323sad that it’s not :/

[@JustXavier]

Why?

[@NichoNick]

@@JustXavier People love hating on him

[@Che1seabluesdrogba11]

Crowdstrike's Twitter is disgusting, acting like they did anything for solutions, when all the work was done by boots on the ground manually deleting the update

[@robd.mp4133]

Dave's Garage mentioned! Never thought my two main KZbin channel subs would overlap

[@mr.joshmateo]

41:06 yes. There’s so many of us that really only upgrade when it’s a form factor change. I had the original iPhone then progressed to 3g, 4, 6, 10, 13. I told my husband I will upgrade when the design changes and Dynamic Island or thinness doesn’t count as a “change”.

[@ThePBJain]

I waited a whole week for this

[@JBrezicPhotography]

Gojira at the opening ceremony was 👌

[@Seahawksnumber1]

tremendous show again !!!!

[@whatjonsaid]

When was the last time David tried self driving in a Tesla? It’s worlds better than even a few months ago. That is the competition for Waymo

[@itsmikedrop]

I don’t think it’s going to happen any time soon, for some reason they have a big blind spot around Tesla. Whenever they talk about EVs they mention Tesla prices that are two years old, they don’t mention that you can get a tax credit on top of the cheaper cars comparing to BMW and Hyundai. The last self driving marques used and showcased was a version 11 I believe. I was expecting them being young to realize that Waymo and Cruise and even Zoox have major problems that will never get resolved and they won’t be able to compete with Uber. All of them require HD mapping of the areas they function in, so the roll out will be extremely slow! Each car because of the sensors they used costs $150k-250k, so from an operations perspective will never pay back when compared to a Uber driver who drives a $20k car. So yeah, the only self driving taxi will be Tesla, even though they will be last to the market they will beat everyone! Point in case the recent Cybertruck release. They already produce more trucks than Rivian and Lightning!

[@JJRicks]

No it is absolutely not. Waymo is 10 years ahead

[@IsaacKittleson]

Please more of Elise cut scene with info (or others). i love it so much nerd out.

[@keshavkunal2195]

Hello, First of all, I really like the podcast even though I don't watch all of them. Also a quick question - what headphones does MKBHD use on this podcast? And is it better than Airpods Max? Apologies if this is a basic question or has been answered before.

[@Muench13]

Professor Ellis is my favourite part of any WVFRM / MKBHD video 🎓

[@Junker_God]

1:08:20 Not me, just listening to you guys chat while I do my work

[@ninjanerdstudent6937]

Ziploc is my most used genericide item.

[@lackoliver55]

I'd throw Tupperware as a heavyweight contender in the kitchen realm of genericiders.

[@whophd]

I feel for Marques on that LG/Samsung question. I could picture the 2013 event clearly in my head. And I knew it was Samsung because they went on to release Flip and Fold. But my brain went “LG Display” because I’m an idiot.

[@briancorrell]

Ellis in that NASCAR shirt goes HARD

[@andrewnitin]

The Flip form factor makes the most sense to take over iPhone mini sales. I've seen several videos of iPhone mini users trying out the Samsung Flip and finding it a viable switch if not for the apps not being as well optimized as on iOS. The compact size and portability of the Flip appeal to mini users. If Apple innovates by making the Flip more durable and insanely light, it could attract even more mini users.

[@SnappyJCM800]

Dude!!!! Once I figured out the wave dashing in smash my whole game changed. Still wasn’t as good as a lot people I played against but it changed the game