wow haha that video was really cool, I heard these courses at lowlevel.academy are also really cool
@LowLevelTV4 ай бұрын
haha yeah
@jamescollier34 ай бұрын
Thank you! Best on the internet
@Mr.Pandey284 ай бұрын
@@LowLevelTV please make a video on the details and what actually caused the issue !!
@ziggy76764 ай бұрын
What if I'm already a low level god?
@Kane01234 ай бұрын
Do you know if customers have the ability to not receive instant updates? Or is it something crowdstrike requires at contract time
@metcaelfe4 ай бұрын
They certainly made a lot of machines unhackable
@Парасолька-х1и4 ай бұрын
now noone can steal their data
@wervicecoding4 ай бұрын
Not until somebody plugs in Serial
@gerdd66924 ай бұрын
That leaves us with the philosophical question: Is a hacked machine unhackable? Or can a dead animal be killed? (Disregarding for the purposes of this discussion that the "hack" was presumably unintentional - it was effective, nonetheless ...
@MathewBoorman4 ай бұрын
A lot like speed limits, No IT is Safe IT.
@corteztt5184 ай бұрын
@@gerdd6692 you got me at ‘can a dead animal be killed’
@XueYlva4 ай бұрын
Crowd-Strike: Global Offensive
@comfortcove4 ай бұрын
Perfection
@UnrealOG1374 ай бұрын
Terrific pun
@birigu4 ай бұрын
hahahah lmao
@Lucky9_94 ай бұрын
BAHAHAHA 😂😂
@kakorotskywalker4 ай бұрын
LMFAOOOOO
@mastaphaaz34244 ай бұрын
So the learned lesson is never push code in production on friday
@Fraket4 ай бұрын
that's been an industry-wide standard for decades
@ovalwingnut4 ай бұрын
Yes.. you need to be "young & bold" to do some things. Like climbing HUGE radio towers. As you get older, you have a tendency not to want to rock the boat. Which has it's own drawbacks So yeah, I've morphed into a chicken 🐔
@neruneri4 ай бұрын
The real lesson tbh is for managers and executives to stop allowing the overworked IT guy to push code to prod on a friday.
@Mordecrox4 ай бұрын
Some gurus actually go to great length to "demistify this myth" and now they must be in shambles that we have ultimate proof to never do that
@FullDupl3x4 ай бұрын
read only Friday!
@Patterner4 ай бұрын
a newsletter email had the following greeting: "Good morning and happy Friday to everyone who doesn’t work in IT."
@DioTheGreatOne4 ай бұрын
Look on the bright side, demand for IT guys has skyrocketed, and what does a sudden high demand spike mean? That's right, IT guys can charge whatever they want now because every company is completely desperate for them.
@SolidIncMedia4 ай бұрын
To be fair, that's how people treat IT workers, even when things are working perfectly.
@cix94204 ай бұрын
@@SolidIncMedia sir you're on call and im working late, bring me some mcdonalds would ya?
@David-gu8hv4 ай бұрын
@@SolidIncMedia Lol
@tbg074 ай бұрын
@@Patterner Which newsletter? Wanna see this.
@georgeprout424 ай бұрын
Crowdstrike, according to their name, worked perfectly.
@justinlinotte29814 ай бұрын
why they choose a name like this?
@emperorarasaka4 ай бұрын
@@justinlinotte2981 all part of the CIA backdoor testing
@context_eidolon_music4 ай бұрын
@@justinlinotte2981 To take down the Internet for real soon, dummy.
@KangJangkrik4 ай бұрын
@@justinlinotte2981 for fun? Why not ¯\_(ツ)_/¯
@murtajiz5454 ай бұрын
@@justinlinotte2981 No idea but it’s painfully poetic
@JonitoFischer4 ай бұрын
”The only safe computer is the one that does not boot" Crowdstrike CEO.
@Shocker994 ай бұрын
Technically true
@mar259474 ай бұрын
😂😂😂
@babybirdhome4 ай бұрын
To be fair, this violates one of the three tenets of cybersecurity, and it's one that people outside (and sometimes even inside) of the field forget the most. Confidentiality, Integrity, and *_AVAILABILITY._* If you're missing any of these three, you haven't got security.
@aisle_of_view4 ай бұрын
All the times I've had to explain to management why we should wait a few days before implementing an update, only to be met with blank stares. I'm loving every second of this.
@InvisibleHotdog4 ай бұрын
@@aisle_of_view they probably conveniently forgot
@devrim-oguz4 ай бұрын
The only thing is this happened via unattended upgrades
@CoderDBF4 ай бұрын
I agree and disagree. For example a server should probably try to be up to date with security. It can happen that a security issue has been released then there will be an entire army of bots sniffing every server that hasn’t been patched yet in an attempt to hack them. So you want to be fast in plugging holes because those bots will be very fast in finding your unpatched servers. Any other update like windows workstations can probably wait a few days without issues.
@benjaminblack914 ай бұрын
@@CoderDBF This particular case would be considered a security update, as it is an update to endpoint security software.
@MrCyanist4 ай бұрын
@@CoderDBF Hard disagree even patches targeting critical security flaws have enough time to at least test in non critical pilot servers or clients. If a security issues found its already known for a while by bad actors while issue becoming well known will increase attackers testing for few hours will hardly change anything compared to auto pushing to every server/client.
@justinlinotte29814 ай бұрын
its cool to see pure technical explanation of how it happens, its far from common medias that only use shocking words to get as much audience as possible while they don't know a thing about what happened
@TheStoneMountain14 ай бұрын
I totally agree! I for one barely read the "regular" news because of how it almost always feels like they have one sentence of information to deliver and extrapolate it to a whole article with a bunch of word poop and no real info. So I prefer this type of delivery every time! Informative, deep dive into the interesting bit, short and quick to the point!
@martin.19764 ай бұрын
@@justinlinotte2981 Definitely! I was quite confused about how this could have happened in the first place - and none of those other channel had covered that - but seeing that the actually delivered driver was all nulls explained it perfectly well. And this is likely why this passed all testing and everything.
@robertjenkins61324 ай бұрын
@@TheStoneMountain1 Yes, and it's not just tech. Today I was reading in the New York Times about how there were violent protests in Bangladesh over a "quota system" for government jobs, but they declined to explain what this "quota system" was. I was curious to know what would provoke such protests. I found better information on Wikipedia, in articles such as "Quota system of Bangladesh Civil Service" and "2018 Bangladesh quota reform movement". MSM reporting is so vague and dumbed-down. (They also do "fact checks" on statements, e.g., by Trump or a random conspiracy theorist on social media, that anyone with a brain would know are _obviously_ false.) It insults my intelligence.
@TRFAD4 ай бұрын
@@TheStoneMountain1 Yeah for 4 pages, and "you'll never guess the shocking reason" but never even mentions it once while you fight through ads and trying to click the next button.
@phoenixflower12254 ай бұрын
100% I love this explanation - typical media always exaggerates everything
@MenaceInc4 ай бұрын
Today was not the best day for me to wear my Crowdstrike t-shirt...
@brutely97184 ай бұрын
@@MenaceInc No don't think.. crowdstrik is saving world from hackers. Mistakes happens
@akpokemon4 ай бұрын
Or if you enjoy small talk, it's the best day--great conversation starter
@araz9114 ай бұрын
this is most likely due to open source software
@shauas42244 ай бұрын
@@araz911 what
@araz9114 ай бұрын
@@shauas4224 the shutdown due to open source libs most likely
@theAmazingJunkman4 ай бұрын
The fact that a segfault just caused Y2K to happen 24.5 years late is wildly amusing to me
@fnytnqsladcgqlefzcqxlzlcgj92204 ай бұрын
HAHAAHAHA didn't think about it like that lol
@ilonachan4 ай бұрын
oh damn ur right, this IS what they were afraid of!
@NickRoman4 ай бұрын
Except, we had to update BIOSs to prepare for Y2K. This was: reboot, delete a file, reboot. I'm thankful that the fix was so easy (albeit time consuming).
@JustSomeDinosaurPerson4 ай бұрын
@@NickRoman Unfortunately this did not work for every affected system. Throughout the organization I work for we had to use restore points because either the files wouldn't delete themselves or deleting said file wouldn't resolve the issue. What an absolute fucking nightmare. Edit: To clarify, we still got all of it fixed. It just took a hell of a lot longer than wanted and many of us stayed overnight. Corporate straight up expensed all of our food orders, no questions asked.
@MrJamesVanEngen4 ай бұрын
We prepare for a primary election while identity theft is rampant. Magnificent! 🤦 ... The oligarchy of American credit scoring companies might as well be assigning random credit scores to each citizen within legal ramifications again. #VoteMillennial in 2024! 🪙💵💳🤖🇺🇸
@lucaslannes40044 ай бұрын
I work on an airline, you dont imagine the mess. Oh Jesus, today was a nightmare. Hope tomorow get better.
@MarianoLu4 ай бұрын
I feel for you buddy
@nicejungle4 ай бұрын
it will happen again as long as airline companies use windows
@malavoy14 ай бұрын
@@nicejungle And if they'd used Linux, they would have went down in April with kernel panics. It's not the OS that's the problem, it's Crowdstrike.
@nicejungle4 ай бұрын
@@malavoy1 And if they'd used Linux, one reboot and you switch back to the previous kernel Down time : one reboot Compare to windows : you're screwed
@malavoy14 ай бұрын
@@nicejungle But Linux users are tech savvy. Most users of Windows are not, so MS hides safe mode behind multiple reboots to prevent them from ruining their system (and they would blame MS if they did ruin their system). Once in safe mode you can roll back system changes.
@MSThalamus-gj9oi4 ай бұрын
This will absolutely *not* be the last time something like this happens. When I first started in the industry, everything was packaged on disk/disc. Fixing a bug after shipping was EXPENSIVE, so we got the product to a 99% stable place and then kept trying to squeeze that last 1% of bugs out. Now? People just throw garbage over the fence, figuring they can just ship a patch later. Kernel mode software just *cannot* be developed that way. But... cyber security companies have time pressure that other kernel mode developers don't necessarily face. To be useful, such an app must be updated and deployed amazingly quickly, especially for kernel space, but that agility comes at the cost of stability. When the cure is worse than the disease, though... there's a problem.
@kugelblitz15574 ай бұрын
At the very least (assuming it's not an immediate security risk) then updates should be delayed a couple hours by region or something, so if this happens then it's a smaller section of customers that get screwed over and they have time to cancel it and get it fixed for the rest of the world.
@Bubblessss4204 ай бұрын
@@MSThalamus-gj9oi exactly! thats why i bought some CS stocks after the fall down. CS is a great vendor but this kind of things unfortunatelly could happen.
@framegrace14 ай бұрын
@@kugelblitz1557 Just apply CI/CD techniques. The first rule is "Only package once, at the beginning of the release, same package tested is the same deployed".
@workmad34 ай бұрын
While I agree that this sort of software can't be developed in the same way as a lot of stuff, there's also not really any evidence here that it was. The fact the update file was completely zeroed out points to a failure way past a dev shipping a bad code update... I can't see any way this happens without it being a build or deploy failure.
@MSThalamus-gj9oi4 ай бұрын
@@Bubblessss420 I was thinking of doing the same. The stock price dropped 20%, but you know it'll bounce back. It's a bargain right now. (No, I'm not a shilling bot. :D)
@yassine-sa4 ай бұрын
They literally waited for most KZbinrs to go on vacation before rolling out the update lol
@prezadent14 ай бұрын
'Literally' as opposed to what? Figuratively?
@katnoto89934 ай бұрын
@@prezadent1 using "literally" in this way is a form of hyperbole. English is cool like that 😎
@Maxawa08514 ай бұрын
@prezadent1 if you google the definition of 'literally' the second definition of literally is literally 'not literally'
@Reydriel4 ай бұрын
@@Maxawa0851Yep, a ton of English words have contradictory meanings like this unfortunately lmao
@NickRoman4 ай бұрын
They probably push out updates every few days.
@joshuac52294 ай бұрын
This is just viral marketing for Captain Crunch's new cereal, "OOPS! ALL NULL!"
@lashlarue79244 ай бұрын
😂😂😂 thank you, please take my like, I cannot even 😂😂😂💀⚰️🪦
@_JohnHammond4 ай бұрын
HAPPY VACATION ED WHAT A DAY (/WEEKEND/WEEK/MONTH) 💥
@brandonw16044 ай бұрын
Can we rename it to Blue Falcon?
@Kane01234 ай бұрын
Bro was definitely having beers by the pool before this. Spring break feels for sure
@darkshoxx4 ай бұрын
@_JohnHammond you don't get any rest at the moment do you 😆
@faust90914 ай бұрын
Why are you not the top comment wtf? Love you both ❤
@lumikarhu4 ай бұрын
you're also gonna farm videos out of that as usually, aren't you don't forget to mention that while CS released a faulty package tons of morons allowed it to update itself (or even worse - pushed updates themselves) on a fucking Friday. they deserve that, it's basic sysadmin knowledge
@BogdanTimofte4 ай бұрын
25 years ago, operating systems started signing drivers. 15 years ago, the same thing happened with the bootloader. A few years ago I heard that all PCs had to be replaced so that Windows could guarantee security. Now I understand that it gives total control over the computer during the initialization phase to a program without the slightest verification, just because it is in the right place in the filesystem and the name seems familiar? Live and learn...
@BrayanRuiz-m3w4 ай бұрын
this was so bad that bro had to make a YT video while been on vacation. what a legend
@MattGreer4 ай бұрын
But why did they roll out this update to every computer in the world all at once? Why didn't they run a canary? Why didn't they do the rollout in phases? It's low level code running in the kernel, and it is deployed in machines all around the world by thousands of businesses, why on earth wouldn't they be more cautious with the rollout? This is quite possibly the most reckless deployment in the entire history of software.
@Pipe04814 ай бұрын
Because AFAIK they didn't update the software itself, but rather they just pushed a new virus signature database file. The real issue is that the Falcon program can't handle invalid files
@MattGreer4 ай бұрын
@@Pipe0481 That's still doable with a slow roll out, canary, dog fooding, etc. Anything at this scale should be done with extreme care. Heck, even if not at this scale, there's no reason to be so reckless.
@michaelharrison10934 ай бұрын
@@MattGreer simple answer - they are morons
@auzziegamer46614 ай бұрын
@@Pipe0481 more technically windows can't handle invalid files or any program for that matter
@ShadoFXPerino4 ай бұрын
@@MattGreer If you canary then all the non-canaries are vulnerable to zero day from the new virus.
@lols114 ай бұрын
"//Just a small fix, no need to test it"
@MarkusHobelsberger4 ай бұрын
Famous last words.
@danielburger25504 ай бұрын
LGTM
@victotronics4 ай бұрын
"The entire internet" for as far as it runs Windows. If it had taken down Linux the actual internet would have gone down.
@Sandromatic4 ай бұрын
Apparently in April, crowdstrike for debian actually went down in a similar manner, (kernel panics.) thankfully I guess no-one actually uses crowdstrike for Linux so no-one actually cared.
@lashlarue79244 ай бұрын
Facts.
@TimothyWhiteheadzm4 ай бұрын
Yeah, my internet was just fine, so not sure what he meant. Airports are not 'the internet'.
@NickRoman4 ай бұрын
@@TimothyWhiteheadzm , well, I'm thankful that KZbin, Netflix, HBO... were all fine or fixed quickly.
@entcraft444 ай бұрын
Worldwide outages + clickbait = "The internet is going down". They claimed the same with the recent massive Facebook outage.
@cslearn30444 ай бұрын
Crowdstrike, tests code only once, at production
@gerdd66924 ай бұрын
Not only once - they do millions of parallel tests on a vast array of systems - without making the code platform independent they could only better this by cranking up a few zillion virtual machines or container "farms" ...
@Brahvim4 ай бұрын
@@gerdd6692 Yeah, this one sounds like a problem with deployment instead, doesn't it?
@martin.19764 ай бұрын
I don't think we can dismiss the problem with such a "simple" explanation. Most certainly, they did test their code very properly and extensively. But they missed one of the most important factors: that things could possibly go very wrong in transmission.
@matt72534 ай бұрын
@Brahvim that's what I was thinking/the build. Development can have different config for build/deploy and you may not see until you pushed to the environment. Even with a UAT sometimes the config can be slightly different than prod even though it should be as close if non near identical to prod.
@cslearn30444 ай бұрын
@@gerdd6692 yeah im just joking
@seeibe4 ай бұрын
My vscode tunnel into my dev machine randomly stopped working yesterday and made me realize how much it sucks to be dependent on someone else for your own setup.
@najtofnin20094 ай бұрын
Dude. You're using vscode. Welcome to Microsoft dependence inc.
@not_kode_kun4 ай бұрын
linux, emacs, vim, helix are waiting for you with wide arms
@deeiks124 ай бұрын
I'm sure lots of people are looking for vulnerabilities in crowdstrike now. I don't know how public this was before that SO many companies are using their product....
@cbaesemanai4 ай бұрын
they just showed us the vulnerability, creating a sys file full of nulls.
@MarianoLu4 ай бұрын
@@deeiks12 it is very public that literally everyone (in corporate IT) uses their products, the thing is that is transparent for most non IT people and they do not have a consumer version. And unfortunately they are ( or were) the best in the business.
@renato360a4 ай бұрын
@@cbaesemanai you had to actually be them to do that in the first place, so.. that's not it. Unless maybe if you hack the provider of their update pipeline, which might actually be what happened. But I guess if you did that you could break so much more.
@cbaesemanai4 ай бұрын
@@renato360a I mean using it as a local exploit.
@Shocker994 ай бұрын
Crowdstrike is well known. They've had hordes of people looking for vulnerabilities in their software for years.
@internetuser89224 ай бұрын
I watched a ton of videos on WTF even happened here. This was the only one that actually explained what went wrong in any kind of detail, and you're on vacation. Absolutely amazing channel here.
@MarkusHobelsberger4 ай бұрын
This. It's a super-simple error, yet no mainstream media explains it in a comprehendable way.
@edwardallenthree4 ай бұрын
Excellent video! My wife has high confidence in her coworkers following the instructions and fixing their desktops and laptops. I think the person who replied all to the instructions asking for his encryption key is proof that this is going to be one long weekend for people in IT. Never been happier to be retired.
@ScottForrest4204 ай бұрын
Me too. Retired two months ago from a law firm who uses Crowd Strike through a consultant/VAR. I was imagining the entire firm losing their mind today due to this. Funny/not funny, but so happy it wasn't me having to deal with it.
@chrisalmighty4 ай бұрын
@@edwardallenthree you're spot-on with that 😅
@j340_official4 ай бұрын
So community notes on Twitter is saying the viral tweet that claimed it was a null pointer dereference is BS. And crowdstrike put out a blog statement that “This is not related to null bytes contained within Channel File 291 or any other Channel File.”
@VioletEmerald4 ай бұрын
Hmm interesting
@yarpen264 ай бұрын
I just got done writing a comment under Fireship's Code Review saying how Ed is bound to release a video on this as well soon enough, I reload my YT start page and I can see it up there, from 11 minutes ago.
@thunderb00m4 ай бұрын
How the hell is a multi billion dollar company not have basic error handling like null check!? Like seriously do they not take functional safety seriously!?
@Shocker994 ай бұрын
How do they not a pre-update push set up that acts as if the machines are actual client computers to test all updates before being pushed to real clients?
@TheFunkyTechGuy4 ай бұрын
@@Shocker99 My thinking exactly, this is basic stuff.
@linkernick53794 ай бұрын
Almost nobody checks the result of many functions, e.g. malloc or printf. Defensive programming techniques are clunky, cumbersome but still dont save you from errors (ref: "The Art of Software Testing" book). The only way to get rid of whole classes of errors is with good type system, in other words to detect by compiler.
@AndrewBrownK4 ай бұрын
cue elitist C++ dev entering the scene and saying "no no this isn't a technology problem, this was a skill issue haha, I'd NEVER do this, I'm too skilled"
@williamforsyth66674 ай бұрын
"error handling like null check!" It should have been caught much earlier. Seem they have no integrity check of their binaries during the build-test-distribution process.
@0xkleo4 ай бұрын
The crowd was striked by a blue screen of death
@samiraperi4674 ай бұрын
Struck.
@vizionthing4 ай бұрын
@@samiraperi467 Stricken
@emusunlimited4 ай бұрын
@@samiraperi467 Moron, your bus is leaving… 🚌 It’s CrowdStrike not CrowdStruck
@emusunlimited4 ай бұрын
@@samiraperi467 Never heard of CrowdStruck… is that some tech company or something?
@TabTray4 ай бұрын
@@vizionthing Stricketh
@theeternalsw0rd4 ай бұрын
By the way, this is the second BSOD software update push George Kurtz, CEO of CrowdStrike has presided over. First was April 21, 2010 as CTO of McAfee when an update inadvertently deleted svchost.exe from Windows XP machines. That would have been more massive but for the lack of always auto-update devices nowadays.
@ernstraedecker61744 ай бұрын
"Who needs svchost.exe anyways? I'm an ordinary Windows user, I don't use all those nerdy tools. Just trust the authorities. Safe and effective!" - All my friends.
@Veptis4 ай бұрын
How did this roll out this widely? Is there no canary? Is there no QA? How is business and airports the first wave of roll out... How do you put all trust in a single third party?
@samniechcial84934 ай бұрын
@@Veptis I’m wondering the exact same thing. From the perspective of a DevOps/Infra engineer - What kind of update deployment strategy is “just hit all billion machines at once”? No canary? No region by region? More questions to be asked here than just what went wrong in the actual code… If this is how they deploy code with a bug, imagine if they deployed code with a serious vulnerability? We need to hear about Q/A and deployment strategy at CrowdStrike!
@outtakontroll33344 ай бұрын
good questions, and someone is damnwell going to have to answer them
@vappyreon11764 ай бұрын
@@samniechcial8493just stop using windows they're not worth the security issues.
@tigerchills20794 ай бұрын
"How is business and airports the first wave of roll out..." someone else in the comments answered that already: they are the only ones to roll out to. it's big business software. there is no consumer grade product
@ShadoFXPerino4 ай бұрын
There's no canary because what is pushed are virus signatures. If you canary then all the non-canaries will be vulnerable to the zero-day virus. Getting instant updates is the entire point of the product. QA probably happened, but after the QA they deployed the tested file to some file server and the file got corrupted in transmit into all-zeros, which causes a crash loop.
@69clouds4 ай бұрын
The fact that it was an Antivirus that performed the single most successful malware attacks ever is just pure poetry. Another win for the "remind me later" to every update gang. My dad: Come on, it's just an internship, what's the worst that could happen? Me: "You can't hack a system if the system doesn't work! " - Cybersecurity The alpha move of doing something that would make your stock value crash, but simultaneously freezing the stock market so that it can't.
@Dead_Goat4 ай бұрын
Windows update doesnt matter. THis is not a win for remind me later as you cannot reind me later with this terrible crowdstrike rootkit.
@69clouds4 ай бұрын
I am aware of that, it just sounded like a good joke so I added it up.
@cix94204 ай бұрын
if you would like to make a bet, many futures brokers are still working, so when the market opens you can go short or long with some margin if you think this will have an effect on the worldwide markets (it will)
@ajinkyamogre85154 ай бұрын
So you just compiled the top comments across youtube over this topic and copy pasted them here for likes. Cool.
@cix94204 ай бұрын
@@ajinkyamogre8515 with how internet speech is i didnt even realize it, i just assumed all of this stuff was one full sarcastic comment
@c_ornato4 ай бұрын
It baffles me that people would modify any piece of code that sensible without testing on a friday, hell even I check 3 times that my KBPs are correct before restarting
@lion212974 ай бұрын
I'm sure they did test it.. the issue must have happened when actually pushing the update live. Doesn't excuse a friday update though. That's just asking for trouble
@asynchronerflugelflitzerim84814 ай бұрын
@@c_ornato there was still a network connection, right?
@c_ornato4 ай бұрын
@@lion21297 Doesn't excuse pushing to every user at the same time either, you'd think the devs for a highly-used expensive piece of software would be more rigorous but it seems the dev instinct to push stuff fast does not discriminate.
@ninele74 ай бұрын
@@lion21297 You need to understand that it is a security software. Maybe they've implemented protection from new attack vector. And hackers don't rest on weekends. As we see, problem happened at some unpredictable late stage (file became all zeroes, it's not compiler output). Even if they released it on any other weekday effect would be the same.
@mallninja98054 ай бұрын
@@lion21297 Sounds like there's a gap in their deployment testing...
@JohnFink-p5l4 ай бұрын
As an IT guy, I have been working non-stop all day today. It's utter insanity here at this company. Those dumb bastards
@haroldcruz85504 ай бұрын
Hey at least now they realize how important you are.
@BoraHorzaGobuchul4 ай бұрын
Oh my god, they killed kernel! The bastards!
@pieterbezuidenhout37574 ай бұрын
At least we got our Servers up and running in 2 hours this morning, overtime cash, I feel you, Brother.
@MarkusHobelsberger4 ай бұрын
@@haroldcruz8550 They are rather going to blame him for the problems he's trying to fix. Unfortunate truth of working in IT.
@not_kode_kun4 ай бұрын
@@pieterbezuidenhout3757 yall have windows in your servers? what the fuck are yall smoking
@Somezable4 ай бұрын
I bet one of the reasons for this is the LEAN. A plaque of corporate goal of efficiency that ends up ruining workplaces, if allowed to continue too long. I can guarantee we will later hear, if the company isn't able to hide it, workers at crowdstrike were overworked, overstressed, always pushed into a rush and denied time to make critical quality assurance checks and tests, that would have caught this error.
@rikuleinonen4 ай бұрын
I feel like all the blame is going to be pushed onto the employee that coded in the mistake while the CEO/manager that made them do so via overworking them etc. will get away scott-free.
@observant69534 ай бұрын
Thank you so much for actual getting into technical details. All other articles just repeat "well, everything is down".
@DJJOOLZDE4 ай бұрын
Kinda neat that so much of the internet depends on a few people uploading critical files.
@annmaryjohn32584 ай бұрын
@@DJJOOLZDE The internet is fine, it's the computers using Microsoft that has crashed.
@MK73DS4 ай бұрын
Just use a closed source piece of software on our closed source OS for our critical application, everything will be fine.
@MathewBoorman4 ай бұрын
Boss, The closed source OS and hardware you have on your desk is not good enough to act as a dumb TV or kiosk by itself. First install the closed source drivers from a bunch of random hardware vendors. Then add some tooling to actually install and configure the host to do its thing. Add some more software to manage the truck loads of host security settings from all the stuff we don't need anyway, but can't remove. Add closed source kiosk software or maybe the POS application, which is just a wrapped browser. Buy some more security software since we can't trust any of the perviuos bits to work. Don't think about deploying a cheaper Open Source & Open Hardware solution, like a rasberry Pi.
@football422414 ай бұрын
Open/closed isnt the issue in this case. Corrupt auto software updates is, and they can (and do) happen to both.
@CharGorilla4 ай бұрын
@@football42241 Except you'd be ripped to shreds in an open source project if you committed code that runs kernel mode, downloads dynamic code off the internet and runs it in kernel mode, doesn't have any sort of integrity check on what it downloads off the internet, like a digital sgnature, or even a CRC32, AND the virtual machine / interpreter which runs the code that was downloaded off the internet isn't sand-boxed and lets the dynamic code use naked pointers. How many freaking basic mistakes did this "cybersecurity" company make here. Not to mention that all this time, their "cybersecurity" software has been one giant RCE waiting to happen if you manage to spoof the DNS of the update server, or MITM that HTTP(maybe S) request that we all know doesn't check for a specific root authority). I wonder how long the NSA have known about this one. I'd hope not as long as they kept Eternal Blue under wraps.
@not_kode_kun4 ай бұрын
@@football42241 nope, auto software updates are extremely rare on open source operating systems. since they're made by devs, for devs, and all of us devs hate that shit. This crowdstrike thing could've very well happened even if the whole world ran on linux. but at least, it'd only happen to people who ran the update command, and even then all they'd need to do to get their computer back is rollback to previous kernel and reboot. The same problem could've happened, but it would've been in a smaller scale and easier to fix.
@alexholker13094 ай бұрын
Thanks for the video. The file being corrupted does seem to explain how it could get past testing, if the failure to write only happened after they verified the file was safe.
@garanceadrosehn96914 ай бұрын
Also note this story: *"Major Microsoft 365 outage caused by Azure configuration change"* ... _"Microsoft says an Azure configuration change caused a major Microsoft 365 outage on Thursday, affecting customers across the Central US region. This massive outage started around 6:00 PM EST and prevented users from accessing various Microsoft 365 apps and services."_ This happened hours *before* the Crowdstrike issue surfaced, and I also wonder if there might have been some connection.
@jacominnaar4 ай бұрын
We had issues on Azure with HTTP2 traffic in South Africa. I'm wondering if they are related.
@hellowill4 ай бұрын
yeah I noticed the Azure outage too. Funny how nobody cares or even blames MS for the Crowdstrike/Windows interaction.
@gabriel55ita4 ай бұрын
@@hellowill why blaming ms for something another company did wrong, it's their responsibility to not ship something bad. Windows offers you to boot in safe mode to remove the driver that start prioritized
@workmad34 ай бұрын
The Azure issue caused some of their compute to not be able to access storage. Then hours later, crowdstrike pushes an update that's completely zeroed out. If there isn't some connection and cascading failure discovered in postmortem it's going to be one hell of a coincidence 😅 Still a pretty massive failure on crowdstrike's part to manage to drop the file onto so many machines without some verification raising an alert that this file is screwed
@gzoechi4 ай бұрын
@@workmad3Also updates are usually pushed to a small selection of devices and when nothing bad happens, the number is increased
@donleyp4 ай бұрын
I was supposed to fly to Japan this morning. Crowdstrike canceled my flight. I’m glad you’re already at your destination, Ed. Have fun!
@misogear4 ай бұрын
The most important piece of software don't have rollback mechanism when update is broken is mind blown to me. 😂😂
@Xehlwan4 ай бұрын
@@misogear Oh, there is - Crowdstrike's update was just designed in a way that doesn't make use of that Windows feature.
@not_kode_kun4 ай бұрын
@@Xehlwan except window's rollback is completely useless because you need to boot into the system, then jump through menus to finally rollback. the whole purpose of rollback is for when an update is BROKEN. if an update is broken and you can't rollback, then you dont have rollback. On linux, you can still rollback your kernel after a kernel panic (BSOD) with a single reboot
@kylekatzin15634 ай бұрын
So funny, my first thought was who tf pushes out an update on a Friday..
@MarkusHobelsberger4 ай бұрын
Everyone does (working in software development in a small company here). If something goes wrong you will have time to fix it over the weekend. Something so critical should have been tested better, though.
@brssnkl4 ай бұрын
Your look is giving IT person they had to bring back from middle of his holiday to fix this. It feels so authentic :D
@test403234 ай бұрын
i find it incredible crowdstrike didn't do a staged rollout considering the risks. this is an acute reminder that lab testing can't cover all scenarios and that a fallback plan is always necessary no matter how remote the probability of mishaps. defn: risks = probability(event) × cost of damage (event)
@Sandromatic4 ай бұрын
They actually did apparently. It's just, they didn't notice it until it'd rolled out to the majority of their userbase. Meaning they did a staged rollout but it was too fast/no-one was monitoring it to catch. I mean, that probably happens when you're doing it on a Friday.
@BoraHorzaGobuchul4 ай бұрын
Or testing
@jbird44784 ай бұрын
@@Sandromatic They apparently didn't notice at all. It was Google who pointed the finger, and even then it took them two hours to say "ehm.. yeah, it was us."
@BoraHorzaGobuchul4 ай бұрын
"looks like I picked the wrong week to stop sniffing glue"
@crispybatman4804 ай бұрын
Big hugs to all the people having to manually recover systems today.
@rcstl88154 ай бұрын
I wonder what the Linux download sites are seeing?
@NewKiwiJK4 ай бұрын
yay overtime
@jnawk834 ай бұрын
@@rcstl8815probably not much, corporations and enterprises don't tend to knee jerk quite that fast, if at all.
@ecchioni4 ай бұрын
Where the fuck was the test team? Oh wait... The modus operandi of a modern hackshop is fuck QA.
@joseoncrack4 ай бұрын
Yep. But remote updates are evil anyway. That's a nonsense from a sysadmin POV.
@ecchioni4 ай бұрын
@@joseoncrack They are not paying for QA, do you think they'll pay for a stage environment and a team to manage it where the update is tested before it rolls to the rest of the org?
@DoinThatRag4 ай бұрын
This is so sadly true. My CEO who likes to pretend he's a dev but isn't and has no background operates like this. I swear it is due to the development of CI/CD pipelines and how relatively "easy" it easy to write some APIs, just have the devs write tests for them, and throw out the latest update to your microservice. I mean all software functions like that doesn't it? The cloud systems you run for your little API services or pushing critical and sensitive updates to customer systems, what's the difference? So what if it is a kernel driver, just keep those updates rolling!
@haroldcruz85504 ай бұрын
QA is not very cost effective. Why test when you can do it live
@ecchioni4 ай бұрын
@@haroldcruz8550 Maybe to prevent a 20% stock drop and a big fuck you from the customers?
@glowingone17744 ай бұрын
cloudstroke
@joshuathomasbird4 ай бұрын
jfc lmao
@Fs3i4 ай бұрын
clownstrike
@redreaper50834 ай бұрын
Clownstroke
@haroldcruz85504 ай бұрын
Clownstruck
@bionic_batman4 ай бұрын
Cloudstrife
@Stratelier4 ай бұрын
This incident feels like something out of a Tom Scott spec video (re: "that time Google forgot to check passwords") ... or the definition of an "onosecond". 2:55 - Wow. Just wow. I've personally experienced a few cases of complete individual file loss leaving behind "all nulls" (presumably from a failed deferred write-to-disk). But those were just two or three personal userfiles -- I couldn't imagine this occurring with a critical driver or system file.
@benbohannon4 ай бұрын
So, for the few companies Microsoft trusts to operate at the sys/kernel level, all updates should be funneled through Microsoft test channels. They cannot have companies breaking their install base.
@TonyJewell04 ай бұрын
THIS. I was ranting to my wife yesterday evening about this. Bless her, she knows nothing about IT and made all the right noises.
@tallpaul94754 ай бұрын
Your explanation of how the driver integrates to the system to make it depend on it makes sense. Thanks for giving informative details at an understanding level.
@_hugoi4 ай бұрын
That was just like the *Y2K* bug... but this time nobody was expecting it LOL
@savagej4y2414 ай бұрын
And the Y2K bug probably would have been less disruptive back then compared to a Crowdstrike error nuking systems operations worldwide now, because 24 years ago, you didn't have mission critical systems that require "always online" connections. It was a transitionary phase and older fallbacks were at the ready. Now its more damaging because the more modern systems ARE the fallback.
@nimrodsmusic4 ай бұрын
This is the best and direct answer to the question. Every other big news outlet goes on and on and never actually tells us what went wrong. Well done
@andybreuhan4 ай бұрын
How was this .sys file signed? If it is all 0. how was Windows able to load this? Why are there no checks in place?
@Hexanitrobenzene4 ай бұрын
It seems that it is a submodule which is loaded by a CrowdStrike agent itself. That means they don't do basic checks...
@glitchy_weasel4 ай бұрын
Nobody can't hack into your network if your entire network is down - I say this is mission accomplished 😎
@j_stach4 ай бұрын
Today was the most secure Windows has ever been
@tongpoo89854 ай бұрын
This is what I call a "paradigm shift" in cyber-security.
@m4rvinmartian4 ай бұрын
*Make the SKINNIEST reference at bootdriver position, make sure it works... NEVER change it. Load everything else after the system is stable.*
@jeremybuckets4 ай бұрын
"we're kind of dependent on these companies, and when they get it wrong, the whole world collapses. kind of makes you think." such a cheerful delivery of that truly terrifying statement.
@emilmofardin2.04 ай бұрын
My dad is a senior developer and we watched this happen in real-time. We spent the day installing Linux instead.
@jorper25264 ай бұрын
That isn't the fix. CrowdStrike also has a linux agent. EDR's MUST work at the kernel level to do their job. It is just that in this case, they messed up the file for Windows, not for Linux or Mac..
@ItsCOMMANDer_4 ай бұрын
@@jorper2526 although it is easier to monitor such stuff from userlevel in linux, i heard
@rikuleinonen4 ай бұрын
@@jorper2526 is the Linux agent even Kernel-level? I recall hearing somewhere that it doesn't go nearly as deep as the Windows version.
@jorper25264 ай бұрын
@@rikuleinonen it depends. But by default, yes. They did crash numerous linux machines in April.
@rikuleinonen4 ай бұрын
@@jorper2526 thanks for the info.
@_GhostMiner4 ай бұрын
*Hackers: try, and fail to take various systems down* *Crowdstroke: Fine. I'll do it myself.*
@tejonBiker4 ай бұрын
Crowdstrike switched to MaS: Malware as Service. Pretty wild that a kernel-level software from a cybersecurity company deploy bad software
@Dead_Goat4 ай бұрын
not really that wild. This is exactly why i have been fighting against using this crap. It does not increase security in anyway.
@rythem22574 ай бұрын
@@Dead_Goat If it didnt increase security, people wouldnt use. Who are you fooling lol?
@PvtAnonymous4 ай бұрын
@@rythem2257do you understand the concept of snake oil? That's exactly it.
@jorper25264 ай бұрын
@@PvtAnonymous Explain then. I'm sure you have TONS of great information on this subject, and not just some "hurr durr linux" type of reply.
@PvtAnonymous4 ай бұрын
@@jorper2526 what is there to explain? Just with any AV you introduce a single point of failure with extensive privileges into your OS or even your kernel. As seen in the last few days, there seems to be a lack of testing on Crowdstrikes end which resulted in - again - a single point of failure. Threat actors could just as well find a gullible employee or even infiltrate the whole company and introduce malicious code that could take over all of the machines and do whatever they want with them, basically making Falcon into a rootkit. This false sense of security is, you guessed it, snake oil.
@XerrolAvengerII4 ай бұрын
I've been looking forward to you making a video about this since I saw the first Australian news reports this morning! Thanks for taking time out of your trip to film this!
@pixelshocker77754 ай бұрын
Appreciate getting news from a source that actually knows what they're talking about. Props to LLL for taking the time to make a video while on vacation, and hopefully nothing equally newsworthy happens so he can just relax...
@manualdidact4 ай бұрын
Fireship for the snarky summary, Theo for the details, LLL for the disassembly. I missed out on all of this today, but I feel caught up now.
@henson2k4 ай бұрын
Security nuts make things worse
@NarrowShouldersOpenMind4 ай бұрын
@@henson2k Nassim Taleb will have a good take on this.
@ziprock4 ай бұрын
cool, appreciate the technical take on the issue. This will be a perfect example of what not to do.
@BoosterShot10104 ай бұрын
An intern writing tests with ChatGPT!
@Salted_Potato4 ай бұрын
For CrowdStrike outage, I went to mainstream media for human-readable explanation, I go to Fireship for system level explanation and now blessed with low-level-learning for the autopsy :D
@xmlthegreat4 ай бұрын
I remember seeing an article about how CrowdStrike's CEO regretted not firing people earlier... In 2020. I guess he's reaping now eh?
@Brahvim4 ай бұрын
So... the CEO might regret _regretting_ the thought of _firing people_ who may now provide support to customers? ...Or is it that he would've found that firing these guys early - the ones who couldn't deploy this crashing update correctly, as beneficial?
@renato360a4 ай бұрын
yeah, I don't know how to read this. Should he have fired them earlier, or should he have kept them?
@mudi2000a4 ай бұрын
I guess he fired them later. I mean at the end of the day even someone who was fired could have planted this in a way that it looks just like a technical f-up.
@xmlthegreat4 ай бұрын
You people have misunderstood the context.. the CEO laid off a bunch of people around 2020 and boasted how he probably should have done this earlier. Like Microsoft laying off most of their Windows QA teams so that they could use customers as Beta testers; when you reduce your employees you reduce the amount of slack the remaining people have to pick up on problems and head them off early. So a piece of code ends up in prod without at least 1 person catching a fatal bug. And some poor schmuck or team of schmucks who had to meet some kind of performance metric deadline push code on Friday that should have been tested more than just a few times on CrowdStrike's internal systems.
@MaffeyZilog4 ай бұрын
That was the reason for kicking Kaspersky out! They wanted one for a while and they got it even though every other AV vendor has more privileged access to your computer than Kapserky did!
@norbert.kiszka4 ай бұрын
In Linux module file with all nulls will not crash the kernel because Linux kernel makes multiple checks on a module file. Also modules are single files (sometimes module can request another module but it will be loaded in the same way) instead of multiple files. Couple months ago I tried to read that module loading code, but its poorly documented to be easily readable (if somebody was working with this code for long time it will be much easier).
@vilian91854 ай бұрын
and tobe fair running crowdstrike on linux isn't needed, now with windows...
@norbert.kiszka4 ай бұрын
@@vilian9185 only updates, but that can be done automatically in most cases (sometimes it will reboot some services or You have to reboot system manually when kernel or libc has security patches).
@Turalcar4 ай бұрын
@@vilian9185 but possible. Crowdstrike pushed a buggy debian update few months ago to a similar result (but fewer affected users).
@Dead_Goat4 ай бұрын
@@vilian9185 you do understand that linux is very hackable and much more likely to actually need something like crowdstrike than a windows system right?
@vilian91854 ай бұрын
@@Dead_Goat no?, lmao wtf you're talking about
@martin.19764 ай бұрын
First of all, thank you so very much for covering this during your vacations! I didn't even know this company existed until this morning, and quickly realized I'd only possibly get an explanation as to what happened from somebody like you, who knows about this stuff. One important lesson we need to learn from this is to ensue to do transaction-based updates and to ensure integrity of each update with a cryptographic signature. They have likely performed intensive in-house testing of that update - but just didn’t account for the possibility of the update being corrupted / tampered with in transit. However, if also brings up the question of whether we can really exclude the possibility of this being a dedicated cyber-attack that quickly! Because, to my knowledge, we don't know yet how exactly it could happen that the version of that update that was installed on these billions of devices came up all zeroes. Surely, it shouldn't have happened - and likely wouldn't have if they used even the most basic CRC approach to verity integrity. In that regard, the blame is by that company alone! However, I wouldn't necessarily exclude the possibility that some bad actor knew about this vulnerability and exploited it.
@Iceman2594 ай бұрын
I, for one, am shocked that the company called "CrowdStrike" which thinks it's a great idea to advertise on race cars would ever do something so ill-advised.
@manoflead6434 ай бұрын
That's memorable! Good advertising, honestly.
@Maxler57954 ай бұрын
Fun fact: this happenee the DAY evolution championship happened, which means a lot of competitors just... Couldn't make it. Wasting thousands of dolars per competitor in the process
@eljuano284 ай бұрын
Blistex Inc is proud to announce they've teamed up with Linux to solve the blue screen of death. The "Tucks Medicated Pads" you already know and trust have been rebranded "Tux Crowdstrike Relief Pads" for Microsoft Windows users to relieve that burning sensation.
@MelJandric4 ай бұрын
Finally somebody explained it. Thank you. So far every other statement about this incident was just hot air.
@boy_deploy4 ай бұрын
"Crowd Strike" literally 😂
@johngrant58814 ай бұрын
@@boy_deploy alex jones predicted this
@eitantal7264 ай бұрын
maybe this will change the mentality of pushing pointless updates too often
@erickhar4 ай бұрын
I love your channel. so cool that you took a vacation from your vacation for this :)
@bjornjohnson97534 ай бұрын
This is the video I’ve been looking for! Thanks for the succinct explanation!
@mihainita53254 ай бұрын
What puzzles me is why deleting the file solves the problem. There should be no difference between the missing file and zeroed-out-file. An all-zero driver file it's missing all the standard exe structures. Has no header, no import, export, relocation tables. And no digital signature. So Windows should refuse to load such a "driver". Which would technically be the same as missing file. Who null pointer for zeroed-out-file but not for missing file? Can't load is can't load, no? I really suspect something more is going on here...
@MathewBoorman4 ай бұрын
I am wondering that, might have got signed as nulls. CICD failure
@mihainita53254 ай бұрын
@@MathewBoorman yes, but the signature should also be part of the file. So at least that would not be zeroes. Same for the exe header. If it is all zeroes it would not load. Unless the description as zeroed-out-file is not 100% accurate and the header is present, the signature is present and correct, etc. Which then raises the question what kind of processes do they have to create such a file, signing it, and push it out with no testing. This is not something that "it works on my machine" :-) Some big red flags about their processes...
@YouPlague4 ай бұрын
Because if the file is missing opening it fails with a proper error. Once it opens the contents is trusted to be executed on the cpu.
@tma20014 ай бұрын
the clownstrike background driver that listens for updates should of done at least a hash/crc etc check of the updated file before initiating a reboot. That's what I don't understand.
@juliavixen1764 ай бұрын
I use ZFS to store all my data, because I do *not* trust hardware. I always generate and verify MD5 hashes of my files and I have found so many _silent_ data corruption faults when a hard drive will randomly flip a single bit, or return 512 null bytes... with no hardware errors being detected/reported. SMART self-check says everything's ok; Linux device driver says everything is ok; etc. I mention this, because I have been given NTFS drives by other people, and upon checking the files, there will unexpectedly be 4096 null bytes where I was expecting data. (And I had a second copy of the file for comparison.) There were "no errors" reported at the hardware, filesystem, or OS levels. A file was silently replaced with a bunch of nulls... and I immediately wondered if Crowdstrike's build system is using NTFS.
@Vespyro4 ай бұрын
Within minutes of find out out about the situation I was already so excited for your breakdown, thank you!
@山田ちゃん4 ай бұрын
It's ridiculous that they can't even save a single boolean or something to disk and and clear it when the OS booted, this so next time your driver loads it can check for the boolean to be true and just not load (because you can safely assume the PC crashed), and while in the OS your non-driver part of the code could check for updates...
@az-kalaak62154 ай бұрын
or even better: do it linux style? and don't take down the entire system if there is a recoverable error at boot
@somenameidk52784 ай бұрын
@@az-kalaak6215I'm pretty sure a corrupted kernel module like this would crash Linux as well.
@DamonWakefield4 ай бұрын
Thanks for this outstanding code-level explanation!
@ok-alarm4 ай бұрын
"should have used rust.. " 😂😂
@mushroomcrepes4 ай бұрын
these rust ads are getting crazy
@lythox-bdd4 ай бұрын
@@mushroomcrepes damn best rust ad ever 🤣🤣
@hwstar94163 ай бұрын
wouldn't have helped in this case
@DrKaoliN4 ай бұрын
This has been the clearest explanation I've seen regarding what exactly caused the BSoD.
@delayed_control4 ай бұрын
tfw title starts with lowercase "lol"
@theskelet4r4 ай бұрын
Love the commitment, thank you for taking time out of your vacation to breakdown this incident for us. No go have some fun and Relax!
@typingcat4 ай бұрын
Wait, how could this happen. Don't they digitally-sign driver files? If so, an all-null file should not pass the signature verification, and how did Windows even load the driver? Windows rejects unsigned drivers, doesn't it? This makes no sense.
@iljaseklervl4 ай бұрын
According to the video, it wasn't Windows. It was CrowdStrike kernel mode driver loading a submodule without any NULL checks and error handling.
@typingcat4 ай бұрын
@@iljaseklervl That is dumb. How could a security application do not check the integrity of its module files? This is not just a simple distribution mistake, but a fundamental amateurish dumb coding.
@nigh7swimming4 ай бұрын
Agree, looks like the gist of the issue to me. Mistakes happen, but this is pure negligence in lack of any sanity check verification.
@john63724 ай бұрын
if there are no checks.. sounds like a nice backdoor waiting to be exploited.
@NYYstateofmind4 ай бұрын
@@john6372sounds like it just was
@austinreilly10674 ай бұрын
You never fail to deliver content on current issues and I was absolutely waiting for you to talk on this. Same Day! Crushing it dude, thank you for being you
@codycallaway90574 ай бұрын
I can feel a disturbance, as if millions of crowdstrike memes are being made
@MichaelGrundler4 ай бұрын
This is the first video I've seen that goes into the technical details of what exactly went wrong with the update and why it caused all those blue screens. 👍
@agmnii4 ай бұрын
Rip the 5 hours of production data I lost at work
@kerplop4 ай бұрын
I work in a medical supply warehouse, and we couldn't use our computers at all until after noon, when they finally got one of the shipping computers up and running. We shipped out as many orders as we got, but I'm sure there are plenty more floating out in cyberspace... It could be a difficult Monday.
@Nossody4 ай бұрын
why is counterstrike on the macys computer?
@chrisaustin99494 ай бұрын
So here's a question, why does any company allow Microsoft or Crowdstrike to push programs onto their production system, ever? Back when I was in IT we would never let anyone do that. We would take the software ourselves and try it on a test system first before putting it into production.
@urizezucm2564 ай бұрын
is friday and the computer knows it
@阮榮強4 ай бұрын
Yup. The first thing I thought of after reading an article about the outage was that Riot Vanguard is implemented in practically the same way. That anti-cheat made me quit the game. Most of the community called us paranoid or just didn't care. Now this happens. I know we were right but it certainly feels good for it to be proven.
@gameboardgames4 ай бұрын
So glad I'm making games today and not doing my usual IT gig.