the crowdstrike situation is wild

  Рет қаралды 347,643

Low Level

Low Level

Күн бұрын

Пікірлер: 1 600
@LowLevelTV
@LowLevelTV 4 ай бұрын
wow haha that video was really cool, I heard these courses at lowlevel.academy are also really cool
@LowLevelTV
@LowLevelTV 4 ай бұрын
haha yeah
@jamescollier3
@jamescollier3 4 ай бұрын
Thank you! Best on the internet
@Mr.Pandey28
@Mr.Pandey28 4 ай бұрын
@@LowLevelTV please make a video on the details and what actually caused the issue !!
@ziggy7676
@ziggy7676 4 ай бұрын
What if I'm already a low level god?
@Kane0123
@Kane0123 4 ай бұрын
Do you know if customers have the ability to not receive instant updates? Or is it something crowdstrike requires at contract time
@metcaelfe
@metcaelfe 4 ай бұрын
They certainly made a lot of machines unhackable
@Парасолька-х1и
@Парасолька-х1и 4 ай бұрын
now noone can steal their data
@wervicecoding
@wervicecoding 4 ай бұрын
Not until somebody plugs in Serial
@gerdd6692
@gerdd6692 4 ай бұрын
That leaves us with the philosophical question: Is a hacked machine unhackable? Or can a dead animal be killed? (Disregarding for the purposes of this discussion that the "hack" was presumably unintentional - it was effective, nonetheless ...
@MathewBoorman
@MathewBoorman 4 ай бұрын
A lot like speed limits, No IT is Safe IT.
@corteztt518
@corteztt518 4 ай бұрын
@@gerdd6692 you got me at ‘can a dead animal be killed’
@XueYlva
@XueYlva 4 ай бұрын
Crowd-Strike: Global Offensive
@comfortcove
@comfortcove 4 ай бұрын
Perfection
@UnrealOG137
@UnrealOG137 4 ай бұрын
Terrific pun
@birigu
@birigu 4 ай бұрын
hahahah lmao
@Lucky9_9
@Lucky9_9 4 ай бұрын
BAHAHAHA 😂😂
@kakorotskywalker
@kakorotskywalker 4 ай бұрын
LMFAOOOOO
@mastaphaaz3424
@mastaphaaz3424 4 ай бұрын
So the learned lesson is never push code in production on friday
@Fraket
@Fraket 4 ай бұрын
that's been an industry-wide standard for decades
@ovalwingnut
@ovalwingnut 4 ай бұрын
Yes.. you need to be "young & bold" to do some things. Like climbing HUGE radio towers. As you get older, you have a tendency not to want to rock the boat. Which has it's own drawbacks So yeah, I've morphed into a chicken 🐔
@neruneri
@neruneri 4 ай бұрын
The real lesson tbh is for managers and executives to stop allowing the overworked IT guy to push code to prod on a friday.
@Mordecrox
@Mordecrox 4 ай бұрын
Some gurus actually go to great length to "demistify this myth" and now they must be in shambles that we have ultimate proof to never do that
@FullDupl3x
@FullDupl3x 4 ай бұрын
read only Friday!
@Patterner
@Patterner 4 ай бұрын
a newsletter email had the following greeting: "Good morning and happy Friday to everyone who doesn’t work in IT."
@DioTheGreatOne
@DioTheGreatOne 4 ай бұрын
Look on the bright side, demand for IT guys has skyrocketed, and what does a sudden high demand spike mean? That's right, IT guys can charge whatever they want now because every company is completely desperate for them.
@SolidIncMedia
@SolidIncMedia 4 ай бұрын
To be fair, that's how people treat IT workers, even when things are working perfectly.
@cix9420
@cix9420 4 ай бұрын
@@SolidIncMedia sir you're on call and im working late, bring me some mcdonalds would ya?
@David-gu8hv
@David-gu8hv 4 ай бұрын
@@SolidIncMedia Lol
@tbg07
@tbg07 4 ай бұрын
@@Patterner Which newsletter? Wanna see this.
@georgeprout42
@georgeprout42 4 ай бұрын
Crowdstrike, according to their name, worked perfectly.
@justinlinotte2981
@justinlinotte2981 4 ай бұрын
why they choose a name like this?
@emperorarasaka
@emperorarasaka 4 ай бұрын
@@justinlinotte2981 all part of the CIA backdoor testing
@context_eidolon_music
@context_eidolon_music 4 ай бұрын
@@justinlinotte2981 To take down the Internet for real soon, dummy.
@KangJangkrik
@KangJangkrik 4 ай бұрын
​@@justinlinotte2981 for fun? Why not ¯\_(ツ)_/¯
@murtajiz545
@murtajiz545 4 ай бұрын
@@justinlinotte2981 No idea but it’s painfully poetic
@JonitoFischer
@JonitoFischer 4 ай бұрын
”The only safe computer is the one that does not boot" Crowdstrike CEO.
@Shocker99
@Shocker99 4 ай бұрын
Technically true
@mar25947
@mar25947 4 ай бұрын
😂😂😂
@babybirdhome
@babybirdhome 4 ай бұрын
To be fair, this violates one of the three tenets of cybersecurity, and it's one that people outside (and sometimes even inside) of the field forget the most. Confidentiality, Integrity, and *_AVAILABILITY._* If you're missing any of these three, you haven't got security.
@aisle_of_view
@aisle_of_view 4 ай бұрын
All the times I've had to explain to management why we should wait a few days before implementing an update, only to be met with blank stares. I'm loving every second of this.
@InvisibleHotdog
@InvisibleHotdog 4 ай бұрын
@@aisle_of_view they probably conveniently forgot
@devrim-oguz
@devrim-oguz 4 ай бұрын
The only thing is this happened via unattended upgrades
@CoderDBF
@CoderDBF 4 ай бұрын
I agree and disagree. For example a server should probably try to be up to date with security. It can happen that a security issue has been released then there will be an entire army of bots sniffing every server that hasn’t been patched yet in an attempt to hack them. So you want to be fast in plugging holes because those bots will be very fast in finding your unpatched servers. Any other update like windows workstations can probably wait a few days without issues.
@benjaminblack91
@benjaminblack91 4 ай бұрын
@@CoderDBF This particular case would be considered a security update, as it is an update to endpoint security software.
@MrCyanist
@MrCyanist 4 ай бұрын
@@CoderDBF Hard disagree even patches targeting critical security flaws have enough time to at least test in non critical pilot servers or clients. If a security issues found its already known for a while by bad actors while issue becoming well known will increase attackers testing for few hours will hardly change anything compared to auto pushing to every server/client.
@justinlinotte2981
@justinlinotte2981 4 ай бұрын
its cool to see pure technical explanation of how it happens, its far from common medias that only use shocking words to get as much audience as possible while they don't know a thing about what happened
@TheStoneMountain1
@TheStoneMountain1 4 ай бұрын
I totally agree! I for one barely read the "regular" news because of how it almost always feels like they have one sentence of information to deliver and extrapolate it to a whole article with a bunch of word poop and no real info. So I prefer this type of delivery every time! Informative, deep dive into the interesting bit, short and quick to the point!
@martin.1976
@martin.1976 4 ай бұрын
@@justinlinotte2981 Definitely! I was quite confused about how this could have happened in the first place - and none of those other channel had covered that - but seeing that the actually delivered driver was all nulls explained it perfectly well. And this is likely why this passed all testing and everything.
@robertjenkins6132
@robertjenkins6132 4 ай бұрын
@@TheStoneMountain1 Yes, and it's not just tech. Today I was reading in the New York Times about how there were violent protests in Bangladesh over a "quota system" for government jobs, but they declined to explain what this "quota system" was. I was curious to know what would provoke such protests. I found better information on Wikipedia, in articles such as "Quota system of Bangladesh Civil Service" and "2018 Bangladesh quota reform movement". MSM reporting is so vague and dumbed-down. (They also do "fact checks" on statements, e.g., by Trump or a random conspiracy theorist on social media, that anyone with a brain would know are _obviously_ false.) It insults my intelligence.
@TRFAD
@TRFAD 4 ай бұрын
@@TheStoneMountain1 Yeah for 4 pages, and "you'll never guess the shocking reason" but never even mentions it once while you fight through ads and trying to click the next button.
@phoenixflower1225
@phoenixflower1225 4 ай бұрын
100% I love this explanation - typical media always exaggerates everything
@MenaceInc
@MenaceInc 4 ай бұрын
Today was not the best day for me to wear my Crowdstrike t-shirt...
@brutely9718
@brutely9718 4 ай бұрын
@@MenaceInc No don't think.. crowdstrik is saving world from hackers. Mistakes happens
@akpokemon
@akpokemon 4 ай бұрын
Or if you enjoy small talk, it's the best day--great conversation starter
@araz911
@araz911 4 ай бұрын
this is most likely due to open source software
@shauas4224
@shauas4224 4 ай бұрын
​@@araz911 what
@araz911
@araz911 4 ай бұрын
@@shauas4224 the shutdown due to open source libs most likely
@theAmazingJunkman
@theAmazingJunkman 4 ай бұрын
The fact that a segfault just caused Y2K to happen 24.5 years late is wildly amusing to me
@fnytnqsladcgqlefzcqxlzlcgj9220
@fnytnqsladcgqlefzcqxlzlcgj9220 4 ай бұрын
HAHAAHAHA didn't think about it like that lol
@ilonachan
@ilonachan 4 ай бұрын
oh damn ur right, this IS what they were afraid of!
@NickRoman
@NickRoman 4 ай бұрын
Except, we had to update BIOSs to prepare for Y2K. This was: reboot, delete a file, reboot. I'm thankful that the fix was so easy (albeit time consuming).
@JustSomeDinosaurPerson
@JustSomeDinosaurPerson 4 ай бұрын
@@NickRoman Unfortunately this did not work for every affected system. Throughout the organization I work for we had to use restore points because either the files wouldn't delete themselves or deleting said file wouldn't resolve the issue. What an absolute fucking nightmare. Edit: To clarify, we still got all of it fixed. It just took a hell of a lot longer than wanted and many of us stayed overnight. Corporate straight up expensed all of our food orders, no questions asked.
@MrJamesVanEngen
@MrJamesVanEngen 4 ай бұрын
We prepare for a primary election while identity theft is rampant. Magnificent! 🤦 ... The oligarchy of American credit scoring companies might as well be assigning random credit scores to each citizen within legal ramifications again. #VoteMillennial in 2024! 🪙💵💳🤖🇺🇸
@lucaslannes4004
@lucaslannes4004 4 ай бұрын
I work on an airline, you dont imagine the mess. Oh Jesus, today was a nightmare. Hope tomorow get better.
@MarianoLu
@MarianoLu 4 ай бұрын
I feel for you buddy
@nicejungle
@nicejungle 4 ай бұрын
it will happen again as long as airline companies use windows
@malavoy1
@malavoy1 4 ай бұрын
@@nicejungle And if they'd used Linux, they would have went down in April with kernel panics. It's not the OS that's the problem, it's Crowdstrike.
@nicejungle
@nicejungle 4 ай бұрын
@@malavoy1 And if they'd used Linux, one reboot and you switch back to the previous kernel Down time : one reboot Compare to windows : you're screwed
@malavoy1
@malavoy1 4 ай бұрын
@@nicejungle But Linux users are tech savvy. Most users of Windows are not, so MS hides safe mode behind multiple reboots to prevent them from ruining their system (and they would blame MS if they did ruin their system). Once in safe mode you can roll back system changes.
@MSThalamus-gj9oi
@MSThalamus-gj9oi 4 ай бұрын
This will absolutely *not* be the last time something like this happens. When I first started in the industry, everything was packaged on disk/disc. Fixing a bug after shipping was EXPENSIVE, so we got the product to a 99% stable place and then kept trying to squeeze that last 1% of bugs out. Now? People just throw garbage over the fence, figuring they can just ship a patch later. Kernel mode software just *cannot* be developed that way. But... cyber security companies have time pressure that other kernel mode developers don't necessarily face. To be useful, such an app must be updated and deployed amazingly quickly, especially for kernel space, but that agility comes at the cost of stability. When the cure is worse than the disease, though... there's a problem.
@kugelblitz1557
@kugelblitz1557 4 ай бұрын
At the very least (assuming it's not an immediate security risk) then updates should be delayed a couple hours by region or something, so if this happens then it's a smaller section of customers that get screwed over and they have time to cancel it and get it fixed for the rest of the world.
@Bubblessss420
@Bubblessss420 4 ай бұрын
@@MSThalamus-gj9oi exactly! thats why i bought some CS stocks after the fall down. CS is a great vendor but this kind of things unfortunatelly could happen.
@framegrace1
@framegrace1 4 ай бұрын
@@kugelblitz1557 Just apply CI/CD techniques. The first rule is "Only package once, at the beginning of the release, same package tested is the same deployed".
@workmad3
@workmad3 4 ай бұрын
While I agree that this sort of software can't be developed in the same way as a lot of stuff, there's also not really any evidence here that it was. The fact the update file was completely zeroed out points to a failure way past a dev shipping a bad code update... I can't see any way this happens without it being a build or deploy failure.
@MSThalamus-gj9oi
@MSThalamus-gj9oi 4 ай бұрын
@@Bubblessss420 I was thinking of doing the same. The stock price dropped 20%, but you know it'll bounce back. It's a bargain right now. (No, I'm not a shilling bot. :D)
@yassine-sa
@yassine-sa 4 ай бұрын
They literally waited for most KZbinrs to go on vacation before rolling out the update lol
@prezadent1
@prezadent1 4 ай бұрын
'Literally' as opposed to what? Figuratively?
@katnoto8993
@katnoto8993 4 ай бұрын
@@prezadent1 using "literally" in this way is a form of hyperbole. English is cool like that 😎
@Maxawa0851
@Maxawa0851 4 ай бұрын
@prezadent1 if you google the definition of 'literally' the second definition of literally is literally 'not literally'
@Reydriel
@Reydriel 4 ай бұрын
​@@Maxawa0851Yep, a ton of English words have contradictory meanings like this unfortunately lmao
@NickRoman
@NickRoman 4 ай бұрын
They probably push out updates every few days.
@joshuac5229
@joshuac5229 4 ай бұрын
This is just viral marketing for Captain Crunch's new cereal, "OOPS! ALL NULL!"
@lashlarue7924
@lashlarue7924 4 ай бұрын
😂😂😂 thank you, please take my like, I cannot even 😂😂😂💀⚰️🪦
@_JohnHammond
@_JohnHammond 4 ай бұрын
HAPPY VACATION ED WHAT A DAY (/WEEKEND/WEEK/MONTH) 💥
@brandonw1604
@brandonw1604 4 ай бұрын
Can we rename it to Blue Falcon?
@Kane0123
@Kane0123 4 ай бұрын
Bro was definitely having beers by the pool before this. Spring break feels for sure
@darkshoxx
@darkshoxx 4 ай бұрын
@_JohnHammond you don't get any rest at the moment do you 😆
@faust9091
@faust9091 4 ай бұрын
Why are you not the top comment wtf? Love you both ❤
@lumikarhu
@lumikarhu 4 ай бұрын
you're also gonna farm videos out of that as usually, aren't you don't forget to mention that while CS released a faulty package tons of morons allowed it to update itself (or even worse - pushed updates themselves) on a fucking Friday. they deserve that, it's basic sysadmin knowledge
@BogdanTimofte
@BogdanTimofte 4 ай бұрын
25 years ago, operating systems started signing drivers. 15 years ago, the same thing happened with the bootloader. A few years ago I heard that all PCs had to be replaced so that Windows could guarantee security. Now I understand that it gives total control over the computer during the initialization phase to a program without the slightest verification, just because it is in the right place in the filesystem and the name seems familiar? Live and learn...
@BrayanRuiz-m3w
@BrayanRuiz-m3w 4 ай бұрын
this was so bad that bro had to make a YT video while been on vacation. what a legend
@MattGreer
@MattGreer 4 ай бұрын
But why did they roll out this update to every computer in the world all at once? Why didn't they run a canary? Why didn't they do the rollout in phases? It's low level code running in the kernel, and it is deployed in machines all around the world by thousands of businesses, why on earth wouldn't they be more cautious with the rollout? This is quite possibly the most reckless deployment in the entire history of software.
@Pipe0481
@Pipe0481 4 ай бұрын
Because AFAIK they didn't update the software itself, but rather they just pushed a new virus signature database file. The real issue is that the Falcon program can't handle invalid files
@MattGreer
@MattGreer 4 ай бұрын
@@Pipe0481 That's still doable with a slow roll out, canary, dog fooding, etc. Anything at this scale should be done with extreme care. Heck, even if not at this scale, there's no reason to be so reckless.
@michaelharrison1093
@michaelharrison1093 4 ай бұрын
@@MattGreer simple answer - they are morons
@auzziegamer4661
@auzziegamer4661 4 ай бұрын
@@Pipe0481 more technically windows can't handle invalid files or any program for that matter
@ShadoFXPerino
@ShadoFXPerino 4 ай бұрын
@@MattGreer If you canary then all the non-canaries are vulnerable to zero day from the new virus.
@lols11
@lols11 4 ай бұрын
"//Just a small fix, no need to test it"
@MarkusHobelsberger
@MarkusHobelsberger 4 ай бұрын
Famous last words.
@danielburger2550
@danielburger2550 4 ай бұрын
LGTM
@victotronics
@victotronics 4 ай бұрын
"The entire internet" for as far as it runs Windows. If it had taken down Linux the actual internet would have gone down.
@Sandromatic
@Sandromatic 4 ай бұрын
Apparently in April, crowdstrike for debian actually went down in a similar manner, (kernel panics.) thankfully I guess no-one actually uses crowdstrike for Linux so no-one actually cared.
@lashlarue7924
@lashlarue7924 4 ай бұрын
Facts.
@TimothyWhiteheadzm
@TimothyWhiteheadzm 4 ай бұрын
Yeah, my internet was just fine, so not sure what he meant. Airports are not 'the internet'.
@NickRoman
@NickRoman 4 ай бұрын
@@TimothyWhiteheadzm , well, I'm thankful that KZbin, Netflix, HBO... were all fine or fixed quickly.
@entcraft44
@entcraft44 4 ай бұрын
Worldwide outages + clickbait = "The internet is going down". They claimed the same with the recent massive Facebook outage.
@cslearn3044
@cslearn3044 4 ай бұрын
Crowdstrike, tests code only once, at production
@gerdd6692
@gerdd6692 4 ай бұрын
Not only once - they do millions of parallel tests on a vast array of systems - without making the code platform independent they could only better this by cranking up a few zillion virtual machines or container "farms" ...
@Brahvim
@Brahvim 4 ай бұрын
@@gerdd6692 Yeah, this one sounds like a problem with deployment instead, doesn't it?
@martin.1976
@martin.1976 4 ай бұрын
I don't think we can dismiss the problem with such a "simple" explanation. Most certainly, they did test their code very properly and extensively. But they missed one of the most important factors: that things could possibly go very wrong in transmission.
@matt7253
@matt7253 4 ай бұрын
@Brahvim that's what I was thinking/the build. Development can have different config for build/deploy and you may not see until you pushed to the environment. Even with a UAT sometimes the config can be slightly different than prod even though it should be as close if non near identical to prod.
@cslearn3044
@cslearn3044 4 ай бұрын
@@gerdd6692 yeah im just joking
@seeibe
@seeibe 4 ай бұрын
My vscode tunnel into my dev machine randomly stopped working yesterday and made me realize how much it sucks to be dependent on someone else for your own setup.
@najtofnin2009
@najtofnin2009 4 ай бұрын
Dude. You're using vscode. Welcome to Microsoft dependence inc.
@not_kode_kun
@not_kode_kun 4 ай бұрын
linux, emacs, vim, helix are waiting for you with wide arms
@deeiks12
@deeiks12 4 ай бұрын
I'm sure lots of people are looking for vulnerabilities in crowdstrike now. I don't know how public this was before that SO many companies are using their product....
@cbaesemanai
@cbaesemanai 4 ай бұрын
they just showed us the vulnerability, creating a sys file full of nulls.
@MarianoLu
@MarianoLu 4 ай бұрын
@@deeiks12 it is very public that literally everyone (in corporate IT) uses their products, the thing is that is transparent for most non IT people and they do not have a consumer version. And unfortunately they are ( or were) the best in the business.
@renato360a
@renato360a 4 ай бұрын
@@cbaesemanai you had to actually be them to do that in the first place, so.. that's not it. Unless maybe if you hack the provider of their update pipeline, which might actually be what happened. But I guess if you did that you could break so much more.
@cbaesemanai
@cbaesemanai 4 ай бұрын
@@renato360a I mean using it as a local exploit.
@Shocker99
@Shocker99 4 ай бұрын
Crowdstrike is well known. They've had hordes of people looking for vulnerabilities in their software for years.
@internetuser8922
@internetuser8922 4 ай бұрын
I watched a ton of videos on WTF even happened here. This was the only one that actually explained what went wrong in any kind of detail, and you're on vacation. Absolutely amazing channel here.
@MarkusHobelsberger
@MarkusHobelsberger 4 ай бұрын
This. It's a super-simple error, yet no mainstream media explains it in a comprehendable way.
@edwardallenthree
@edwardallenthree 4 ай бұрын
Excellent video! My wife has high confidence in her coworkers following the instructions and fixing their desktops and laptops. I think the person who replied all to the instructions asking for his encryption key is proof that this is going to be one long weekend for people in IT. Never been happier to be retired.
@ScottForrest420
@ScottForrest420 4 ай бұрын
Me too. Retired two months ago from a law firm who uses Crowd Strike through a consultant/VAR. I was imagining the entire firm losing their mind today due to this. Funny/not funny, but so happy it wasn't me having to deal with it.
@chrisalmighty
@chrisalmighty 4 ай бұрын
@@edwardallenthree you're spot-on with that 😅
@j340_official
@j340_official 4 ай бұрын
So community notes on Twitter is saying the viral tweet that claimed it was a null pointer dereference is BS. And crowdstrike put out a blog statement that “This is not related to null bytes contained within Channel File 291 or any other Channel File.”
@VioletEmerald
@VioletEmerald 4 ай бұрын
Hmm interesting
@yarpen26
@yarpen26 4 ай бұрын
I just got done writing a comment under Fireship's Code Review saying how Ed is bound to release a video on this as well soon enough, I reload my YT start page and I can see it up there, from 11 minutes ago.
@thunderb00m
@thunderb00m 4 ай бұрын
How the hell is a multi billion dollar company not have basic error handling like null check!? Like seriously do they not take functional safety seriously!?
@Shocker99
@Shocker99 4 ай бұрын
How do they not a pre-update push set up that acts as if the machines are actual client computers to test all updates before being pushed to real clients?
@TheFunkyTechGuy
@TheFunkyTechGuy 4 ай бұрын
@@Shocker99 My thinking exactly, this is basic stuff.
@linkernick5379
@linkernick5379 4 ай бұрын
Almost nobody checks the result of many functions, e.g. malloc or printf. Defensive programming techniques are clunky, cumbersome but still dont save you from errors (ref: "The Art of Software Testing" book). The only way to get rid of whole classes of errors is with good type system, in other words to detect by compiler.
@AndrewBrownK
@AndrewBrownK 4 ай бұрын
cue elitist C++ dev entering the scene and saying "no no this isn't a technology problem, this was a skill issue haha, I'd NEVER do this, I'm too skilled"
@williamforsyth6667
@williamforsyth6667 4 ай бұрын
"error handling like null check!" It should have been caught much earlier. Seem they have no integrity check of their binaries during the build-test-distribution process.
@0xkleo
@0xkleo 4 ай бұрын
The crowd was striked by a blue screen of death
@samiraperi467
@samiraperi467 4 ай бұрын
Struck.
@vizionthing
@vizionthing 4 ай бұрын
@@samiraperi467 Stricken
@emusunlimited
@emusunlimited 4 ай бұрын
@@samiraperi467 Moron, your bus is leaving… 🚌 It’s CrowdStrike not CrowdStruck
@emusunlimited
@emusunlimited 4 ай бұрын
@@samiraperi467 Never heard of CrowdStruck… is that some tech company or something?
@TabTray
@TabTray 4 ай бұрын
@@vizionthing Stricketh
@theeternalsw0rd
@theeternalsw0rd 4 ай бұрын
By the way, this is the second BSOD software update push George Kurtz, CEO of CrowdStrike has presided over. First was April 21, 2010 as CTO of McAfee when an update inadvertently deleted svchost.exe from Windows XP machines. That would have been more massive but for the lack of always auto-update devices nowadays.
@ernstraedecker6174
@ernstraedecker6174 4 ай бұрын
"Who needs svchost.exe anyways? I'm an ordinary Windows user, I don't use all those nerdy tools. Just trust the authorities. Safe and effective!" - All my friends.
@Veptis
@Veptis 4 ай бұрын
How did this roll out this widely? Is there no canary? Is there no QA? How is business and airports the first wave of roll out... How do you put all trust in a single third party?
@samniechcial8493
@samniechcial8493 4 ай бұрын
@@Veptis I’m wondering the exact same thing. From the perspective of a DevOps/Infra engineer - What kind of update deployment strategy is “just hit all billion machines at once”? No canary? No region by region? More questions to be asked here than just what went wrong in the actual code… If this is how they deploy code with a bug, imagine if they deployed code with a serious vulnerability? We need to hear about Q/A and deployment strategy at CrowdStrike!
@outtakontroll3334
@outtakontroll3334 4 ай бұрын
good questions, and someone is damnwell going to have to answer them
@vappyreon1176
@vappyreon1176 4 ай бұрын
​@@samniechcial8493just stop using windows they're not worth the security issues.
@tigerchills2079
@tigerchills2079 4 ай бұрын
"How is business and airports the first wave of roll out..." someone else in the comments answered that already: they are the only ones to roll out to. it's big business software. there is no consumer grade product
@ShadoFXPerino
@ShadoFXPerino 4 ай бұрын
There's no canary because what is pushed are virus signatures. If you canary then all the non-canaries will be vulnerable to the zero-day virus. Getting instant updates is the entire point of the product. QA probably happened, but after the QA they deployed the tested file to some file server and the file got corrupted in transmit into all-zeros, which causes a crash loop.
@69clouds
@69clouds 4 ай бұрын
The fact that it was an Antivirus that performed the single most successful malware attacks ever is just pure poetry. Another win for the "remind me later" to every update gang. My dad: Come on, it's just an internship, what's the worst that could happen? Me: "You can't hack a system if the system doesn't work! " - Cybersecurity The alpha move of doing something that would make your stock value crash, but simultaneously freezing the stock market so that it can't.
@Dead_Goat
@Dead_Goat 4 ай бұрын
Windows update doesnt matter. THis is not a win for remind me later as you cannot reind me later with this terrible crowdstrike rootkit.
@69clouds
@69clouds 4 ай бұрын
I am aware of that, it just sounded like a good joke so I added it up.
@cix9420
@cix9420 4 ай бұрын
if you would like to make a bet, many futures brokers are still working, so when the market opens you can go short or long with some margin if you think this will have an effect on the worldwide markets (it will)
@ajinkyamogre8515
@ajinkyamogre8515 4 ай бұрын
So you just compiled the top comments across youtube over this topic and copy pasted them here for likes. Cool.
@cix9420
@cix9420 4 ай бұрын
@@ajinkyamogre8515 with how internet speech is i didnt even realize it, i just assumed all of this stuff was one full sarcastic comment
@c_ornato
@c_ornato 4 ай бұрын
It baffles me that people would modify any piece of code that sensible without testing on a friday, hell even I check 3 times that my KBPs are correct before restarting
@lion21297
@lion21297 4 ай бұрын
I'm sure they did test it.. the issue must have happened when actually pushing the update live. Doesn't excuse a friday update though. That's just asking for trouble
@asynchronerflugelflitzerim8481
@asynchronerflugelflitzerim8481 4 ай бұрын
@@c_ornato there was still a network connection, right?
@c_ornato
@c_ornato 4 ай бұрын
@@lion21297 Doesn't excuse pushing to every user at the same time either, you'd think the devs for a highly-used expensive piece of software would be more rigorous but it seems the dev instinct to push stuff fast does not discriminate.
@ninele7
@ninele7 4 ай бұрын
@@lion21297 You need to understand that it is a security software. Maybe they've implemented protection from new attack vector. And hackers don't rest on weekends. As we see, problem happened at some unpredictable late stage (file became all zeroes, it's not compiler output). Even if they released it on any other weekday effect would be the same.
@mallninja9805
@mallninja9805 4 ай бұрын
@@lion21297 Sounds like there's a gap in their deployment testing...
@JohnFink-p5l
@JohnFink-p5l 4 ай бұрын
As an IT guy, I have been working non-stop all day today. It's utter insanity here at this company. Those dumb bastards
@haroldcruz8550
@haroldcruz8550 4 ай бұрын
Hey at least now they realize how important you are.
@BoraHorzaGobuchul
@BoraHorzaGobuchul 4 ай бұрын
Oh my god, they killed kernel! The bastards!
@pieterbezuidenhout3757
@pieterbezuidenhout3757 4 ай бұрын
At least we got our Servers up and running in 2 hours this morning, overtime cash, I feel you, Brother.
@MarkusHobelsberger
@MarkusHobelsberger 4 ай бұрын
@@haroldcruz8550 They are rather going to blame him for the problems he's trying to fix. Unfortunate truth of working in IT.
@not_kode_kun
@not_kode_kun 4 ай бұрын
@@pieterbezuidenhout3757 yall have windows in your servers? what the fuck are yall smoking
@Somezable
@Somezable 4 ай бұрын
I bet one of the reasons for this is the LEAN. A plaque of corporate goal of efficiency that ends up ruining workplaces, if allowed to continue too long. I can guarantee we will later hear, if the company isn't able to hide it, workers at crowdstrike were overworked, overstressed, always pushed into a rush and denied time to make critical quality assurance checks and tests, that would have caught this error.
@rikuleinonen
@rikuleinonen 4 ай бұрын
I feel like all the blame is going to be pushed onto the employee that coded in the mistake while the CEO/manager that made them do so via overworking them etc. will get away scott-free.
@observant6953
@observant6953 4 ай бұрын
Thank you so much for actual getting into technical details. All other articles just repeat "well, everything is down".
@DJJOOLZDE
@DJJOOLZDE 4 ай бұрын
Kinda neat that so much of the internet depends on a few people uploading critical files.
@annmaryjohn3258
@annmaryjohn3258 4 ай бұрын
@@DJJOOLZDE The internet is fine, it's the computers using Microsoft that has crashed.
@MK73DS
@MK73DS 4 ай бұрын
Just use a closed source piece of software on our closed source OS for our critical application, everything will be fine.
@MathewBoorman
@MathewBoorman 4 ай бұрын
Boss, The closed source OS and hardware you have on your desk is not good enough to act as a dumb TV or kiosk by itself. First install the closed source drivers from a bunch of random hardware vendors. Then add some tooling to actually install and configure the host to do its thing. Add some more software to manage the truck loads of host security settings from all the stuff we don't need anyway, but can't remove. Add closed source kiosk software or maybe the POS application, which is just a wrapped browser. Buy some more security software since we can't trust any of the perviuos bits to work. Don't think about deploying a cheaper Open Source & Open Hardware solution, like a rasberry Pi.
@football42241
@football42241 4 ай бұрын
Open/closed isnt the issue in this case. Corrupt auto software updates is, and they can (and do) happen to both.
@CharGorilla
@CharGorilla 4 ай бұрын
@@football42241 Except you'd be ripped to shreds in an open source project if you committed code that runs kernel mode, downloads dynamic code off the internet and runs it in kernel mode, doesn't have any sort of integrity check on what it downloads off the internet, like a digital sgnature, or even a CRC32, AND the virtual machine / interpreter which runs the code that was downloaded off the internet isn't sand-boxed and lets the dynamic code use naked pointers. How many freaking basic mistakes did this "cybersecurity" company make here. Not to mention that all this time, their "cybersecurity" software has been one giant RCE waiting to happen if you manage to spoof the DNS of the update server, or MITM that HTTP(maybe S) request that we all know doesn't check for a specific root authority). I wonder how long the NSA have known about this one. I'd hope not as long as they kept Eternal Blue under wraps.
@not_kode_kun
@not_kode_kun 4 ай бұрын
@@football42241 nope, auto software updates are extremely rare on open source operating systems. since they're made by devs, for devs, and all of us devs hate that shit. This crowdstrike thing could've very well happened even if the whole world ran on linux. but at least, it'd only happen to people who ran the update command, and even then all they'd need to do to get their computer back is rollback to previous kernel and reboot. The same problem could've happened, but it would've been in a smaller scale and easier to fix.
@alexholker1309
@alexholker1309 4 ай бұрын
Thanks for the video. The file being corrupted does seem to explain how it could get past testing, if the failure to write only happened after they verified the file was safe.
@garanceadrosehn9691
@garanceadrosehn9691 4 ай бұрын
Also note this story: *"Major Microsoft 365 outage caused by Azure configuration change"* ... _"Microsoft says an Azure configuration change caused a major Microsoft 365 outage on Thursday, affecting customers across the Central US region. This massive outage started around 6:00 PM EST and prevented users from accessing various Microsoft 365 apps and services."_ This happened hours *before* the Crowdstrike issue surfaced, and I also wonder if there might have been some connection.
@jacominnaar
@jacominnaar 4 ай бұрын
We had issues on Azure with HTTP2 traffic in South Africa. I'm wondering if they are related.
@hellowill
@hellowill 4 ай бұрын
yeah I noticed the Azure outage too. Funny how nobody cares or even blames MS for the Crowdstrike/Windows interaction.
@gabriel55ita
@gabriel55ita 4 ай бұрын
​@@hellowill why blaming ms for something another company did wrong, it's their responsibility to not ship something bad. Windows offers you to boot in safe mode to remove the driver that start prioritized
@workmad3
@workmad3 4 ай бұрын
The Azure issue caused some of their compute to not be able to access storage. Then hours later, crowdstrike pushes an update that's completely zeroed out. If there isn't some connection and cascading failure discovered in postmortem it's going to be one hell of a coincidence 😅 Still a pretty massive failure on crowdstrike's part to manage to drop the file onto so many machines without some verification raising an alert that this file is screwed
@gzoechi
@gzoechi 4 ай бұрын
​@@workmad3Also updates are usually pushed to a small selection of devices and when nothing bad happens, the number is increased
@donleyp
@donleyp 4 ай бұрын
I was supposed to fly to Japan this morning. Crowdstrike canceled my flight. I’m glad you’re already at your destination, Ed. Have fun!
@misogear
@misogear 4 ай бұрын
The most important piece of software don't have rollback mechanism when update is broken is mind blown to me. 😂😂
@Xehlwan
@Xehlwan 4 ай бұрын
@@misogear Oh, there is - Crowdstrike's update was just designed in a way that doesn't make use of that Windows feature.
@not_kode_kun
@not_kode_kun 4 ай бұрын
@@Xehlwan except window's rollback is completely useless because you need to boot into the system, then jump through menus to finally rollback. the whole purpose of rollback is for when an update is BROKEN. if an update is broken and you can't rollback, then you dont have rollback. On linux, you can still rollback your kernel after a kernel panic (BSOD) with a single reboot
@kylekatzin1563
@kylekatzin1563 4 ай бұрын
So funny, my first thought was who tf pushes out an update on a Friday..
@MarkusHobelsberger
@MarkusHobelsberger 4 ай бұрын
Everyone does (working in software development in a small company here). If something goes wrong you will have time to fix it over the weekend. Something so critical should have been tested better, though.
@brssnkl
@brssnkl 4 ай бұрын
Your look is giving IT person they had to bring back from middle of his holiday to fix this. It feels so authentic :D
@test40323
@test40323 4 ай бұрын
i find it incredible crowdstrike didn't do a staged rollout considering the risks. this is an acute reminder that lab testing can't cover all scenarios and that a fallback plan is always necessary no matter how remote the probability of mishaps. defn: risks = probability(event) × cost of damage (event)
@Sandromatic
@Sandromatic 4 ай бұрын
They actually did apparently. It's just, they didn't notice it until it'd rolled out to the majority of their userbase. Meaning they did a staged rollout but it was too fast/no-one was monitoring it to catch. I mean, that probably happens when you're doing it on a Friday.
@BoraHorzaGobuchul
@BoraHorzaGobuchul 4 ай бұрын
Or testing
@jbird4478
@jbird4478 4 ай бұрын
@@Sandromatic They apparently didn't notice at all. It was Google who pointed the finger, and even then it took them two hours to say "ehm.. yeah, it was us."
@BoraHorzaGobuchul
@BoraHorzaGobuchul 4 ай бұрын
"looks like I picked the wrong week to stop sniffing glue"
@crispybatman480
@crispybatman480 4 ай бұрын
Big hugs to all the people having to manually recover systems today.
@rcstl8815
@rcstl8815 4 ай бұрын
I wonder what the Linux download sites are seeing?
@NewKiwiJK
@NewKiwiJK 4 ай бұрын
yay overtime
@jnawk83
@jnawk83 4 ай бұрын
​@@rcstl8815probably not much, corporations and enterprises don't tend to knee jerk quite that fast, if at all.
@ecchioni
@ecchioni 4 ай бұрын
Where the fuck was the test team? Oh wait... The modus operandi of a modern hackshop is fuck QA.
@joseoncrack
@joseoncrack 4 ай бұрын
Yep. But remote updates are evil anyway. That's a nonsense from a sysadmin POV.
@ecchioni
@ecchioni 4 ай бұрын
@@joseoncrack They are not paying for QA, do you think they'll pay for a stage environment and a team to manage it where the update is tested before it rolls to the rest of the org?
@DoinThatRag
@DoinThatRag 4 ай бұрын
This is so sadly true. My CEO who likes to pretend he's a dev but isn't and has no background operates like this. I swear it is due to the development of CI/CD pipelines and how relatively "easy" it easy to write some APIs, just have the devs write tests for them, and throw out the latest update to your microservice. I mean all software functions like that doesn't it? The cloud systems you run for your little API services or pushing critical and sensitive updates to customer systems, what's the difference? So what if it is a kernel driver, just keep those updates rolling!
@haroldcruz8550
@haroldcruz8550 4 ай бұрын
QA is not very cost effective. Why test when you can do it live
@ecchioni
@ecchioni 4 ай бұрын
@@haroldcruz8550 Maybe to prevent a 20% stock drop and a big fuck you from the customers?
@glowingone1774
@glowingone1774 4 ай бұрын
cloudstroke
@joshuathomasbird
@joshuathomasbird 4 ай бұрын
jfc lmao
@Fs3i
@Fs3i 4 ай бұрын
clownstrike
@redreaper5083
@redreaper5083 4 ай бұрын
Clownstroke
@haroldcruz8550
@haroldcruz8550 4 ай бұрын
Clownstruck
@bionic_batman
@bionic_batman 4 ай бұрын
Cloudstrife
@Stratelier
@Stratelier 4 ай бұрын
This incident feels like something out of a Tom Scott spec video (re: "that time Google forgot to check passwords") ... or the definition of an "onosecond". 2:55 - Wow. Just wow. I've personally experienced a few cases of complete individual file loss leaving behind "all nulls" (presumably from a failed deferred write-to-disk). But those were just two or three personal userfiles -- I couldn't imagine this occurring with a critical driver or system file.
@benbohannon
@benbohannon 4 ай бұрын
So, for the few companies Microsoft trusts to operate at the sys/kernel level, all updates should be funneled through Microsoft test channels. They cannot have companies breaking their install base.
@TonyJewell0
@TonyJewell0 4 ай бұрын
THIS. I was ranting to my wife yesterday evening about this. Bless her, she knows nothing about IT and made all the right noises.
@tallpaul9475
@tallpaul9475 4 ай бұрын
Your explanation of how the driver integrates to the system to make it depend on it makes sense. Thanks for giving informative details at an understanding level.
@_hugoi
@_hugoi 4 ай бұрын
That was just like the *Y2K* bug... but this time nobody was expecting it LOL
@savagej4y241
@savagej4y241 4 ай бұрын
And the Y2K bug probably would have been less disruptive back then compared to a Crowdstrike error nuking systems operations worldwide now, because 24 years ago, you didn't have mission critical systems that require "always online" connections. It was a transitionary phase and older fallbacks were at the ready. Now its more damaging because the more modern systems ARE the fallback.
@nimrodsmusic
@nimrodsmusic 4 ай бұрын
This is the best and direct answer to the question. Every other big news outlet goes on and on and never actually tells us what went wrong. Well done
@andybreuhan
@andybreuhan 4 ай бұрын
How was this .sys file signed? If it is all 0. how was Windows able to load this? Why are there no checks in place?
@Hexanitrobenzene
@Hexanitrobenzene 4 ай бұрын
It seems that it is a submodule which is loaded by a CrowdStrike agent itself. That means they don't do basic checks...
@glitchy_weasel
@glitchy_weasel 4 ай бұрын
Nobody can't hack into your network if your entire network is down - I say this is mission accomplished 😎
@j_stach
@j_stach 4 ай бұрын
Today was the most secure Windows has ever been
@tongpoo8985
@tongpoo8985 4 ай бұрын
This is what I call a "paradigm shift" in cyber-security.
@m4rvinmartian
@m4rvinmartian 4 ай бұрын
*Make the SKINNIEST reference at bootdriver position, make sure it works... NEVER change it. Load everything else after the system is stable.*
@jeremybuckets
@jeremybuckets 4 ай бұрын
"we're kind of dependent on these companies, and when they get it wrong, the whole world collapses. kind of makes you think." such a cheerful delivery of that truly terrifying statement.
@emilmofardin2.0
@emilmofardin2.0 4 ай бұрын
My dad is a senior developer and we watched this happen in real-time. We spent the day installing Linux instead.
@jorper2526
@jorper2526 4 ай бұрын
That isn't the fix. CrowdStrike also has a linux agent. EDR's MUST work at the kernel level to do their job. It is just that in this case, they messed up the file for Windows, not for Linux or Mac..
@ItsCOMMANDer_
@ItsCOMMANDer_ 4 ай бұрын
@@jorper2526 although it is easier to monitor such stuff from userlevel in linux, i heard
@rikuleinonen
@rikuleinonen 4 ай бұрын
@@jorper2526 is the Linux agent even Kernel-level? I recall hearing somewhere that it doesn't go nearly as deep as the Windows version.
@jorper2526
@jorper2526 4 ай бұрын
@@rikuleinonen it depends. But by default, yes. They did crash numerous linux machines in April.
@rikuleinonen
@rikuleinonen 4 ай бұрын
@@jorper2526 thanks for the info.
@_GhostMiner
@_GhostMiner 4 ай бұрын
*Hackers: try, and fail to take various systems down* *Crowdstroke: Fine. I'll do it myself.*
@tejonBiker
@tejonBiker 4 ай бұрын
Crowdstrike switched to MaS: Malware as Service. Pretty wild that a kernel-level software from a cybersecurity company deploy bad software
@Dead_Goat
@Dead_Goat 4 ай бұрын
not really that wild. This is exactly why i have been fighting against using this crap. It does not increase security in anyway.
@rythem2257
@rythem2257 4 ай бұрын
@@Dead_Goat If it didnt increase security, people wouldnt use. Who are you fooling lol?
@PvtAnonymous
@PvtAnonymous 4 ай бұрын
@@rythem2257do you understand the concept of snake oil? That's exactly it.
@jorper2526
@jorper2526 4 ай бұрын
@@PvtAnonymous Explain then. I'm sure you have TONS of great information on this subject, and not just some "hurr durr linux" type of reply.
@PvtAnonymous
@PvtAnonymous 4 ай бұрын
@@jorper2526 what is there to explain? Just with any AV you introduce a single point of failure with extensive privileges into your OS or even your kernel. As seen in the last few days, there seems to be a lack of testing on Crowdstrikes end which resulted in - again - a single point of failure. Threat actors could just as well find a gullible employee or even infiltrate the whole company and introduce malicious code that could take over all of the machines and do whatever they want with them, basically making Falcon into a rootkit. This false sense of security is, you guessed it, snake oil.
@XerrolAvengerII
@XerrolAvengerII 4 ай бұрын
I've been looking forward to you making a video about this since I saw the first Australian news reports this morning! Thanks for taking time out of your trip to film this!
@pixelshocker7775
@pixelshocker7775 4 ай бұрын
Appreciate getting news from a source that actually knows what they're talking about. Props to LLL for taking the time to make a video while on vacation, and hopefully nothing equally newsworthy happens so he can just relax...
@manualdidact
@manualdidact 4 ай бұрын
Fireship for the snarky summary, Theo for the details, LLL for the disassembly. I missed out on all of this today, but I feel caught up now.
@henson2k
@henson2k 4 ай бұрын
Security nuts make things worse
@NarrowShouldersOpenMind
@NarrowShouldersOpenMind 4 ай бұрын
@@henson2k Nassim Taleb will have a good take on this.
@ziprock
@ziprock 4 ай бұрын
cool, appreciate the technical take on the issue. This will be a perfect example of what not to do.
@BoosterShot1010
@BoosterShot1010 4 ай бұрын
An intern writing tests with ChatGPT!
@Salted_Potato
@Salted_Potato 4 ай бұрын
For CrowdStrike outage, I went to mainstream media for human-readable explanation, I go to Fireship for system level explanation and now blessed with low-level-learning for the autopsy :D
@xmlthegreat
@xmlthegreat 4 ай бұрын
I remember seeing an article about how CrowdStrike's CEO regretted not firing people earlier... In 2020. I guess he's reaping now eh?
@Brahvim
@Brahvim 4 ай бұрын
So... the CEO might regret _regretting_ the thought of _firing people_ who may now provide support to customers? ...Or is it that he would've found that firing these guys early - the ones who couldn't deploy this crashing update correctly, as beneficial?
@renato360a
@renato360a 4 ай бұрын
yeah, I don't know how to read this. Should he have fired them earlier, or should he have kept them?
@mudi2000a
@mudi2000a 4 ай бұрын
I guess he fired them later. I mean at the end of the day even someone who was fired could have planted this in a way that it looks just like a technical f-up.
@xmlthegreat
@xmlthegreat 4 ай бұрын
You people have misunderstood the context.. the CEO laid off a bunch of people around 2020 and boasted how he probably should have done this earlier. Like Microsoft laying off most of their Windows QA teams so that they could use customers as Beta testers; when you reduce your employees you reduce the amount of slack the remaining people have to pick up on problems and head them off early. So a piece of code ends up in prod without at least 1 person catching a fatal bug. And some poor schmuck or team of schmucks who had to meet some kind of performance metric deadline push code on Friday that should have been tested more than just a few times on CrowdStrike's internal systems.
@MaffeyZilog
@MaffeyZilog 4 ай бұрын
That was the reason for kicking Kaspersky out! They wanted one for a while and they got it even though every other AV vendor has more privileged access to your computer than Kapserky did!
@norbert.kiszka
@norbert.kiszka 4 ай бұрын
In Linux module file with all nulls will not crash the kernel because Linux kernel makes multiple checks on a module file. Also modules are single files (sometimes module can request another module but it will be loaded in the same way) instead of multiple files. Couple months ago I tried to read that module loading code, but its poorly documented to be easily readable (if somebody was working with this code for long time it will be much easier).
@vilian9185
@vilian9185 4 ай бұрын
and tobe fair running crowdstrike on linux isn't needed, now with windows...
@norbert.kiszka
@norbert.kiszka 4 ай бұрын
@@vilian9185 only updates, but that can be done automatically in most cases (sometimes it will reboot some services or You have to reboot system manually when kernel or libc has security patches).
@Turalcar
@Turalcar 4 ай бұрын
@@vilian9185 but possible. Crowdstrike pushed a buggy debian update few months ago to a similar result (but fewer affected users).
@Dead_Goat
@Dead_Goat 4 ай бұрын
@@vilian9185 you do understand that linux is very hackable and much more likely to actually need something like crowdstrike than a windows system right?
@vilian9185
@vilian9185 4 ай бұрын
@@Dead_Goat no?, lmao wtf you're talking about
@martin.1976
@martin.1976 4 ай бұрын
First of all, thank you so very much for covering this during your vacations! I didn't even know this company existed until this morning, and quickly realized I'd only possibly get an explanation as to what happened from somebody like you, who knows about this stuff. One important lesson we need to learn from this is to ensue to do transaction-based updates and to ensure integrity of each update with a cryptographic signature. They have likely performed intensive in-house testing of that update - but just didn’t account for the possibility of the update being corrupted / tampered with in transit. However, if also brings up the question of whether we can really exclude the possibility of this being a dedicated cyber-attack that quickly! Because, to my knowledge, we don't know yet how exactly it could happen that the version of that update that was installed on these billions of devices came up all zeroes. Surely, it shouldn't have happened - and likely wouldn't have if they used even the most basic CRC approach to verity integrity. In that regard, the blame is by that company alone! However, I wouldn't necessarily exclude the possibility that some bad actor knew about this vulnerability and exploited it.
@Iceman259
@Iceman259 4 ай бұрын
I, for one, am shocked that the company called "CrowdStrike" which thinks it's a great idea to advertise on race cars would ever do something so ill-advised.
@manoflead643
@manoflead643 4 ай бұрын
That's memorable! Good advertising, honestly.
@Maxler5795
@Maxler5795 4 ай бұрын
Fun fact: this happenee the DAY evolution championship happened, which means a lot of competitors just... Couldn't make it. Wasting thousands of dolars per competitor in the process
@eljuano28
@eljuano28 4 ай бұрын
Blistex Inc is proud to announce they've teamed up with Linux to solve the blue screen of death. The "Tucks Medicated Pads" you already know and trust have been rebranded "Tux Crowdstrike Relief Pads" for Microsoft Windows users to relieve that burning sensation.
@MelJandric
@MelJandric 4 ай бұрын
Finally somebody explained it. Thank you. So far every other statement about this incident was just hot air.
@boy_deploy
@boy_deploy 4 ай бұрын
"Crowd Strike" literally 😂
@johngrant5881
@johngrant5881 4 ай бұрын
@@boy_deploy alex jones predicted this
@eitantal726
@eitantal726 4 ай бұрын
maybe this will change the mentality of pushing pointless updates too often
@erickhar
@erickhar 4 ай бұрын
I love your channel. so cool that you took a vacation from your vacation for this :)
@bjornjohnson9753
@bjornjohnson9753 4 ай бұрын
This is the video I’ve been looking for! Thanks for the succinct explanation!
@mihainita5325
@mihainita5325 4 ай бұрын
What puzzles me is why deleting the file solves the problem. There should be no difference between the missing file and zeroed-out-file. An all-zero driver file it's missing all the standard exe structures. Has no header, no import, export, relocation tables. And no digital signature. So Windows should refuse to load such a "driver". Which would technically be the same as missing file. Who null pointer for zeroed-out-file but not for missing file? Can't load is can't load, no? I really suspect something more is going on here...
@MathewBoorman
@MathewBoorman 4 ай бұрын
I am wondering that, might have got signed as nulls. CICD failure
@mihainita5325
@mihainita5325 4 ай бұрын
@@MathewBoorman yes, but the signature should also be part of the file. So at least that would not be zeroes. Same for the exe header. If it is all zeroes it would not load. Unless the description as zeroed-out-file is not 100% accurate and the header is present, the signature is present and correct, etc. Which then raises the question what kind of processes do they have to create such a file, signing it, and push it out with no testing. This is not something that "it works on my machine" :-) Some big red flags about their processes...
@YouPlague
@YouPlague 4 ай бұрын
Because if the file is missing opening it fails with a proper error. Once it opens the contents is trusted to be executed on the cpu.
@tma2001
@tma2001 4 ай бұрын
the clownstrike background driver that listens for updates should of done at least a hash/crc etc check of the updated file before initiating a reboot. That's what I don't understand.
@juliavixen176
@juliavixen176 4 ай бұрын
I use ZFS to store all my data, because I do *not* trust hardware. I always generate and verify MD5 hashes of my files and I have found so many _silent_ data corruption faults when a hard drive will randomly flip a single bit, or return 512 null bytes... with no hardware errors being detected/reported. SMART self-check says everything's ok; Linux device driver says everything is ok; etc. I mention this, because I have been given NTFS drives by other people, and upon checking the files, there will unexpectedly be 4096 null bytes where I was expecting data. (And I had a second copy of the file for comparison.) There were "no errors" reported at the hardware, filesystem, or OS levels. A file was silently replaced with a bunch of nulls... and I immediately wondered if Crowdstrike's build system is using NTFS.
@Vespyro
@Vespyro 4 ай бұрын
Within minutes of find out out about the situation I was already so excited for your breakdown, thank you!
@山田ちゃん
@山田ちゃん 4 ай бұрын
It's ridiculous that they can't even save a single boolean or something to disk and and clear it when the OS booted, this so next time your driver loads it can check for the boolean to be true and just not load (because you can safely assume the PC crashed), and while in the OS your non-driver part of the code could check for updates...
@az-kalaak6215
@az-kalaak6215 4 ай бұрын
or even better: do it linux style? and don't take down the entire system if there is a recoverable error at boot
@somenameidk5278
@somenameidk5278 4 ай бұрын
​@@az-kalaak6215I'm pretty sure a corrupted kernel module like this would crash Linux as well.
@DamonWakefield
@DamonWakefield 4 ай бұрын
Thanks for this outstanding code-level explanation!
@ok-alarm
@ok-alarm 4 ай бұрын
"should have used rust.. " 😂😂
@mushroomcrepes
@mushroomcrepes 4 ай бұрын
these rust ads are getting crazy
@lythox-bdd
@lythox-bdd 4 ай бұрын
@@mushroomcrepes damn best rust ad ever 🤣🤣
@hwstar9416
@hwstar9416 3 ай бұрын
wouldn't have helped in this case
@DrKaoliN
@DrKaoliN 4 ай бұрын
This has been the clearest explanation I've seen regarding what exactly caused the BSoD.
@delayed_control
@delayed_control 4 ай бұрын
tfw title starts with lowercase "lol"
@theskelet4r
@theskelet4r 4 ай бұрын
Love the commitment, thank you for taking time out of your vacation to breakdown this incident for us. No go have some fun and Relax!
@typingcat
@typingcat 4 ай бұрын
Wait, how could this happen. Don't they digitally-sign driver files? If so, an all-null file should not pass the signature verification, and how did Windows even load the driver? Windows rejects unsigned drivers, doesn't it? This makes no sense.
@iljaseklervl
@iljaseklervl 4 ай бұрын
According to the video, it wasn't Windows. It was CrowdStrike kernel mode driver loading a submodule without any NULL checks and error handling.
@typingcat
@typingcat 4 ай бұрын
@@iljaseklervl That is dumb. How could a security application do not check the integrity of its module files? This is not just a simple distribution mistake, but a fundamental amateurish dumb coding.
@nigh7swimming
@nigh7swimming 4 ай бұрын
Agree, looks like the gist of the issue to me. Mistakes happen, but this is pure negligence in lack of any sanity check verification.
@john6372
@john6372 4 ай бұрын
if there are no checks.. sounds like a nice backdoor waiting to be exploited.
@NYYstateofmind
@NYYstateofmind 4 ай бұрын
@@john6372sounds like it just was
@austinreilly1067
@austinreilly1067 4 ай бұрын
You never fail to deliver content on current issues and I was absolutely waiting for you to talk on this. Same Day! Crushing it dude, thank you for being you
@codycallaway9057
@codycallaway9057 4 ай бұрын
I can feel a disturbance, as if millions of crowdstrike memes are being made
@MichaelGrundler
@MichaelGrundler 4 ай бұрын
This is the first video I've seen that goes into the technical details of what exactly went wrong with the update and why it caused all those blue screens. 👍
@agmnii
@agmnii 4 ай бұрын
Rip the 5 hours of production data I lost at work
@kerplop
@kerplop 4 ай бұрын
I work in a medical supply warehouse, and we couldn't use our computers at all until after noon, when they finally got one of the shipping computers up and running. We shipped out as many orders as we got, but I'm sure there are plenty more floating out in cyberspace... It could be a difficult Monday.
@Nossody
@Nossody 4 ай бұрын
why is counterstrike on the macys computer?
@chrisaustin9949
@chrisaustin9949 4 ай бұрын
So here's a question, why does any company allow Microsoft or Crowdstrike to push programs onto their production system, ever? Back when I was in IT we would never let anyone do that. We would take the software ourselves and try it on a test system first before putting it into production.
@urizezucm256
@urizezucm256 4 ай бұрын
is friday and the computer knows it
@阮榮強
@阮榮強 4 ай бұрын
Yup. The first thing I thought of after reading an article about the outage was that Riot Vanguard is implemented in practically the same way. That anti-cheat made me quit the game. Most of the community called us paranoid or just didn't care. Now this happens. I know we were right but it certainly feels good for it to be proven.
@gameboardgames
@gameboardgames 4 ай бұрын
So glad I'm making games today and not doing my usual IT gig.
if you view this image, YOU GET HACKED.
8:40
Low Level
Рет қаралды 389 М.
DoubleSpeak, How to Lie without Lying
16:15
What I've Learned
Рет қаралды 11 МЛН
Don't underestimate anyone
00:47
奇軒Tricking
Рет қаралды 16 МЛН
Каха и лужа  #непосредственнокаха
00:15
You Pay For It, We Own It - Sony’s $7.9B Lawsuit
13:28
Logically Answered
Рет қаралды 394 М.
Interviewing at Google
16:31
Philipp
Рет қаралды 731
how is this hacking tool legal?
11:42
Low Level
Рет қаралды 407 М.
You're Being Lied To About Ozempic | Truth Complex | Business Insider
21:05
this new Linux feature makes hacking IMPOSSIBLE
11:08
Low Level
Рет қаралды 484 М.
new critical linux exploit has been hiding for 10 years.
9:32
Low Level
Рет қаралды 123 М.
this vulnerability shouldn’t even exist
14:33
Low Level
Рет қаралды 221 М.
They keep trying to backdoor Open Source
7:19
Eric Parker
Рет қаралды 179 М.