The CrowdStrike Problem Isn’t A Simple Fix…

[@t3dotgg]

Sorry about the frame rate issues, CrowdStrike took down my main recording rig and I had to do this on my Mac :(

[@t3dotgg]

TO BE CLEAR THIS IS A JOKE. My recording rig does run windows though… 🙃🙃🙃

[@timk9847]

BS First only Enterprise PCs and Servers were affected unless someone bought Crowdstrike. Second, only a fraction of devices with Crowdstrike were affected. I work for a major nonprofit Hospital and about 25% of our servers were affected. I spent 11 hours manually remediating servers because in a large enterprise environment, that can be a large number. But get the freaking facts right. The reason this was so impactful is that it was large corporations affected. And the fix was a very manual process since Microsoft had a "feature" that put everything into "recovery mode" after two failed boots.

[@Duckless137]

@@timk9847What part of “to be clear this is a joke” did you not get? 💀

[@joshuacheung6518]

The whole thing

[@rdvansloten]

@@timk9847 everyone who installed this spyware willingly had it coming

[@samcalder6946]

This is the best named company in history. This is the exact same outcome as if the entire crowd went on strike.

[@morsemurraidh1314]

The entire _IT Crowd_ ...? There was that episode where they borrowed (then dropped) *the internet.*

[@notme8232]

@@morsemurraidh1314 Bet the CrowdStrike CEO found some "irregularities in the pension fund" today

[@nardu]

Would make a good movie title too.

[@ZakiWasik]

Exactly! The company name sounds like the name of an exploit!

[@essennagerry]

I interpreted it as the company struck the crowds 😅

[@mrtnsnp]

To everyone cleaning up this mess: my condolences, may your weekend rest in peace.

[@YourLocalAltAccount]

Their weekends will probably be full of people complaining and PC boot looping

[@dead-claudia]

it's log4j all over again except it's actually crashing everything instead of hypothetically being possible to turn into a crash

[@Warwck24]

😂😂😂😂

[@a_lethe_ion]

well at least if youre self employed (so you can demand a fuck this is my weekend surcharge)or live in any country with workers protections (aka not the usa or like india) you gonna get at leas weekend pays and overtime pay

[@C.A._Old]

Thank god that thing never happens to me because Thanks by my PC never get windows update.

[@kameronbrooks2372]

When I got multiple calls at 2AM I knew this was going to go down as one of the worst days in recent IT history.

[@ioannischristovasilis3279]

Aouchh

[@C.A._Old]

Thank god that thing never happens to me because Thanks by my PC never get windows update.

[@goaliedude32]

​@@C.A._Oldthis never would have happened to you because there's absolutely no reason for you to have crowdstrike

[@TimelapsingGames]

@@goaliedude32 doesn’t windows as a company have crowdstrike?

[@goaliedude32]

@@TimelapsingGames i dont know and that wouldn't have mattered for this thing that happened

[@PatNeedhamUSA]

The largest disruption in human history caused by a missing try/catch block

[@chinesesparrows]

Seriously windows acts like it has nothing to do with them but any driver issue can result in BSOD bootloop is ridiculous

[@SahilP2648]

@@chinesesparrows we would be in 2142 with probably Nuclear Fusion, a Mars and Moon colony, new space stations, BFR, multiple cures for cancer, synthetic lifeforms, probably alien contact, and we would still be running 120+ years old windows code 😕

[@omri9325]

Do you mean the windows code that loads drivers? What do you think the catch block should when it reads a file full of empty bytes? ignoring the file would mean booting up without the security features the system was supposed to have, the fault is mostly at crowdStrike

[@chinesesparrows]

@@omri9325 i agree with that, just that pretty much any 3rd party driver can cause BSOD bootloop is a massive vulnerability. I dont know maybe MS could add a group policy that adds a standard flow where windows creates "driver backup point" before any new driver updates, if after driver update repeated BSOD revert to driver backup point and contain the problem driver with alert and log.

[@azmah1999]

@@chinesesparrows No? It's a driver with kernel access. This makes the driver very powerful but also makes crashes easier, due to the fact that the OS cannot babysit it. I'm pretty sure you can cause a kernel panic on linux by writing a bad driver as well

[@gorak9000]

Crowdstrike is ransomware, they just have a different payment plan. You pay up front for the privilege of being ransomwared at some unknown point in the future. Turns out the unknown point in the future was today! Surprise!

[@gorak9000]

The other problem is not just bitlocker, but if your company locked the microsoft account so you can only sign in from company devices to get the recovery key, but all your company devices are hosed. I can sign into the account from my personal linux box, but it just says it's restricted, and I can't do anything, or get the recovery key. I was on hold with IT support for hours today, and at one point, the phone system hung up on everyone, and calling back the number went to a busy signal for the next hour. I spent 3 more hours on hold with IT, and they either answered when I went to get something to eat, or hung up on me again. What a cluster F

[@growtocycle6992]

It's how McAfee works on all the PCs of retired folks I know, who installed this "shiny, free antivirus software." 🤦

[@Walter_]

LOL call it reversalware

[@LutherDePapier]

This is facts.

[@jacquelinel1618]

And don’t forget that Crowdstrike was responsible for the 2016 story that Russia hacked the DNC in order to get dirt on Hillary and favor Trump. Which was a lie.

They failed to do a smoketest of their agent after build but before deploying it worldwide. Sounds like their software and update development process is just really not up to professional software engineering standards. At Meta, we had to have other engineers, sometimes multiple, review diffs before they would be accepted. And then there were multiple layers of CI/CD testing before exponential deployment with canary testing. You don't just push new code to all the machines all at once, because it's way too dangerous.

[@grastant6819]

And deployment of system files, much less kernel level files should have a hash/checksum too, no?

[@Montoyax]

And still they fucked up all their systems for a DNS error

[@mjwchapman]

things go wrong even after full in-house testing. that's why you have canary testing and phased roll outs. the ceo comes across as totally disingenuous and is ultimately the reason for the company's poor practices.

[@momchilandonov]

It's weird how they got to around 80 billion $ market cap with this incompetence!

[@momchilandonov]

@@grastant6819 this wasn't a kernel level driver/file. The .sys is misleading.

[@AndrewEddie]

Definitely a "zero" day problem. The only things saving CrowdStrike from a class action is most law firms are Windows users too :)

[@DanielSmith-lv5ed]

Crowdstrike says "hey, they didnt respond like this during covid/bitcoin. Which was also highjacked, but nobody said anything and everyone joined in for the fun. Lmao It may not have even been on purpose

[@Jimothy-723]

@@DanielSmith-lv5eddoesnt matter. this level of negligence is actualy criminal. somone will go to prison over this.

[@lashlarue7924]

😂😂

[@CHURCHISAWESUM]

Apparently an idiot dereferencing a null pointer and another senior idiot pushing it to production past code review is now a “zero day” No it’s just a really obvious bug. There was no hacker here unless the bad code was intentionally put there by the employees in order to sabotage the company. So if there’s any hack, it’s internal.

[@monad_tcp]

​@@Jimothy-723 what , criminal negligence , last week there was that event and the person in command got no punishment nor lose their job for that absurd amount of negligence.

[@thezoidmaster]

the fact that one company can take everything down like this is scary, one bad actor and this could've been a mass malware attack instead of a simple driver error

[@blakenolingberg1556]

Nope. Crowdstrike had this access because they were trusted. Malware doesn't get to waltz this close to the kernel as easily.

[@monad_tcp]

​@@blakenolingberg1556 that was the mistake. Don't trust anything running on the ring0 that's third party, except drivers made by the vendor. No toy software or rootkit is allowed.

[@CRhetorix]

Capitalism market innovation... this is freedom of choice... The truth is corporate American hates competition, and capitalism always produces monopoly.

[@dead-claudia]

@@monad_tcplotta people run this due to legal obligation, and crowdstrike has historically been far better than their competition (let that sink in)

[@Warwck24]

No - Microsoft have a security feature if bad data updates it's designed to crash. Falcon - it's been busy - on this direction for a while I'd guess

[@kwilt]

I literally spent my entire friday manually fixing computers and explaining to people at remote locations how to fix their computers. Our entire IT department became helpdesk because of this update. You don't know the pain of explaining to a non-tech person over the phone - how to make a bootable USB, boot to it, and then enter their bitlocker recovery key so they can delete a file via command prompt until you've done it personally. I got to do that dozens of times on Friday and theres going to be lots more of this for the foreseeable future... I cannot express how much this sucks to fix even though its a relatively simple fix. It just can't be automated and it's horrible for that reason.

[@erroneum]

I mean, Windows might be the least secure how most people use it, but there's another huge facet to why it's the target of randsomware: it's absolutely dominates the end-user/workstation market, especially when you are wagering the victim can't just restore from a backup and ignore you.

[@Texas3Percenter]

It's purposely written like swiss cheese, full of back doors and vulnerabilities so they and intelligence agencies can access your computer any time they want. Linux is not. So, Linux users don't have to worry about viruses or malware and don't have to put middle-man software between them and their machines to protect them from their malware/spyware OS. The only thing I use on my Linux servers is a firewall and Fail2Ban to prevent brute force pwd cracking.

[@unaquetzadilla]

​@@Texas3Percenter This specific issue is not exclusive to Windows. This is an issue of operating systems architecture and how drivers are able to run in Kernel mode. Falcon have Kernel driver for Linux, Windows and macOS. The IT departments of companies are asking the OS to load Falcon driver an allow it to run in Kernel mode, allowing it to watch for user data but also enabling it get OTA updates and not able to choose of get or not the update. Same bad driver could be for Linux or macOS.

[@J-wm4ss]

​@@unaquetzadillaalso, macOS still needs antivirus/endpoint protection. It just works a bit different and the audience of people who get MacBooks at work is probably more technical

[@markcruise]

Just what I was thinking. The reason Windows is targeted more is because it’s sitting on 73% of desktops. The PC Security channel showed that malware vendors absolutely have Linux versions of their tools. It is not immune.

[@marcus141]

​@@unaquetzadilla​What you said is partly true. In my previous role, I deployed crowdstrike for a major broadcaster and one common misconception in all of this, is that crowdstrike can push updates to customer endpoints without their knowledge or consent. It's simply not true. Endpoint management is handled centrally by IT admin and we can choose if we want to use the latest Falcon sensor version or not. You can of course configure crowdstrike to auto update the sensors but that would be ludacris.

[@MasterOfMisc]

The problem with the bootable USB thing is that a lot of corporate devices block booting from USB by default, which means the IT Team would have to tell the end user the BIOS password to get into the BIOS to change the boot order to enable booting from USB. Its a total nightmare!

[@BryanK-y5y]

Would the block not be a gpo ? So won't apply to a local admin or admin profile

[@Micloren]

Most companies I’ve been around weren’t secure enough to lock the BIOS.

[@Micloren]

@@BryanK-y5yBIOS loads before the Windows OS.

[@MrThebigcheese75]

Yeah, as a former IT support bod in logistics, taking a user through the steps will be painful and in some cases practically impossible. Can you get into safe mode please, power on the computer, wait eight seconds, hold power to turn off. Repeat again. Err, it's not coming on again. Oh, it's is now, oh god blue screen again. We need to start again. Warehouse bod, eff this you'll have to come over and do it. That's before you even get to bit locker and talking though command line. Lots of journeys will be happening this weekend.

[@BoStark]

Just use PXE.

[@michaelgebauer5235]

Sending out a sys file filled with nulls looks to me like sabotage

[@dead-claudia]

or a failure of automation sending out nulls is something that 100% should've been detected before it got sent out, tho.

[@D0XXX4]

The file wasn’t full of nulls.

[@celiem4352]

YES IT "WAS SABOTAGE" YOU ARE SPOT ON. PEOPLE ARE SO EASILY FOOLED, IT'S INCREDIBLE. I KNEW IT IN MY GUT. NO ONE CAN CONVINCE ME THAT IT WASN'T SABOTAGE. OH, EXCUSE MY CAPS PLEASE. I CAN'T SEE THE SMALL LETTERS WELL ENOUGH TO USE THEM. GOD BLESS YOU FOR THE TRUTH

[@einargs]

I mean, all the malware also targets windows because that's the big user facing desktop OS.

[@itsmefrancois6825]

That's the biggest reason. I bet he can't even name 3 reasons why Windows is much more vulnerable than the other OS

[@haroldcruz8550]

Linux runs most of the world's servers though. Cyber criminals alwyas go for the path of least resistance and that's what Microsoft provides.

[@torquetheprisoner]

crowdstrike did the same thing to mac and Linux as well

[@benheidemann3836]

@@torquetheprisoner when was this? Can you link to news articles?

[@J-wm4ss]

​@@benheidemann3836you can google "red hat crowdstrike", it states that the driver works in kernel mode and user mode

[@jasonfreeman8022]

Where in the hell is the testing cluster they should have deployed to first? CrowdStrike should deploy their Falcon updates to all their own machines and if they don’t BSOD after a week THEN release to the entire galactic empire.

[@Betadesk]

Yeah, I was gonna say don't they do gradual deployment? To like 1% of machines first, then 10%, 25% etc. or at least some A/B testing damn

[@piquat1]

Lets accept for a second that they did that, because they probably did, they've been doing this for a while. They probably did NOT send out a null file. So somewhere between them releasing it and the end users getting it, it got nulled out. That's where the problem was. Azure went down right before all this happened...

[@AZaqZaqProduction]

This is tough because as an antivirus you want updates deployed as quickly as possible. If some new exploit comes out you wouldn't want your customers to be vulnerable to it for over a week.

[@jasonfreeman8022]

If they tested, then they either didn’t test what they were deploying or they didn’t deploy what they tested. This is a basic control problem. I have had to point out to management numerous times that whatever cockamamie plan they have for maintaining a claim that the product was tested, that they weren’t testing what they were deploying. That internal process needs to be seriously scrutinized.

[@mjwchapman]

@@piquat1 i cannot believe they were smart enough to do a phased roll out, but then neglect a simple hash check of the deploy-able. the evidence suggests they did neither.

[@AvanaVana]

I literally just turned down an offer from Crowdstrike two weeks ago in favor of another job offer…it was a tough decision to make at the time but now it’s definitely looking like I made the right decision! 😬

[@collinsfondong2319]

😂

[@ulysses-pact]

dodged a bullet

[@ValZarGaming]

"Windows is the only OS that is insecure enough to have problems like this" Let me tell you why that's bullshit - Crowdstrike did this to our production Linux fleet back on April 19th.

[@theoryianabsolute8777]

Don't say something like that that's unpresedented

[@JustSomeGuyCG4]

Biased video for sure, but any opportunity to bash Windows they take it!

[@ValZarGaming]

@@theoryianabsolute8777 I do not believe that word means what you think it means. This has as of now also been confirmed by other news sources.

[@lcarsos]

uh, no. Crowdstrike on mac is just as deep, and slows down my work mac just as much.

[@ivocass4332]

Shhh, man. Macs are suposed to be fancy.

[@lcarsos]

@@ivocass4332 it's a very slow, hot, but pretty piece of aluminum after corporate IT gets to it. XD

[@karmatraining]

Oh that suuuuuuuuucks

[@petargolubovic5300]

But it difference is that is doesn't run on kernel level. Mac and Linux have fixed this particular problem long ago

[@brandonn.1275]

​@@petargolubovic5300 Agreed Mac booted Anti-,virus software from the kernel after creating an endpoint security API for them to use and Linux has ebpf hooks and bpf programs to screen for AV to screen for potentially malicious activity with a guarantee that they can't crash the kernel (bpf programs have strict security/stability guarantees while being non-turing complete)

[@WilkinsonX]

Our company installed CS on thousands of Windows clients recently. A few weeks ago they uninstalled it because it was causing massive system performance issues. Giant bullet dodged. On the Windows side, it's just nuts that any driver causing continual stop errors is not auto disabled /quarantined by the OS.

[@dead-claudia]

are you sure you aren't neo? bc that's some wicked bullet dodging ability

[@tiredguy709]

The driver affects the boot process which is why it wouldn’t fail until a restart. Cant auto detect a failing driver before that driver gets used by the system.

[@bltzcstrnx]

Kernel drivers will cause severe problems on any OS, not just Windows. Just search RHEL CrowdStrike if you don't believe it.

[@Kas-tle]

Most drivers do not, but generally AVs mark theirs as a boot driver, so the system cannot boot if it is failing. So it's not just "any driver" as you state.

[@paavobergmann4920]

The fact it has to be implemented as a driver at all is nuts.

[@MultiMojo]

Root cause analysis - 1) Using Windows for mission critical work in 2024 2) Terrible code that somehow made it past code review 3) Build system that corrupts files 4) No validation checks prior to rollouts 5) Rollouts to the entire install base rather than a staged rollout

[@gorak9000]

I've never anywhere seen a build system that produces a file of the right size, but filled with all 0's - how does that even happen?

[@lashlarue7924]

It's WILD to me that so many big companies have built critical business infrastructure around Windows. I do it for my little piddly business but I'm aware of the shortcomings!

[@JacobProbst-z2l]

@@Jimothy-723 1. DEI isn't a word, it's an acronym. 2. These kind of issues occurred long before 'DEI' was a thing, so how do you explain those issues if this is obviously because of DEI? I'll wait.

[@timk9847]

1-4 are BS, but 5 is on point

[@qwaszx2]

@@JacobProbst-z2l DEI is just an acronym for affirmative action. It's always been a thing in the internet age. Sorry that you're wrong. It's more prevalent today since there are few qualified white males willing to work for crap wages.

[@Atomicjtx]

I was dealing with this today as an IT tech. Oh boy, What a joke. Took around 20 minutes on average per person affected. Since hundreds of devices were affected it made for a long day... :(

[@joeypritchard6320]

😮😮😮😮

[@theairaccumulator7144]

20 minutes? You could've set up a few rubber ducky USBs to automatically run those commands.

[@sirklatt]

@@theairaccumulator7144 He gets paid by the hour

[@koyotecow7102]

They don't want to apologize cuz they don't want to admit fault and open them up to lawsuits.

[@momchilandonov]

Their asses are already opened up to lawsuits big time! 3 billion in cash $ watch it evaporate! Their customers are now PUBLIC DATA which is a huge liability for their future cash flows too.

[@debbyolivier5122]

too late for that! this company shoudnt exist in the future!

[@celiem4352]

THEY'D NOT BE ABLE TO "PAY FOR ALL THE QUATRILLIONS OF DOLLARS LOST😂

[@stephenjames2951]

hey grandma, all you have to do is start up in safe mode, grandma? Grandma?

[@kinamonsterrawr]

Honestly, when I was an i.t. call center tech, the older people could often be counted on to listen to my instructions and not go off script. It depended on the person of course, but I was often able to wrangle an older person to listen. 😅

[@firstprib7742]

Grandma just rebooted into safe mode

[@andrewreed1329]

grandmas on mint

[@torquetheprisoner]

grandma:beeeeeeeseeeseeeeeeeeeeeee

[@kattmilk]

Please be patient: Grandma is busy beta testing in production. 😅👵🏾

[@redeuxx_]

Linux versions of Falcon also hook into the kernel. Talking about how this fuck is somehow because it is Windows is disingenuous. Why can't we just blame Crowdstrike and just Crowdstrike instead of bringing up and blaming Microsoft?

[@kipoyedcl]

i wholly agree with you, this guy just wanted to hate just because its windows, i also don't like windows myself but its pretty disingenuous to blame Windows when its not their fault.

[@Abaddon231]

He didn't blame windows , he said they are the most insecure OS on the market , and that's why software like Falcon exists

[@redeuxx_]

@@Abaddon231 falcon also exists on Linux and Mac. Is Falcon there because of Windows? Falcon exists because there are malicious actors, not because Windows exists. Then he mentions kernel drivers without acknowledging that this also happens on Linux and the fact that if you want a solution like Falcon, they should hook into the kernel, because that is how they can most effectively do their job. Or why bother having and EDR at all. He would have been able to effectively convey his message instead he went all mainstream media ranting about Windows. This shit isn't his forte, but he still has a take on it. Low Level Learning has a much better and nuanced take on this.

[@brandonn.1275]

Falcon on Linux uses ebpf hooks which can't crash the kernel (they have extraordinarily strict guarantees, restrictions, and limitations that would prevent an ebpf program from doing that, they aren't even turing complete) Apple on the other hand has an API for AVs to do their job and doesn't permit them from installing a driver.

[@redeuxx_]

​@@brandonn.1275 Falcon on Linux in User Mode uses eBPF. There are still many systems that use Kernel mode. Although Falcon is transitioning to User Mode, Kernel Mode is still officially the default mode for Falcon on Linux. Per official CrowdStrike docs. Many of my Linux systems still use Kernel Mode.

[@JonitoFischer]

Crowdstrike is the company that shuts down your computer when you're hacked, and they wont allow it to turn on until they check manually how bad is the hack...

[@D0XXX4]

We don’t shut it down, we hit a button labelled “network containment” which only allows it to communicate with the admin dashboard for forensic analysis. Your MSSP should be ringing you immediately if they have to use network containment

[@ZachFrank714]

There was an issue with CrowdStrike on Debian several months back, that caused the OS to not boot… This isn’t the first time CrowdStrike has massively broken an operating system

[@jhcato]

Let's not forget all the people who probably put their very important bit locker passwords... inside of their bit lockers.

[@tatumsh9]

Thats why they call IT. The amount of people who do not know how to log in to their work email on their phones so I dont have to read out their 69000 digit recovery key is incredible... and then hope to god that they wrote it down correctly or heard you correctly... and then hope to god that they enter it correctly so that 75 year old Dorothy doesn't have to be talked through booting into safe mode again BEFORE you have to talk her through deleting a file.

[@morsemurraidh1314]

So, there was this episode of _The Munsters_ that followed the same idea... They wound up blowing up a very expensive antique box to find a video cassette (and not a heap of treasure).

[@rlstrength]

There was a post on reddit about a huge org that has the bitlocker keys on a box with bitlocker and they don't know where the key for the centralized box is because the documentation is also behind bitlocker

[@monad_tcp]

I did that by mistake once, I stored by password database on the bit locker drive, the bit locker password was on the password database. Lucky I have a offline backup of the password database.

[@gorak9000]

@@tatumsh9 Or maybe the company shouldn't limit the microsoft account to only be accessed from company devices, when the only company device is hosed! Sure, don't allow access to everything, but they could allow access to the damn recovery key. Also if I have the company email setup on the phone, then it creates a huge nightmare in outlook making me authenticate on the phone every time to use outlook on the pc - I discovered by accident (when the company "mistakenly" removed email from everyone's phone) that not having it on the phone solved all the auth issues on the PC too, so I just left it off the phone. I don't need company email on my phone 24/7 - company email is only for during work time on the PC

[@aronjacobson5403]

loved the title "The day the world went blue"

[@LagowiecDev]

Video with that title is delisted

[@aronjacobson5403]

@@LagowiecDev yeah i know i just loved that title this one doesnt flow off the tounge like that one did

[@t3dotgg]

I loved that title and will likely change back to it. This one's performing way better though :(

[@aronjacobson5403]

@@t3dotgg :)

[@koto9x]

just make it ur description or a pinned comment

[@ellipsis...1986]

My favourite part of the disappearing air traffic example is that while they will occasionally get crippling downtime from their infrastructure, Southwest still running primarily Windows 3.1 with a sprinkling of Windows 95 here and there rather isolated them from the CrowdStrike issue.

[@bullpup1337]

surely that is a joke… surely…??

[@Ignisami]

​​@@bullpup1337 of course. No way southwest is running something as modern as windows 3.1 :p

[@dead-claudia]

@@bullpup1337🙂

[@mollusckscramp4124]

Once again Southwest comin in for the win

[@SirWickMusic]

@@Ignisami Actually, they are

[@joshuathomasbird]

its shockingly inept. the fact they did a rollout with no percentage based rollout and metrics on how the rollout was performing and no rollback plan is literally insane and then on top of that the fact their deploy pipeline distributed this with tests either not run or failed.... makes it seem like theres more to this story than just oh we just pushed some bad code. There are safeguards in code and policy that should prevent this.

[@joshuathomasbird]

also the fact it's literally *in* the kernel and not something like ebpf where it can have hooks in the kernel and have saftey guarantees about it not crashing the computer... thats microsoft fault for writing a steaming pile of bitrot masquerading as an operating system.

[@jeronimo196]

once you remotely brick a pc, the rollback becomes difficult...

[@joshuathomasbird]

@@jeronimo196 thats why its also done as a percentage rollout.

[@Winnetou17]

While I hate Microsoft about as much as Apple nowadays, your take on Windows in this specific case is totally wrong and uncalled for. Only Windows was affected because only that update was affected. Windows doesn't have anything worse than Mac or Linux here. Also Mac and Linux could've been just as much affected. That whole section is very cringe. Edit: I mean the section starting at 3:35 The rest of the video is a-ok

[@Whoami-b5c]

Yeah, this could’ve happen to any of the OSs especially in a corporate setting. That said, Apple actively discourages kernel extensions and have built alternatives in user mode.

[@jonnyso1]

In this particular case I *suspect* Linux would handle a faulty kernel module bettter, but I'm not sure. This is crowdstrikes fault for sure, but I wonder if the way Windows handles these kernel level drivers could be better though.

[@fallingintime]

I believe cloud strike has uploaded a faulty module not a while a go there was a red hat incident posted for it. But I guess it was not as widespread.

[@jonnyso1]

​@@fallingintime But was it as catastrophic as a unbootable BSOD ? Edit: Although it would probably be hard to compare unless it was at least a similar mistake.

[@fallingintime]

@@jonnyso1 kernel panic that required a kernel fix I believe. Probably wasn't widespread as it only affected a specific kernel version and kernel updates are not ota

[@temp50]

3:40 Not true. Falcon sensor is available for Mac and for Linux too. The real reason that it has happened on Windows only, because 1.: CrowdStrike seemingly made a mistake only in the Windows driver, 2.: Windows is waaaay more popular in businneses (both server and desktop side) than anything else.

[@AppleAlumDotBlogSpot]

@t3dotgg Your description of the Windows / Active Directory / BitLocker login process is inaccurate. The bitlocker key is not retrieved from AD or other remote DB when you auth, but rather from the device’s local TPM.

[@uzlonewolf]

I believe he is talking about the recovery key, which *is* retrieved from AD or another DB.

[@aryankothari4634]

its insane that crowdstrike didnt integration test the update, and even more insane that mission-critical infrastructure is OK with automatic OTA patches.

[@SirWickMusic]

How in the WORLD do you not test the system BEFORE you send it out. CRAZY!

[@ARandomUserOfThisWorld]

Lesson learned: use Linux (I use arch btw(I use arch btw))

[@dyto2287]

After using Linux for 10 years and Arch for 5 years I will say... use Mac instead. If you need something linux or windows specific you can spin up a vm with Parallels. And overall, mac laptops lasts long and have overall great performance & build quality. As for servers - linux is the only choice.

[@connerreimers6506]

What if I want to play Elden Ring ​@@dyto2287

[@lck0ut348]

@@dyto2287 That or, if you do want to use Linux, just use ubuntu.

looks like you use lisp btw too

[@quinndirks5653]

​@@dyto2287Until your display cable is too short and rips when you open it and apple won't cover it under warranty... Not to mention the throttling that occurs because they don't put fans in their laptops. Hope you didn't need performance...

[@JamieHicks154]

One point to make, (I am a Mac user so don’t come at me 😂) Microsoft has the biggest market share for desktop by a large margin so make sense for hackers to focus on them, not sayings it shouldn’t be more secure but also theywill get bigger focus from hackers just due to market share

[@haroldcruz8550]

Linux runs most of the world's servers though. If Linux is much more vulnerable than Windows it would make more sense to focus on Linux since you can have more control. The thing is Windows simply is a lot less secure and more prone to crashes like this.

[@DimkaTsv]

​@@haroldcruz8550 you know, it is much harder to trick user to install malware on server, than it is to make user to launch it on own PC. Which is, coincidentally, predominantly Windows.

[@bltzcstrnx]

​@@haroldcruz8550Linux flaws are invisible to the general public and end-user Linux enthusiasts. That said, there are many well known attacks on servers.

[@vivekbernard]

One thing though, using a kernel mode driver is not exclusive to crowdstrike. Many other AV/EDR systems use drivers as well. In fact a very similar thing happened with Symantec a while ago.

[@brandonn.1275]

At this point windows is going to need to boot Anti- virus software out of the kernel and provide an API for AVs to do their job instead of having them insert a driver into the kernel. This is what Mac did when they booted AV vendors from the kernel after publishing an endpoint security API for them to use. Linux has something similar in the form of ebpf hooks and bpf programs that can run in Kernelspace while being guaranteed to be unable to crash the kernel.

[@dead-claudia]

iirc even linux security software sometimes needs kernel mode drivers. stuff like cloudstrike can avoid needing a driver on linux bc they can just use ebpf tho. mac still needs a kernel driver.

[@JSmith73]

Yeah the affected CS update just happened to target a Windows named pipe vulnerability, so in this case only Windows was updated. So to just blanket blaming using Windows like OP did is a bit lazy. Their Apple and Linux customers just got lucky.

[@theairaccumulator7144]

@@JSmith73 macbrained webdevs can't see the world as it actually is. They're used to rewriting their entire app (which is just plumbing between AWS, databases and APIs) in the latest JS framework every 3 months. They don't realize that most legacy vendor software which is what the world actually runs on is a piece of crap and was written by a team of 10 contractors over a few months for a specific pentium windows xp machine in 2005 and has become a mountain of hacks and patchwork because management doesn't want to spend money on improving it.

[@brandonn.1275]

@@dead-claudia Mac doesn't allow AVs into the kernel anymore instead they published an API for them to use and won't allow AVs to install kernel extensions anymore. In fact kernel extensions have been deprecated for a while now and only userspace system extensions can be installed.

[@TallinuTV]

“Pretty much every PC in the world”… What? No. Nobody’s home computer would have CrowdStrike software. No Mac or Unix or Linux systems would be affected. The number of business computers running Windows with this software loaded is absolutely mind-boggling, though. I mean, we’re in seriously WTF territory. I hope people can get things straightened out quickly, especially for the more critical areas.

[@AdderoYuu]

I don't understand why everyone is so caught up over this "kernel level driver" thing - this is not built for consumer PC's. EDR solutions REQUIRE kernel level access to even be effective at catching as much malicious software as possible - it gives you such an upper hand and allows you to check and scan EVERYTHING on the machine. For a consumer non-business user device, this is super undesirable and would not be a good solution - but for a business that requires intense security to protect their data? At least for the moment, there is no other way. Those saying that Windows is the only OS unsecure enough to need this and jabbing at windows... Yeah. Just Windows things. I mean it's not like Linux and MacOS are perfectly secure, but the general consensus is those OS's are more resilient to viruses than windows. (Though it's not a bad thing to point out, that most malware is written for windows, because of how ubiquitous it is in industry.)

[@uzlonewolf]

No, they do not require kernel level access. In fact they are not allowed to have kernel access on Macs and even Linux is moving over to ePBF where a bad driver can't crash the system.

[@dead-claudia]

⁠​⁠@@uzlonewolfkernel drivers are allowed in macs. and linux kernel drivers are limited in a number of ways. notably, a kernel module is needed to monitor syscalls for processes you didn't spawn. and seccomp filters don't let you count anything, only filter.

[@AdderoYuu]

@@uzlonewolf Because of the way that Windows is built and the way that threats/malware currently operate, the only way you can hope to catch everything, at least right now with current technology, ideas, and software, is with kernel level access. Hopefully this changes, but as of right now it is what we have. I am only applying this to Windows however because, obviously, we've found alternate solutions for Mac and Linux and have not needed to do this.

[@miquelfire]

If you search hard enough, you'll find that there were two Linux Distros that got a bad update from CrowdStrike that resulted in the same issue. I think I read that it was cases of kernel panics in this case.

[@cheekoandtheman]

Crowdstrike painted the town BLUE !

[@WiseWeeabo]

this seems like one of those applications where you'd expect every pull requests to go through a "committee" such that you don't have some one-guy write a bug into the code..

[@nineflames2863]

Or malware. Seriously, if this could happen due to some stupid mistake, imagine how bad it would have been if an actual bad actor had social engineered their way into position to abuse the hell out of it.

[@WillDelish]

This is going to be a LONG weekend for some folks in tech

[@aug.jam.1]

Weeks sir... weeks

[@Texas3Percenter]

No one goes home til this is fixed!

[@haroldcruz8550]

I'm a glass half full type of guy. At least now companies know how important their IT are

[@cabpacedilla]

i guess this takes time because fix need to be done manually on every computer

[@aug.jam.1]

@@cabpacedilla yeap

[@BinaryReader]

"Pretty much every PC in the world just BSOD" - Incorrect, only PC's that ran CrowdStrike.

[@brentlidstone1982]

Every single time something really shitty like this happens.. almost without fail. EVERY SINGLE TIME: Look at the education of the CEO. George Kurtz: Degree in Accounting. (no formal science, tech, engineering, or computer education of any kind... I can see he claims he can program but as far as I can tell, he's never actually worked a job involving programming or science of any kind.) The fact that this company pushed a driver rollout to hundreds of millions of people SIMULTANEOUSLY without checking it worked first tells you EVERYTHING you need to know about how this dude runs this company. If he had any actual knowledge of computer systems he wouldn't have allowed that to happen. And yet he did. When will the world wake up and start to realize that shit like this always happens when you put business people in charge of technology they don't comprehend. Now I'm not saying George Kurtz knows nothing about programming, its completely fair to be self-taught. But as far as I can tell he never did anything science-related at his job in any capacity, and his claim to fame is that he co-wrote a book about computer hacking with some actual computer scientists back in the 90s. Ever since then everyone has treated him as though he himself is a computer scientist, even though he's not actually. And after a #$#@ up this extraordinarily bad, it seems it was wrong to believe he knew what he's doing. There's NO WAY a mistake this bad could have happened without his express knowledge and instruction that this is how they operate. Stop trusting bean counters with important technology.

[@ItsEverythingElse]

Not sure what is scarier, that CrowdStroke released a bad version or that so many companies just blindly went with it without testing and staging it first.

[@dead-claudia]

this was supposed to be more like a config or signature update. this is like pushing a bad signature file to windows defender and causing it to crash.

[@egria]

Airlines, banks etc. without testing updates in isolated environments is absurd yet reality. And most systems if done properly don't even need antivirus because they suppose to be nit connected to public networks. So this would be some management's push to order that software. Important systemd should be setup in a way that assumes that something goes wrong that means having staging environment. This incident shows massive tech incompetence either by itself or with push if higher management wanting to reduce cost or just fall on lies of vendors of how great everything would be if company trust them blindly.

[@peanut3438]

The update was automatic I think D:

[@jgndev]

Companies that run something like CrowdStrike often use BitLocker AND have take measures to block USB devices. You have to lock a Windows down way more to be ‘compliant’ for auditing

[@TheOtherNEO]

Friday morning at the office I was jokingly asked if I caused it. The day before in the company Town Hall I announced that I cancelled the CrowdStike contract and we have mostly removed Falcon from all devices. Only two left over machines had issues.

[@Micloren]

Curious, what was your reason for cancelling? Was it affecting productivity?

[@TheOtherNEO]

@@Micloren basic Falcon didn’t do much and got better data from the Checkpoint and Fortigate UTMs. Unless you shell out for the full SIEM, felt limited. Decided to up the network security instead and pay for a SOC/NOC service.

[@mageos98]

I find a lot of the take in this video to be misguided. Modern anti-malware software works by monitoring application behavior in addition to traditional known signature matching. The only way to get the level of access to protect at the level is through kernel modules (aka drivers). The falcon scanner for linux also has a kernel module for this reason. While there are a number of vulnerabilities in Windows, it is also the most used desktop OS. Securing desktops is a lot harder than securing a server because you have a user who may or may not do stupid stuff that you worry less about than a server. There are multiple reports of crowdstrike falcon causing linux kernel panics...they just are not as wide spread as windows.

[@MexMario]

CrowdStrike engineers: “Update is ready, let’s deploy it to the world Friday morning, and let’s test on production”

[@garydrago]

At first glance this is actually hilarious to see, but I feel so bad for the patients at hospitals affected by this. That's the worst part. Sometime like this will literally cost lives. Crazy

[@AstralPhnx]

Do note Crowdstrike has a kmode driver for Linux as well. And that also broke RHEL recently... OOPS

[@llamatronian101]

Yup, this isn't just an accident. It's a pattern.

[@piquat1]

Wow, fortune 50 company I used to work for had all the users on windows, of course, the back end for the most critical things ran on RHEL. Wonder how they're doing now. lol

[@wckvn]

Running bit-locker feels more like a "Hurt Locker"...

[@fatalglory777]

Why does a badly written driver stop the machine from booting? Shouldn’t that driver just be skipped and whatever device it targets not work? Seems like a terrible design within Windows.

[@DimkaTsv]

It is not Windows design issue. It is driver being written as kernel-mode, to gain extensive privileges over system in attempt to prevent malware activity. Meaning it becomes required to boot before even Windows takes over. And if it crashes, well, Windows hadn't booted yet. And it cannot exclude this driver from list as it is listed as mandatory. Loop repeats. And no revovery window appears because system crashes before it even reaches said state. Someone even said that CrowdStrike already did similar stuff to Linux systems at 19-th April. Linux and Macs versions of CrowdStrike are also using kernel mode drivers, albeit with some nuances (like there being restricted version for Linux). And similar case could've also caused bootloop (on Linux BSOD is more known as Kernel Panic).

[@norbert.kiszka]

@@DimkaTsv This is a Windows design issue. Try to load exact same file as a Linux module. It will not crash, but You will have a simple warning and that's it.

[@paavobergmann4920]

the prob is it had to be done as a driver to get kernel mode. MS tried to develop and license an interface that would allow that without having to load potentially sketchy kernel-mode drivers before boot, but then the EU stepped in and forbade it, because, they figured, it would give MS an unfair advantage on the market, if they were to choose who would get costly privileged kernel access. So you can blame the EU as well. Or you can blame Crowdstrike for knowing they are doing very, very sensitive stuff, but OTA-pushing a bad, untested update regardless.

[@RickOShay]

3rd party access to kernel mode plus cloud service = recipe for disaster. Crowdstrike - aptly named - soon to be a Null company pointer.

[@lashlarue7924]

OMFG, Theo thank you I had no idea how terrible this was!! The encrypted bitlocker problem is absolutely horrendous! Oh my god, I could maybe get this sorted but most people definitely can't, this is BAD!

[@goldguilder9554]

Imagine a robotaxi malfunction due to crowdstrike

[@somerandompersonintheinternet]

WOW. I'm a developer currently on vacation, and I'll be back to my job on Monday. My computer has been off for the past two weeks so I guess I'm lucky? Assuming right now they are no longer shipping the bad update and I can safely turn on my PC, but will definitely make sure next week!

[@Texas3Percenter]

Lol, you dodged a bullet, brother!

[@IAT1964]

Disconnect from internet and then boot up.

[@bmanpura]

They fixed it. I just booted my computer no problem after not using it at all for the past 2 days.

[@Warwck24]

Urrrgh must check mine grrrr

[@Sandy-o4p]

Unplug it from the internet when you boot, and then turn off the updates.

[@everbliss7955]

3:34 - People always think hackers target Windows because it is insecure but that's actually not true. Microsoft Windows is the most used at 72.22%, followed by Apple's macOS at 14.73%, desktop Linux at 3.88%. Just by looking at this, one can easily deduce what operating system a sensible hacker would target if they wanted to create malware. So, its not that the other Operating Systems are secure but a matter of ROI. If you spend a month creating malware for windows you get 72% possible targets while on the other hand spending a month creating macOS malware will give you only 14.73% possible targets.

[@Texas3Percenter]

It's purposely written like swiss cheese, full of back doors and vulnerabilities so they and intelligence agencies can access your computer any time they want. Linux is not. So Linux users don't have to worry about viruses or malware and don't have to put middle-man software between them and their machines to protect them from their malware/spyware OS. The only thing I use on my Linux servers is a firewall and Fail2Ban to prevent brute force pwd cracking.

[@nalstudio_official]

​@@Texas3Percenter bruh you repeat this borderline insane conspiracy shit on every single comment

[@Texas3Percenter]

@@nalstudio_official You're just not knowledgeable of these things, bruh. Educate yourself before you go talking shit.

[@kuro0021]

Problem with bootable USB device is that, a lot of corporate systems also disable USB for security reasons, this gets more interesting 😂

[@H4KnSL4K]

@3:42 - Re: This only happens on Windows, because it's the only one that's insecure enough to have problems like this? I don't know about this .. if Linux or Mac had the most marketshare, such that a company like CrowdStrike ran its software in the kernel to prevent malware, and they pushed a bad update .. maybe the screen wouldn't be *blue*, but you'd have a kernel OOPS or a reboot or a kernel panic, etc. It's not because Windows is more inherently less secure. (Even though that might be the case) And it's just a pile of hacks on-top of each other? Dude, I think Windows has quality problems, but I think this statement is just too simple and immature. I've lost a lot of respect for you now.

[@timothyvandyke9511]

I’m shocked how much windows there is in infrastructure

[@xlerb2286]

I don't quite get the bit about most antivirus not using drivers. Drivers have been an important part of AV and other security software going back to the Windows 95 days, I worked for a company doing security back in those days and we had a file system driver that was the core component for the file system security portion of the product, and a keyboard driver that was part of the system that ensured commands given to our system were coming from the interactive user and not from some script or application. Drivers are so important as they are outside of Windows. A normal application, even if running as a privileged user, cannot kill or modify the driver, nor can it bypass the driver when talking to the devices controlled by the driver.

[@philipsauers4987]

Southwest Airlines unaffected. Use older version of Windows. Brilliant. Latest/Greatest not always good.

[@TruthSeeker-m3w]

Could also be that they simply not using Falcon...

[@luketurner314]

11:05 did their Markdown rendering server also go down? because that looks like the syntax for a link in Markdown: [human readable label](URL/URI)

[@liningpan7601]

The file full of zeros look suspicious. Could it be supply-chain attack?

[@Wayoutthere]

CS biggest investors/owner...Blackrock

[@andrewhooper7603]

​@@Wayoutthere blackrock hacked crowdstrike?

[@gfixler]

The amount of nonsense I haven't dealt with over two decades, since switching away from Windows to Linux in 2006. I was on most versions of Windows since 3.1 in 1991 (3.1, 3.11, 95, 2k, XP, NT, 7, 8, 10, the last three for work), and Linux has been like a breath of fresh air for 18 years. It's really nice to be able to actually control everything on my system.

[@darkshoxx]

And Hammond gets a shoutout here as well 😎. Also, great video of course, really enjoyed the take on how to and how not to communicate in such a situation.

[@t3dotgg]

Absolutely! If I didn't shout him out in the video directly that was an absolutely L on my part

[@darkshoxx]

@@t3dotgg Nono, you did, 12:40 👍Hammond collab when? 😉

[@ProfessionalBirdWatcher]

My rage at everyone downplaying this for CrowdStrike is immeasurable. This is a billion dollar company, with a B, trusted by critical government, public, and private services and they shafted each and everyone. The lack of outrage from our authorities is absolutely disgusting. Speaks a lot to the state of cybersecurity and tech in general

[@Lucius4992]

Every time I hear about this it starts saying (pretty much every PC in the world was affected). I never heard about CrowdStrike before. Every person I know is unaffected and I didn't hear about any company or service affected where I live. Anyway, good luck guys.

[@JoshuaRotimi]

Lol. So Annoying. His own PC was not even affected so I'm wondering how he came about "every PC in the world"

[@craigalexander9421]

As soon as he said that I stopped listening.I wonder what else he is going to get wrong. So much misinformation going around.

[@TalynOne]

Yep, this video is just full of misinformation.

[@adedayoadedapo472]

I think it primarily affected enterprise clients, and that's what he should have led with. But I guess he couldn't resist a little Microsoft slander 😂😂

[@Joealbert83]

At this moment there are probably hundreds of foreign agents planted at all levels in important companies. Imagine the damage that can/will be done when the time comes.

[@jaguarj1942]

The take on windows being the least secure OS is a bit biased. The real reason why most cyber attacks happen through windows is because 1. It is the most used OS by a huge margin. 2. Since it is so widely used, hackers focus on finding vulnerabilities in windows instead of other OS like Linux.

[@Texas3Percenter]

It's purposely written like swiss cheese, full of back doors and vulnerabilities so they and intelligence agencies and access you computer any time they want. Linux is not. So Linux users don't have to worry about viruses or malware and don't have to put middle-man software between them and their machines to protect them from their malware/spyware OS. The only thing I use on my Linux servers is a firewall and Fail2Ban to prevent brute force pwd cracking.

[@insu_na]

Complete fabrication. Linux is the most widely used OS. Get your facts straight

[@AlexanderOsias]

@@insu_nareally? How so? I thought it was windows.

[@bambooindark1]

@@insu_na In which context did you mean Linux is the most widely used OS?

[@JustFacts42]

@@Texas3Percenter Oh so your Linux box is insanely easy to get into. Interesting that you let us know this....

[@diogotrindade444]

This situation can happen again if we do not fix this broken system: - Stop using Windows only, if you buy multiple OS types it can be more work but it is the only way, even if it is not Windows it is better. - We cannot have deploys without lots of testing pipelines, I am sure that they did not test it, if they test the tests are really bad. - We cannot buy a PC that has forced updates, even if it is not fully secure for some hours the users need more control over it. Even if we keep using force updates they need to have some stages to pass first, for example, let's test a small number of users first, then scale up in a controlled way.

[@BryanK-y5y]

I mean there's no way a company with that many skilled people rolls out a zero bytes file does anyone think this was deliberate? Theres more to this

[@mykeprior3436]

a null file? No it's a low level admin, it's flat out a cyberattack by a disgruntled employee or malicious actor with access. Or someone playing around and MASSIVELY fucking up. For you to zero a file is deliberate, any checksum bundling the update would've instantly failed. Anyone with hexedit can zero a file.

[@RonnieDenzel]

RIP to the intern😢

[@TheJFerg24]

If an intern did the coding or deployment, then their supervisor needs to be in big trouble.

[@pokefreak2112]

The fact their stock price barely took a dent is insane. This is the kind of mistake I'd expect from a solo startup with 30 users, not a multi billion dollar corporation!

[@gothmog2441]

Stock market PCs are probably down. Next week though …

[@dead-claudia]

1. it takes more than a day for such issues to result in stock price drops. 2. cloudstrike fixed it same-day. it's just the damage done was so severe, it's taking customers a disproportionate amount of time to fix.

[@samuelgunter]

more like clown strike haha gottem

[@corymollak2093]

The real questions that everyone should be asking is ...... What happened during, DURING the bootloop, because that's when anyone can access ANYTHING....and yesterday was an entire day of chances!

[@nicejungle]

there is no network, genius, that's why, this problem cannot be remotely fixed

[@corymollak2093]

@@nicejungle 😄😆🤣

[@lcarsos]

Hah! That USB boot idea would be fine, if it weren't booting you into safe mode after unlocking your bitlocker. That's prime attack territory for planting a rootkit while "helping" you clean up that crowdstrike BSOD. You'd have to audit everything about how that USB key came to be and all the software on there, from extremely trustworthy sources.

[@FengLengshun]

Legitimately, this is when image based or Atomic OS a la rpm-ostree would really benefit the userbase. If you can't boot the update, you just boot the previous one, and wait for it to be fixed. No CrowdStrike taking down your entire system, no GRUB breaking your entire system. Seriously, people should just use immutable OS these days.

[@innervoicesrpg]

Ooooh yeah I love the voice that tech influencers have when they weren't expecting to go recording and their humanness comes out more??? Like, idk what it is about it, it sounds like you just woke up (compliment)

[@amagicmuffin1191]

​@@NormCantoralthose weren't tech issues

[@amagicmuffin1191]

@NormCantoral that makes sense, I just thought that interpretation was so unreasonable that it was more likely he just hasn't ever seen a software bug of this scale and severity that was caused by something so easily preventable. made sense to me bc he runs a tech channel.

[@Hurricayne92]

This being an accident is more terrifying that if it was done on purpose.

[@michaelwills1926]

Maybe this was the canary test for that intent. Integration has teeth

[@DEEPMMA]

alot of hospital computers are down too which is very dangerous

[@shaunweinberg2463]

Somebody changed one life of code on a Friday afternoon, pushed the pipeline, and now we get this

[@silverknightgundam1196]

It's not Microsoft bug/error. it's a Crowdstrike bug/error

[@randomeman3]

At my workplace i responded to this i incident at 12:45 am, the local dispatch center had all the conputers down. I was not given approval by my higherups to run the fix until around 8 am that same day. Man did I sleep good Friday evening.

[@NillKitty]

You really lost me on this one theo. There isn't anything here that couldn't happen on Linux, in fact, it did -- about a month ago. It's not guarding your bootup and preventing you from booting in an unsafe environment, its just a broken kernel mode driver. I got bluescreened. You know what happened? Windows dumped it's ram, rebooted into last know good config, i shrugged, and moved on. I can't believe this overshadowed Trump being shot to the point i hadnt heard about the latter until an hour ago

[@mwahlert]

Exactly! Ignorant people acting like this cant’t happen on Linux. EDR agents on Linux are just as deeply engrained, and highly privileged.

[@NillKitty]

@@mwahlert "omg Windows"... No... Omg the market share! Bet you didn't know Windows ran in all *these* places :3. "Omg Windows needs this?" Up until 2019 CS didn't even have any offensive antimalware tech, it purely was marketed as a distributed, crowd sourced IDS for setting a baseline and then detecting anamolies (whether abnormal for your business, or just plain university unusual activity) "Omg a driver?" Yeah .. if you want network inspection (NDIS) or any kind of endpoint DLP, and unless you want your employees ripping it out, changing it's permissions, etc. As someone inconvenience by it daily since i run executables classified as "hacking tools", CS does an amazing job of preventing bad stuff from running, no matter who you are. First CS issue I've ever faced the wasn't as simple as requesting an exception for a given piece of software.

[@sub-harmonik]

I mean this happened last night and trump got shot 6 days ago.. living under a rock sounds like a choice also windows is the hodgepodge of configurations and programs and apis he said it is. Everything is an inconsistent mess and it seems reasonable to think that would leave more security issues..

[@deedos]

I wouldn't say this overshadowed the Trump shooting, that's just a reflection of the content you interact with, there's been tons of coverage over the last week regarding Trump

[@vast634]

Apart from obviously missing QA on the shipped update, they also pushed it to everyone at once, instead of first having a limited roll out, that could signal potential problems ahead of time, on a limited number of systems.

[@HumanAction76]

CEOs can't apologise to that extent for legal reasons. That proposed comment would bankrupt the company.

[@dead-claudia]

yep, especially since they're publicly traded. privately owned companies can get away with more, but only bc there's a lot fewer people who could have standing to sue over that.

[@DohTheOpinionator]

DUDE!!! I managed to use a combination of your gudance and a "shortcut" and I fixed my workstation. You're awesome!!!! My company doesn't allow access to the folder so I had to shortcut my email to my personal machine to get temp admin access. Other than that, followed the instructions, and viola, I'm in. Thanks.

[@ronanru]

Why are there fireworks behind Theo at 7:05

[@GameKornel]

On new macs the camera sees your hand gestures and makes these animations. There are few of them

[@dealloc]

To clarify; it is not confirmed whether it was a _driver_ or just data used by the driver that caused the incident. The .sys extension is not only used for kernel drivers, but also data (i.e. see Windows' own KEYBOARD.sys which are is not a driver in and of itself, but contain a database of keyboard layout and P-Code sequences that a interpreter in the KEYB driver can execute) Removing the file in question has solved the issue for people, meaning it is not a critical part of the driver itself in that case.

[@DimkaTsv]

It probably was data, as devs referred to it as "channel" file.

[@starupiva]

I am a support partner for Microsoft. To resolve this issue, boot into safe mode or recovery environment and then go to the C:\Windows\System32\drivers\CrowdStrike directory and delete the “C-00000291*.sys.” file. Then restart the system in normal mode. That should fix the issue.

[@malvoliosf]

If you have an unencrypted drive...

[@starupiva]

@@malvoliosf even if you han encryption enabled, if you have the recovery key. The recovery key can be obtained from your Microsoft account. Enter it to get into the recovery console. Once you do that, get into safe mode and delete the crowdstrke.sys file.

[@uzlonewolf]

@@malvoliosf There are now instructions on how to do it on encrypted drives without a known key as well.

[@Micloren]

Average person & a good chunk of IT people don’t know what safe mode is not how to navigate a command line.

[@death_au]

It asks for uac, too. So still need IT to do it for me, even if I know how 🤷

[@MaxGarmin]

Crowdstrike, now probably the most hated company by end users and IT folks globally. Customers will start to make plans to exit from their offerings and consider other options.

[@mohamed1208]

This had to happen during my vacation week

[@vladfather916]

Lol

[@Jimothy-723]

oof

[@chaseywoot]

This had to happen during my school's online subject selection

[@Texas3Percenter]

That sucks! I have to close on 2 real estate deals Monday. They won't happen til this is fixed. Thanks, banks, for using crowdstrike! If you were using Linux instead of Windows, you wouldn't need to.

[@andrewreed1329]

suck it up lol

[@joyjit_roy]

*Hello Theo, let me give some perspective to the level of headache this is from a corporate IT team perspective. Our company has 10k servers across the globe - all down. Last night MS reached out and our limited IT team last night (it's Saturday in India) could only bring up 250 something servers ! A lot of them we couldn't even log into ! And it was saturday night !*

[@ttrev007]

i personally think that while the emergency is occurring i don't want the CEO wasting time crafting apologies. Their goal should be dispensing accurate actionable information to help people recover. apologies can be crafted after.

This event exposes the risks inherent to over-consolidation. While standardization is generally "a good thing", there can be too much power given over to CrowdStrike and MDE plan 2 to just let them automatically delete whatever files they want without a definitions update approval and review process that the customers' security team can first review before applying to say 100k or 10M endpoints. One of Microsoft's under-tested definitions deleted a bunch of users' application shortcuts in 2022 and users panicked thinking they were hacked and all their apps were removed.

[@taterrhead]

anyone else notice how 'muhhh windows vulnerabilities' only became a big thing once ol Billy Gates got into the anti-virus business? (don't dare notice the same pattern with him entering the MRNA jab space ...)

[@Texas3Percenter]

Yep. The OS IS the malware!