CrowdStrike IT outage: How and why it happened | BBC News

[@venkateshthiruvalluvan9070]

In this situation, Linux & Mac users should be having a great day.

[@hydraflyer]

Not every Windows runs Crowd strike you know right, they have 24,000 customers. Not every single Windows machines. Every personal machine is not affected

[@cheasamnang3306]

៩១😂រឲ១២ឲុ៣២៤😂តតតឥរុ៨េ០ុ៩😢េោឆឆ៧ឆិ១១😢

[@cheasamnang3306]

🎉៥ឥ្៧៥ុ៣០ោ៨ឆ៦យ

[@cheasamnang3306]

៣ឥរ១😅ោ២១យ ុោុ១១៧2:12 ផ១ឆឆ៥៧៣ 2:39 😅ិោ៦៩២ឆ១ិុុោិ្៦🎉យ០យ៣៣😂៥ឪឹតយឆ៩៦២😢😂១៦៧

[@TomNook.]

Someone didn't read the news. No personal computer user is affected. Only Enterprise windows using Crowdstrike software.

[@CMansfield]

Did they try turning them off and back on again?

[@phoenixxam5812]

Reminds me of Skyscraper with Dwayne Johnson and Neve Campbell

[@jaycfc6955]

Worked for the hospital receptionist who helped Howard Wolowitz in TBBT when the robot hand was stuck on his .......😂

[@orlagh277]

Ironically I saw reports that that works for some. If you reboot many times the software is likely able to not crash for long enough to download and apply the fix.

[@o_m8717]

😂😂😂

[@I.____.....__...__]

@@orlagh277 If you access a system in person, there's no need for that, you can just boot into safe mode and delete the driver file.

[@TomNook.]

There's such delicious irony in spending thousands using a Cybersecurity company to protect your systems from going down, when it's the reason for your systems going down

[@jacquelinel1618]

Crowdstrike was responsible for the 2016 story that Russia hacked the DNC in order to get dirt on Hillary and favor Trump.

[@cheese-power]

As a computer scientist, I can tell you that an opening system is the exact line of defense for this kind of issue. It should never let an application bring down the entire system. Therefore, it is the problem of both Microsoft and CrowdStrike.

[@hongyu_he]

Exactly 👍

[@Sloane1_4]

To add; Back then, there was a "PatchGuard" which would prevent modifications to the Kernel. Many software (including EDRs like CrowdStrike) would like to gain this level of access for performance and telemetry. This didn't sit well with many companies though, so kernel callbacks were made to allow telemetry for EDR solutions like CS. Not exactly modifying the Kernel but allows creation of callback routines when an event happens. EDR benefits a lot from this level of access by having telemetry for processes and threads which is super useful in detecting malware activity. So we can also think of it as security solutions trying to protect you better but inducing risks along the way.

[@sinewrites6581]

You can say that, but having an application blue screen a windows machine isn't that hard. The problem lies with crowdstrike. This update should have never been pushed. It speaks to a lack of testing. No doubt they're going to lose a lot of customers to this, if they even survive. Of course, the companies that actually automatically update critical systems kinda had this coming too. That's just bad practice, it was bound to go wrong at some point.

[@I.____.....__...__]

Thank you! Yes, CrowdStrike had bad code, but it wouldn't be a problem if Microsoft hadn't made Windows so poorly designed to ALLOW it to do this. If Windows weren't so awful, it would make it quick and easy to diagnose it, disable it, and continue running, not become COMPLETELY UNUSABLE. 😒 Microsoft is 100% to blame.

[@MorningNapalm]

They really struck the crowds hard with this update.

[@apl175]

Hope y'all have your bitlocker recovery keys stored somewhere handy.

[@gbraadnl]

4:35 unless you run BitLocker. In that case a lot more steps are needed. Plus, you forget possible remote workers, systems in different datacenters, etc. You are also wrong, as this is Windows specific as they use a driver model for this tool. Linux is not less used in corporate environments, but see a different use and also a different method to protect.

[@Woodzta]

If you're fixing it manually then Bitlocker is only more step - typing in the recovery key at the command prompt before navigating to delete the sensor update file. Without BitLocker I agree it can be far easier to automate and spread a fix through a startup script but I guess it depends on the environment

[@i.abhshek]

The issue is with organisations which are understaffed and do recovery... Who asked you for the job cuts and rely on computers

[@kirishima638]

This is why I NEVER install updates automatically. Let someone else beta test a patch first.

[@I.____.....__...__]

I used to update everything ASAP, but no longer update anything at all. Most updates are unwanted crap that ruin the UI/UX for no reason other than to make change for change's sake. 😠 And "security" is a specious reason since every patch that fixes a bug creates several new bugs. Coders are really incompetent these days, they just don't make them like they used to.

[@yasaminwhy8212]

I can't be the only person who liked the lack of clutter on the BBC News screen, for one brief morning. No rolling ticker tape, no updates flashing anywhere, just the current news item. Much calmer.

[@Greenpoloboy3]

Liz Truss' first few days working for CrowdStrike went well it seems.

[@MiftahulJannat-g4c]

Please save Bangladesh, our country is disconnected from the world, the government turned off electricity and internet... we are in a big crisis. We can't contact our family members from Bangladesh, this situation is outrageous. We desperately need any help we can get. Please save Bangladesh and it's student. Over 40 students has been killed and thousands are injured.

[@sugathsuranjeewa]

I think the best thing is using Linux instead of Windows at situations like this

[@danielxtma]

yes, but we need more corporativew support for user interfaced linux

[@earthluma]

What about macOS?

[@Kreze202]

​@@earthluma Using Mac devices for servers is highly cost inefficient.

[@theblackhand6485]

Indeed use a Linux based software. No problems here.

[@brianfunt2619]

Enterprise Linux users were affected by a Crowdstrike issue about 3 months ago

[@WhippedForJeonJungkook]

Work has been a nightmare todaywith this down

[@Solo-gamer46]

They should compensate everyone affected, all the holidays cancelled, missed appointments and so on, what a bunch of bumbling bafoons.

[@GrandpaVince]

Old Vincey is an IT expert

[@xlpeoplepower]

I can't have been the only one noticing the use of the old pound notes (and thus the use of old footage for illustrative purposes) at 3:48

[@ScienceChild]

You were congrats

[@I.____.....__...__]

2:04 Everybody is always being told to always update everything to the latest version always everywhere on everything no matter what always. And they use hyper-specious fearmongering to do so. The fact is, every patch that fixes an old bug, introduces several new bugs and zero-day vulnerabilities that hackers are slavering to reverse-engineer. And that's just the security aspect, forget about how updates usually bring unwanted, unneeded, and awful UX changes that ruin everything. 😒 3:04 Microsoft deflecting blame is also specious. Sure, the main problem is with CrowdStrike's code, but Microsoft's GARBAGE design of Windows is the reason this is happening. If Windows wasn't such a pile of delicate rubbish-code, then CrowdStrike's bad code wouldn't have caused this, it would have been possible to quickly and easily diagnose it and quickly and easily disable it and quickly and easily revert the change and quickly and easily get back up and running. But no, Windows is TERRIBLE, so all anybody saw was a generic, unhelpful BSOD and no way to get around it. So yes, Microsoft is culpable. 4:42 Exactly. If Windows wasn't so poorly designed, it wouldn't be COMPLETELY UNUSABLE with a problem like this, it should have been easy to remotely disable the problematic module and remotely patch it.

[@duncanstewart6381]

Considering the monetary losses an investigation and report must be produced. The update was flawed itself. The lost capacity and opportunity cost is in the millions already as the airlines can not function.

[@brilliantbuilder4228]

That happened to me when I was working on the Gateline at Staines Station today. For all the passengers that had E-tickets and paper barcode tickets, we had to keep.the widegate open.

[@philboast8841]

Windows having a larger client base than Linux or Mac is completely irrelevant. The fact is that it is a completely compromised hack built on hack and riddled with security issues.

[@KarlJones73]

Updates should be pushed out in progressive waves to avoid exactly this situation. Change Management challenges processes for blackout and recovery plans. There are more issues than just the update pushed out and breaking.

[@Not_First_Of_His_Name]

exactly.

[@Kreze202]

An update that automatically BSODs a machine post update is not a roll out problem, it's a QA/testing problem. Did no one test the update to make sure that, you know, it doesn't brick machines at the very least???

[@satyamevajayate7263]

Such incidents will never occur on Debian stable or even Ubuntu server. They do not implement any changes instantly even for months.

[@sinewrites6581]

So many companies having automatic updates enabled on critical systems. It would be hilarious if it wasn't so sad.

[@りくび-z1v]

人間は青い空を１日1時間以上見るとストレス軽減になるらしいので、だからパソコンの画面が青くなった　と真顔でブラックジョークを話したら、夫が笑ってました。

[@Maeni]

Boring couple

[@Snoopyzell]

Clearly explained... thanks

[@helloman121220]

Could not play my game, packet burst, and high latency. It negatively affected my K/D ratio.

[@mussy7065]

rooight

[@SofaKingShit]

Oh the humanity

[@jamiebarnes3539]

It would have been advantageous to have a PiKVM connected to the network, as this would have allowed for remote resolution of the issue. About PiKVM A very simple and fully functional Raspberry Pi-based KVM (Keyboard-Video-Mouse) over IP that you can make with your own hands. This device helps to manage servers or workstations remotely, regardless of the health of the operating system or whether one is installed. You can fix any problem, configure the BIOS, and even reinstall the OS using the virtual CD-ROM or Flash Drive.

[@SoonGone]

Somebody's in so much trouble.

[@misswaychannel9186]

Ive mistakenly deployed an update to all Estate before, the intern is acting non challant on twitter CrowdStrike's "Falcon Sensor" software was causing Microsoft Windows to crash and display a blue screen

[@icebergfitnessofficial]

My flight just canceled because of that.

[@BritishMoralHQ]

they should have done a version release..and clients have time to test it ...update right away should be illegal ...! or atlest cloudfront should do some ..track of all consumers of endpoint and test it

[@jacksoncremean1664]

it was a signature update that caused the issue, this is something you can't delay.

[@Kreze202]

​@@jacksoncremean1664 Surely proper testing/QA would spot something like this prior to roll out? It's baffling this update even got to being rolled out

[@beastmode00714]

They sent the wrong file. It was all zeros. True story.

[@joetran8798]

This is not a new piece of software. It is an update to an existing component.

[@amyhogarten5038]

Maybe don’t use the terrible Windows operating systems?

[@brianfunt2619]

Omg BBC News is on the screen behind him!! He's recursive!!!

[@lorenzoluizdesouza1215]

Indian IT help desk please help us😂

[@HolloMatlala1]

We really need all that technical help desks right now

[@Maeni]

Stereotypical so ignorant

[@HolloMatlala1]

@@Maeni A Fact is a Fact: India dominates in IT

[@lorenzoluizdesouza1215]

@@Maeni and you're dumb af for not understanding a joke. They say if someone can't handle sarcasm their iq is as low as horseshit

[@imik3y]

So are you all ready for a cashless society with an AI overlord?

[@zerothm1]

Practice Run

[@The_Savage_Wombat]

Linux saves lives. Show the person who wrote the bad code...hilarious to hear their side of the story.

[@tacorevenge87]

It’s windows fault. If windows did not have so many holes . Also why did windows allow a vendor handle updates at the kernel level as an automatic update without asking users? Second. Why do companies run windows in prod servers?

[@Kushal-v8b]

Global IT outage

[@m.rugsunshine]

Is the answer "hubris"?

[@strikezonefishingadventures]

CrowdStrike screwed up on a massive level. Their IT Project Management Team dropped the ball in a disastrous manner. The only way this happens is that the appropriate levels of testing were not completed on a live test system from end to end prior to implementation. What a disaster. Now CrowdStrike has a major financial problem and a major credibility problem. They can most assuredly expect that they will be facing lawsuits demanding tens of millions of dollars in compensation and damages for this mess from customers around the world. Question is, are their pockets deep enough to cover this, or are they now facing insolvency as a result? Either way, expect a pile of customers are also going to be looking at alternative platforms, so a safe bet that they are going to be losing a pile of customers as well. No matter how you slice this, there are a pile of senior staff at CrowdStrike who can expect a pink slip in the coming days for absolute and total incompetence!!!!!

[@a0um]

I’m pretty sure most of those computers don’t need to run a general purpose operating system so bloated and complex and therefore vulnerable to security flaws that it can’t guarantee being secure?!? They need to install another software to protect the operating system?!? That’s sick! The government should invest in small, secure, open source, operating systems… systems that are secure by design. But there is a lot of money sucked by inefficient insecure operative systems once the manage to lock you in! People, ask the government to run the infrastructure on open source systems!

[@Sloane1_4]

You missed one big component: Cybercriminals

[@a0um]

@@Sloane1_4 please explain

[@Sloane1_4]

@@a0um Most systems and software are designed with security in mind. But continuous need for features adds dependencies on top of dependencies and introduces the need for continuous bug fixing and improvements. Parallel to this, Cybercriminals are also continuously improving at the same rate and requires developers to be proactive and get ahead. This scenario with Crowdstrike is a good example: With Cybercriminals introducing rootkits and kernel level exploits, Microsoft needed to create what they called a Kernel PatchGuard to prevent modifications fo the Kernel. However, this also prevented anti malware solutions like CrowdStrike to gather telemetry from the kernel and effectively prevent other types of malwares from executing. So Microsoft introduces what is called Kernel Callback which allows anti malware solutions to create callback routines when an event happens. This however increases the risk of crashing the kernel, but has been very effective in detecting malware. So there are multiple streams at play here that will prevent having one open source secure system: The consumers need to for better tech. The cybercriminals looking to break them. The cybersecurity trying to stop them.

[@theblackhand6485]

Stupid people thinking that an update is the best thing to do. Of course not. Besides that it is a way to controle the masses you always have to wait until bugs are neutralized. But lazy and ignorant people turn on their automatic update slider on their computer or iDevice. Then you get what happened in this video.

[@mattmoreira210]

Go Horse strategy for the win!

[@raaajit1]

modern real world example of butterfly effect , chaos theory

[@kolz4ever1980]

Lil bro just stringed a few cool works from books.. cool!

[@StephenWilcoxon]

Fewer users on MacOS or Linux has nothing to do with it. It is likely similar updates went out to all OSs.

[@croaker6099]

And that's fewer non-commercial users. Back office servers that run everything important are overwhelmingly linux.

[@Jpex369]

This is why we don’t want See Bee Dee Sees!

[@rogerc7960]

automatic coded by chatgpt?

[@Hana-p9y-j5d]

Please report on the misdeeds of the Japanese Ministry of Finance. The Japanese people will be grateful to the BBC.

[@samsung40_media87]

mercedes f1 team not winning this weekend

[@squirreltismw8295]

And I'm doing a course in cybersecurity lol

[@Mark-kh1ny]

There are several remote management tools that can restart an end point in safe mode. I spend my entire life telling companies to get their sh*t together. This doesn’t surprise me….. “It’ll never happen to me” syndrome……

[@7_of_9]

Crowdstrike is a backdoor for hackers and they did get hacked two years ago 😅😂😅😂 who in the world trust a third installing files without testing it first

[@fantabulous6657]

I finished my work early on my Friday thanks to the outrage 😂

[@makoto5071]

Is this only effecting machine running CrowdStrinke? or Machines connected to Machines at Microsoft running CrowdStrike?? I have a Tiny datacenter for my personal use with 20 rackmounted Servers running Windows Server 2016 on AMD Epic CPUs that had no problems at all. I update the security patches manually and have automatic updates turned off.

[@vitaliyrusinov4788]

Yes. Only the machines running CrowdStrike.

[@Kreze202]

It's a problem with CrowdStrike drivers on Windows machines (correct me if I'm wrong), so as long as your machine does not have CrowdStrike you're perfectly fine.

[@Sloane1_4]

Mate was just trying to flex my mans, no need to answer him

[@AlexR_44]

@@Sloane1_4 😂👍😂

[@ricnyc2759]

Winners: Linux and China. "Microsoft outage leaves China largely untouched as tech self-sufficiency campaign pays off" (SCMP, today).

[@daledevernon56]

So who thought that a cashless Society would be a good idea.

[@Lynn22823]

💯

[@doctorpsylus]

test run ?

[@misterfunnybones]

The EULA said it might happen.

[@oofooflemon]

crazy

[@_Tannu786]

Mere bhi computer mein aisa same aata hai toh iska kya matlab hai??

[@AfroMuggleKing]

PXE boot?

[@TomTKK]

Bitlocker

[@Twirlyhead]

Who needs Russian trolls with friends like these.

[@YusukeYamamoto-x8k]

missing pacifism is.... By the way, enthus, of that...Shizo Abe assassinated...certainly. Donald Trump depict that? as....golf freind? pacifism destination...all tye way to Japan? But neither English nor Japanese...in that threatening. "Pacifism is the opposition or resistance to war, militarism (including conscription and mandatory military service) or violence. The word pacifism was coined by the French peace campaigner Émile Arnaud and adopted by other peace activists at the tenth Universal Peace Congress in Glasgow in 1901.[1] A related term is ahimsa (to do no harm), which is a core philosophy in Indian religions such as Hinduism, Buddhism, and Jainism. While modern connotations are recent, having been explicated since the 19th century, ancient references abound." By the way

[@lesliegrace8360]

"Trust Me Bro"

[@hi-tech-guy-1823]

For Mission critical (Can be a risk to life) You Should have a Live Linux USB Flash Pen drive you can plug in and Run Linux Versions of your Software (Identical UI & GUI Just running on a Different OS ) That can Run Without a cloud & Internet so your Mission critical functions Can still Run - and just Export the DATA Back to the cloud when it back up ( I would thow in Local P2P Mesh + Grid parallel computing (Boinc - Berkeley Open Infrastructure for Network Computing) ) Even is is just for a sec ~ a few days ~ A Few months ~Till the END OF Time

[@mimimi3440]

do they get persecuted for somthing like that?

[@天下為公1911]

Microsoft: it's really not our fault this time.😅

[@CPORD-ss2db]

Good for let us know this cannot be simply fixed by "patch auto update". But we still don't how and why it happened.😅

[@croaker6099]

One 35KB file was wrong and caused the OS to crash. The CS software can prevent bad things from executing but if it misidentifies some critical OS process or component as bad it can crash the whole thing.

[@CPORD-ss2db]

@@croaker6099 Thank you so much for the explanation. Guess CS did its job but overreacted.

[@Kreze202]

​@@croaker6099 Do we know if there were any kind of test or QA involved in this update? This seems like something that could have easily been spotted had someone tested it

[@The_Lost_Subrosian]

Skynet

[@GabroPlayz]

Skynet

[@stabilis8895]

Leeds

[@GabroPlayz]

Leeds

[@michael.j.mueller]

Your expert utters mostly bs. He is still very young to be an expert. I give him that.

[@QSZXQSZXQSZX]

Windows11 and bug

[@pedrosmmc]

DEI fingers on the keyboard?!

[@souvikdas5662]

Win 11 or win 10 ?

[@MushramboMu]

Would be great if someday Europe and UK can build their own big Operating system and a new search engine that surpass the current popular ones from outside UK and eu OpenKylin = china Windows = usa Europe= ??? Uk= ???

[@reddragon8]

Maybe no one should install updates on day 1 itself 🙄

[@MorningNapalm]

And then, because no one tested it on day 1, no one should install the updates on day 2...

[@Not_First_Of_His_Name]

it is not possible in crowdstrike. It is similar to windows. You need to know before an update even gets released and you should pause manually for all the endpoints, information displays at airports , etc.,

[@reddragon8]

@@MorningNapalm Well technically Crowdstrike should have tested before day1 😐 QA/QC anyone?

[@paperplane-db8qf]

Well at least not on disaster recovery servers so companies can failover quickly.

[@Mark-kh1ny]

@@paperplane-db8qfdepends how DR is commissioned and using what tool. For example if Zerto is used, you’ve already replicated the problem to the second site. Point in time restore? What’s the lost data worth? Complicated decision invoking DR

[@dr9gonkid20]

AI CODE AND SUPPORT FOR THE RIGHT WING

[@HaraldEngels]

Why there are still companies which are not running Linux to prevent the never ending issues with Windows?

[@thelastofthehitachi972]

backdoor "π"

[@samholden4171]

They probably blame the russians for it😔

[@kennethworkman4740]

Computer says no lol

[@theflipflapchannelcreatedb8160]

Remember the Y2K millennium bug? Look further 24 years later!

[@rebelm.s4481]

❤

[@theblackhand6485]

Now Jo Biden is a head in the polls?

[@hdtravel1]

How could the whole world's computers be brought down by ONE companies software ? We are hugely vulnerable to the bad guys !!!

[@KenBarberGeek]

Crowdstrike cut corners this time to everyone’s detriment. Also they have bloated ram requirements already (badly tuned jvm based agents), but didn’t listen to customers. Money makes companies do silly things. I hope they have learnt their lesson, but I doubt it.

[@hafirenggayuda]

Other comments also mention, they do this on thursday/friday (different time zones) which is VERY bad timing.

[@KenBarberGeek]

@@hafirenggayuda Friday releases are bad luck

[@hafirenggayuda]

Bad update and bad timing, I suspect bad work atmosphere there. This feels like last minute essay

[@Kreze202]

​@@hafirenggayuda This feels like an overworked dev pushing an update past the overworked PM who approved the update without proper testing or evaluation.

[@RoanGui]

What about the cultural enrichment in Leeds

[@greenfly1264]

CROWDSTRIKE The clue is in the name guys.

[@ZackYounguk]

BBC doing some of the rep management, instead of demanding better.

[@MrChakyb]

It outage ?! Or Microsoft fak up .

[@metapol118]

crowdstrike strikes the crowd

[@ปราณีเดิมกลาง]

Top 10 Richest man in the world 2022 - 2024 . God bless you all and God saved safe you all come home safe saved meet your family friends and love one. Police CIA FBI military NYPD officer frie man BodyGuard sniper smart Power nice Great Strong Richest faster the best better and win. Please saved safe anyone if you can help please thank you

[@CadeNg]

Sboutout to all mega digimons

[@hughw2377]

Russia?

[@GabroPlayz]

Russia doesn't have anything to do about this.

[@jeovanneramos5066]

I honestly think they got hacked. But now the billions of dollars in lawsuits coming up.

[@timotheos0190]

Switch to Linux without antivirus 😂

[@kolz4ever1980]

Trash community made os that can't even play League of legends of all things or use full support with HDMI 2.0. absolute garbage. 🤣