CrowdStrike IT outage: How and why it happened | BBC News

A major software outage has stopped computers working across the world in a number of sectors, causing severe disruption to healthcare, flights, trains, businesses and media organisations.
The outage was caused by a software update from CrowdStrike, a cyber security firm. A bug affected Microsoft Windows computers, knocking them offline around the world.
CrowdStrike chief executive George Kurtz says he is “deeply sorry” for the incident.
Subscribe here: bit.ly/1rbfUog
For more news, analysis and features visit: www.bbc.com/news
#ITOutage #CrowdStrike #BBCNews

[@venkateshthiruvalluvan9070]

In this situation, Linux & Mac users should be having a great day.

[@hydraflyer]

Not every Windows runs Crowd strike you know right, they have 24,000 customers. Not every single Windows machines. Every personal machine is not affected

[@cheasamnang3306]

៩១😂រឲ១២ឲុ៣២៤😂តតតឥរុ៨េ០ុ៩😢េោឆឆ៧ឆិ១១😢

[@cheasamnang3306]

🎉៥ឥ្៧៥ុ៣០ោ៨ឆ៦យ

[@cheasamnang3306]

៣ឥរ១😅ោ២១យ ុោុ១១៧2:12 ផ១ឆឆ៥៧៣ 2:39 😅ិោ៦៩២ឆ១ិុុោិ្៦🎉យ០យ៣៣😂៥ឪឹតយឆ៩៦២😢😂១៦៧

[@TomNook.]

Someone didn't read the news. No personal computer user is affected. Only Enterprise windows using Crowdstrike software.

[@CMansfield]

Did they try turning them off and back on again?

[@phoenixxam5812]

Reminds me of Skyscraper with Dwayne Johnson and Neve Campbell

[@jaycfc6955]

Worked for the hospital receptionist who helped Howard Wolowitz in TBBT when the robot hand was stuck on his .......😂

[@orlagh277]

Ironically I saw reports that that works for some. If you reboot many times the software is likely able to not crash for long enough to download and apply the fix.

[@o_m8717]

😂😂😂

[@I.____.....__...__]

@@orlagh277 If you access a system in person, there's no need for that, you can just boot into safe mode and delete the driver file.

[@TomNook.]

There's such delicious irony in spending thousands using a Cybersecurity company to protect your systems from going down, when it's the reason for your systems going down

[@jacquelinel1618]

Crowdstrike was responsible for the 2016 story that Russia hacked the DNC in order to get dirt on Hillary and favor Trump.

[@cheese-power]

As a computer scientist, I can tell you that an opening system is the exact line of defense for this kind of issue. It should never let an application bring down the entire system. Therefore, it is the problem of both Microsoft and CrowdStrike.

[@hongyu_he]

Exactly 👍

[@Sloane1_4]

To add; Back then, there was a "PatchGuard" which would prevent modifications to the Kernel. Many software (including EDRs like CrowdStrike) would like to gain this level of access for performance and telemetry. This didn't sit well with many companies though, so kernel callbacks were made to allow telemetry for EDR solutions like CS. Not exactly modifying the Kernel but allows creation of callback routines when an event happens. EDR benefits a lot from this level of access by having telemetry for processes and threads which is super useful in detecting malware activity. So we can also think of it as security solutions trying to protect you better but inducing risks along the way.

[@sinewrites6581]

You can say that, but having an application blue screen a windows machine isn't that hard. The problem lies with crowdstrike. This update should have never been pushed. It speaks to a lack of testing. No doubt they're going to lose a lot of customers to this, if they even survive. Of course, the companies that actually automatically update critical systems kinda had this coming too. That's just bad practice, it was bound to go wrong at some point.

[@I.____.....__...__]

Thank you! Yes, CrowdStrike had bad code, but it wouldn't be a problem if Microsoft hadn't made Windows so poorly designed to ALLOW it to do this. If Windows weren't so awful, it would make it quick and easy to diagnose it, disable it, and continue running, not become COMPLETELY UNUSABLE. 😒 Microsoft is 100% to blame.

[@MorningNapalm]

They really struck the crowds hard with this update.

[@kirishima638]

This is why I NEVER install updates automatically. Let someone else beta test a patch first.

[@I.____.....__...__]

I used to update everything ASAP, but no longer update anything at all. Most updates are unwanted crap that ruin the UI/UX for no reason other than to make change for change's sake. 😠 And "security" is a specious reason since every patch that fixes a bug creates several new bugs. Coders are really incompetent these days, they just don't make them like they used to.

[@gbraadnl]

4:35 unless you run BitLocker. In that case a lot more steps are needed. Plus, you forget possible remote workers, systems in different datacenters, etc. You are also wrong, as this is Windows specific as they use a driver model for this tool. Linux is not less used in corporate environments, but see a different use and also a different method to protect.

[@Woodzta]

If you're fixing it manually then Bitlocker is only more step - typing in the recovery key at the command prompt before navigating to delete the sensor update file. Without BitLocker I agree it can be far easier to automate and spread a fix through a startup script but I guess it depends on the environment

[@sugathsuranjeewa]

I think the best thing is using Linux instead of Windows at situations like this

[@danielxtma]

yes, but we need more corporativew support for user interfaced linux

[@earthluma]

What about macOS?

[@Kreze202]

​@@earthluma Using Mac devices for servers is highly cost inefficient.

[@theblackhand6485]

Indeed use a Linux based software. No problems here.

[@brianfunt2619]

Enterprise Linux users were affected by a Crowdstrike issue about 3 months ago

[@MiftahulJannat-g4c]

Please save Bangladesh, our country is disconnected from the world, the government turned off electricity and internet... we are in a big crisis. We can't contact our family members from Bangladesh, this situation is outrageous. We desperately need any help we can get. Please save Bangladesh and it's student. Over 40 students has been killed and thousands are injured.

[@apl175]

Hope y'all have your bitlocker recovery keys stored somewhere handy.

[@Greenpoloboy3]

Liz Truss' first few days working for CrowdStrike went well it seems.

[@Abhishek-wm6fv]

The issue is with organisations which are understaffed and do recovery... Who asked you for the job cuts and rely on computers

[@helloman121220]

Could not play my game, packet burst, and high latency. It negatively affected my K/D ratio.

[@mussy7065]

rooight

[@SofaKingShit]

Oh the humanity

[@duncanstewart6381]

Considering the monetary losses an investigation and report must be produced. The update was flawed itself. The lost capacity and opportunity cost is in the millions already as the airlines can not function.

[@arnie1046]

They should compensate everyone affected, all the holidays cancelled, missed appointments and so on, what a bunch of bumbling bafoons.

[@村上涼子-w8f]

人間は青い空を１日1時間以上見るとストレス軽減になるらしいので、だからパソコンの画面が青くなった　と真顔でブラックジョークを話したら、夫が笑ってました。

[@Maeni]

Boring couple

[@philboast8841]

Windows having a larger client base than Linux or Mac is completely irrelevant. The fact is that it is a completely compromised hack built on hack and riddled with security issues.

[@I.____.....__...__]

2:04 Everybody is always being told to always update everything to the latest version always everywhere on everything no matter what always. And they use hyper-specious fearmongering to do so. The fact is, every patch that fixes an old bug, introduces several new bugs and zero-day vulnerabilities that hackers are slavering to reverse-engineer. And that's just the security aspect, forget about how updates usually bring unwanted, unneeded, and awful UX changes that ruin everything. 😒 3:04 Microsoft deflecting blame is also specious. Sure, the main problem is with CrowdStrike's code, but Microsoft's GARBAGE design of Windows is the reason this is happening. If Windows wasn't such a pile of delicate rubbish-code, then CrowdStrike's bad code wouldn't have caused this, it would have been possible to quickly and easily diagnose it and quickly and easily disable it and quickly and easily revert the change and quickly and easily get back up and running. But no, Windows is TERRIBLE, so all anybody saw was a generic, unhelpful BSOD and no way to get around it. So yes, Microsoft is culpable. 4:42 Exactly. If Windows wasn't so poorly designed, it wouldn't be COMPLETELY UNUSABLE with a problem like this, it should have been easy to remotely disable the problematic module and remotely patch it.

[@satyamevajayate7263]

Such incidents will never occur on Debian stable or even Ubuntu server. They do not implement any changes instantly even for months.

[@GrandpaVince]

Old Vincey is an IT expert

[@Snoopyzell]

Clearly explained... thanks

[@wHiPpEdFoRJeOnJuNgKoOk]

Work has been a nightmare todaywith this down

[@misswaychannel9186]

Ive mistakenly deployed an update to all Estate before, the intern is acting non challant on twitter CrowdStrike's "Falcon Sensor" software was causing Microsoft Windows to crash and display a blue screen

[@sinewrites6581]

So many companies having automatic updates enabled on critical systems. It would be hilarious if it wasn't so sad.

[@KarlJones73]

Updates should be pushed out in progressive waves to avoid exactly this situation. Change Management challenges processes for blackout and recovery plans. There are more issues than just the update pushed out and breaking.

[@Not_First_Of_His_Name]

exactly.

[@Kreze202]

An update that automatically BSODs a machine post update is not a roll out problem, it's a QA/testing problem. Did no one test the update to make sure that, you know, it doesn't brick machines at the very least???

[@brilliantbuilder4228]

That happened to me when I was working on the Gateline at Staines Station today. For all the passengers that had E-tickets and paper barcode tickets, we had to keep.the widegate open.

[@amyhogarten5038]

Maybe don’t use the terrible Windows operating systems?

[@SoonGone]

Somebody's in so much trouble.

[@jamiebarnes3539]

It would have been advantageous to have a PiKVM connected to the network, as this would have allowed for remote resolution of the issue. About PiKVM A very simple and fully functional Raspberry Pi-based KVM (Keyboard-Video-Mouse) over IP that you can make with your own hands. This device helps to manage servers or workstations remotely, regardless of the health of the operating system or whether one is installed. You can fix any problem, configure the BIOS, and even reinstall the OS using the virtual CD-ROM or Flash Drive.

[@brianfunt2619]

Omg BBC News is on the screen behind him!! He's recursive!!!

[@zerothm1]

Practice Run

[@xlpeoplepower]

I can't have been the only one noticing the use of the old pound notes (and thus the use of old footage for illustrative purposes) at 3:48

[@ScienceChild]

You were congrats

[@beastmode00714]

They sent the wrong file. It was all zeros. True story.

[@strikezonefishingadventures]

CrowdStrike screwed up on a massive level. Their IT Project Management Team dropped the ball in a disastrous manner. The only way this happens is that the appropriate levels of testing were not completed on a live test system from end to end prior to implementation. What a disaster. Now CrowdStrike has a major financial problem and a major credibility problem. They can most assuredly expect that they will be facing lawsuits demanding tens of millions of dollars in compensation and damages for this mess from customers around the world. Question is, are their pockets deep enough to cover this, or are they now facing insolvency as a result? Either way, expect a pile of customers are also going to be looking at alternative platforms, so a safe bet that they are going to be losing a pile of customers as well. No matter how you slice this, there are a pile of senior staff at CrowdStrike who can expect a pink slip in the coming days for absolute and total incompetence!!!!!

[@BritishMoralHQ]

they should have done a version release..and clients have time to test it ...update right away should be illegal ...! or atlest cloudfront should do some ..track of all consumers of endpoint and test it

[@jacksoncremean1664]

it was a signature update that caused the issue, this is something you can't delay.

[@Kreze202]

​@@jacksoncremean1664 Surely proper testing/QA would spot something like this prior to roll out? It's baffling this update even got to being rolled out

[@imik3y]

So are you all ready for a cashless society with an AI overlord?

[@7_of_9]

Crowdstrike is a backdoor for hackers and they did get hacked two years ago 😅😂😅😂 who in the world trust a third installing files without testing it first

[@lorenzoluizdesouza1215]

Indian IT help desk please help us😂

[@HolloMatlala1]

We really need all that technical help desks right now

[@Maeni]

Stereotypical so ignorant

[@HolloMatlala1]

@@Maeni A Fact is a Fact: India dominates in IT

[@lorenzoluizdesouza1215]

@@Maeni and you're dumb af for not understanding a joke. They say if someone can't handle sarcasm their iq is as low as horseshit

[@tacorevenge87]

It’s windows fault. If windows did not have so many holes . Also why did windows allow a vendor handle updates at the kernel level as an automatic update without asking users? Second. Why do companies run windows in prod servers?

[@The_Savage_Wombat]

Linux saves lives. Show the person who wrote the bad code...hilarious to hear their side of the story.

[@icebergfitnessofficial]

My flight just canceled because of that.

[@StephenWilcoxon]

Fewer users on MacOS or Linux has nothing to do with it. It is likely similar updates went out to all OSs.

[@croaker6099]

And that's fewer non-commercial users. Back office servers that run everything important are overwhelmingly linux.

[@Mark-kh1ny]

There are several remote management tools that can restart an end point in safe mode. I spend my entire life telling companies to get their sh*t together. This doesn’t surprise me….. “It’ll never happen to me” syndrome……

[@joetran8798]

This is not a new piece of software. It is an update to an existing component.

[@a0um]

I’m pretty sure most of those computers don’t need to run a general purpose operating system so bloated and complex and therefore vulnerable to security flaws that it can’t guarantee being secure?!? They need to install another software to protect the operating system?!? That’s sick! The government should invest in small, secure, open source, operating systems… systems that are secure by design. But there is a lot of money sucked by inefficient insecure operative systems once the manage to lock you in! People, ask the government to run the infrastructure on open source systems!

[@Sloane1_4]

You missed one big component: Cybercriminals

[@a0um]

@@Sloane1_4 please explain

[@Sloane1_4]

@@a0um Most systems and software are designed with security in mind. But continuous need for features adds dependencies on top of dependencies and introduces the need for continuous bug fixing and improvements. Parallel to this, Cybercriminals are also continuously improving at the same rate and requires developers to be proactive and get ahead. This scenario with Crowdstrike is a good example: With Cybercriminals introducing rootkits and kernel level exploits, Microsoft needed to create what they called a Kernel PatchGuard to prevent modifications fo the Kernel. However, this also prevented anti malware solutions like CrowdStrike to gather telemetry from the kernel and effectively prevent other types of malwares from executing. So Microsoft introduces what is called Kernel Callback which allows anti malware solutions to create callback routines when an event happens. This however increases the risk of crashing the kernel, but has been very effective in detecting malware. So there are multiple streams at play here that will prevent having one open source secure system: The consumers need to for better tech. The cybercriminals looking to break them. The cybersecurity trying to stop them.

[@lesliegrace8360]

"Trust Me Bro"

[@Kushal6831]

Global IT outage

[@Jpex369]

This is why we don’t want See Bee Dee Sees!

[@ricnyc2759]

Winners: Linux and China. "Microsoft outage leaves China largely untouched as tech self-sufficiency campaign pays off" (SCMP, today).

[@hi-tech-guy-1823]

For Mission critical (Can be a risk to life) You Should have a Live Linux USB Flash Pen drive you can plug in and Run Linux Versions of your Software (Identical UI & GUI Just running on a Different OS ) That can Run Without a cloud & Internet so your Mission critical functions Can still Run - and just Export the DATA Back to the cloud when it back up ( I would thow in Local P2P Mesh + Grid parallel computing (Boinc - Berkeley Open Infrastructure for Network Computing) ) Even is is just for a sec ~ a few days ~ A Few months ~Till the END OF Time

[@hiro-p9y]

Please report on the misdeeds of the Japanese Ministry of Finance. The Japanese people will be grateful to the BBC.

[@mattmoreira210]

Go Horse strategy for the win!

[@squirreltismw8295]

And I'm doing a course in cybersecurity lol

[@m.rugsunshine]

Is the answer "hubris"?

[@theblackhand6485]

Stupid people thinking that an update is the best thing to do. Of course not. Besides that it is a way to controle the masses you always have to wait until bugs are neutralized. But lazy and ignorant people turn on their automatic update slider on their computer or iDevice. Then you get what happened in this video.

[@raaajit1]

modern real world example of butterfly effect , chaos theory

[@kolz4ever1980]

Lil bro just stringed a few cool works from books.. cool!

[@rogerc7960]

automatic coded by chatgpt?

[@makoto5071]

Is this only effecting machine running CrowdStrinke? or Machines connected to Machines at Microsoft running CrowdStrike?? I have a Tiny datacenter for my personal use with 20 rackmounted Servers running Windows Server 2016 on AMD Epic CPUs that had no problems at all. I update the security patches manually and have automatic updates turned off.

[@vitaliyrusinov4788]

Yes. Only the machines running CrowdStrike.

[@Kreze202]

It's a problem with CrowdStrike drivers on Windows machines (correct me if I'm wrong), so as long as your machine does not have CrowdStrike you're perfectly fine.

[@Sloane1_4]

Mate was just trying to flex my mans, no need to answer him

[@AlexR_44]

@@Sloane1_4 😂👍😂

[@samsung40_media87]

mercedes f1 team not winning this weekend

[@daledevernon56]

So who thought that a cashless Society would be a good idea.

[@Lynn22823]

💯

[@mimimi3440]

do they get persecuted for somthing like that?

[@天下為公1911]

Microsoft: it's really not our fault this time.😅

[@fantabulous6657]

I finished my work early on my Friday thanks to the outrage 😂

[@YusukeYamamoto-x8k]

missing pacifism is.... By the way, enthus, of that...Shizo Abe assassinated...certainly. Donald Trump depict that? as....golf freind? pacifism destination...all tye way to Japan? But neither English nor Japanese...in that threatening. "Pacifism is the opposition or resistance to war, militarism (including conscription and mandatory military service) or violence. The word pacifism was coined by the French peace campaigner Émile Arnaud and adopted by other peace activists at the tenth Universal Peace Congress in Glasgow in 1901.[1] A related term is ahimsa (to do no harm), which is a core philosophy in Indian religions such as Hinduism, Buddhism, and Jainism. While modern connotations are recent, having been explicated since the 19th century, ancient references abound." By the way

[@CPORD-ss2db]

Good for let us know this cannot be simply fixed by "patch auto update". But we still don't how and why it happened.😅

[@croaker6099]

One 35KB file was wrong and caused the OS to crash. The CS software can prevent bad things from executing but if it misidentifies some critical OS process or component as bad it can crash the whole thing.

[@CPORD-ss2db]

@@croaker6099 Thank you so much for the explanation. Guess CS did its job but overreacted.

[@Kreze202]

​@@croaker6099 Do we know if there were any kind of test or QA involved in this update? This seems like something that could have easily been spotted had someone tested it

[@_Tannu786]

Mere bhi computer mein aisa same aata hai toh iska kya matlab hai??

[@Twirlyhead]

Who needs Russian trolls with friends like these.

[@hdtravel1]

How could the whole world's computers be brought down by ONE companies software ? We are hugely vulnerable to the bad guys !!!

[@oofooflemon]

crazy

[@misterfunnybones]

The EULA said it might happen.

[@reddragon8]

Maybe no one should install updates on day 1 itself 🙄

[@MorningNapalm]

And then, because no one tested it on day 1, no one should install the updates on day 2...

[@Not_First_Of_His_Name]

it is not possible in crowdstrike. It is similar to windows. You need to know before an update even gets released and you should pause manually for all the endpoints, information displays at airports , etc.,

[@reddragon8]

@@MorningNapalm Well technically Crowdstrike should have tested before day1 😐 QA/QC anyone?

[@paperplane-db8qf]

Well at least not on disaster recovery servers so companies can failover quickly.

[@Mark-kh1ny]

@@paperplane-db8qfdepends how DR is commissioned and using what tool. For example if Zerto is used, you’ve already replicated the problem to the second site. Point in time restore? What’s the lost data worth? Complicated decision invoking DR

[@MushramboMu]

Would be great if someday Europe and UK can build their own big Operating system and a new search engine that surpass the current popular ones from outside UK and eu OpenKylin = china Windows = usa Europe= ??? Uk= ???

[@HaraldEngels]

Why there are still companies which are not running Linux to prevent the never ending issues with Windows?

[@KenBarberGeek]

Crowdstrike cut corners this time to everyone’s detriment. Also they have bloated ram requirements already (badly tuned jvm based agents), but didn’t listen to customers. Money makes companies do silly things. I hope they have learnt their lesson, but I doubt it.

[@hafirenggayuda]

Other comments also mention, they do this on thursday/friday (different time zones) which is VERY bad timing.

[@KenBarberGeek]

@@hafirenggayuda Friday releases are bad luck

[@hafirenggayuda]

Bad update and bad timing, I suspect bad work atmosphere there. This feels like last minute essay

[@Kreze202]

​@@hafirenggayuda This feels like an overworked dev pushing an update past the overworked PM who approved the update without proper testing or evaluation.

[@michael.j.mueller]

Your expert utters mostly bs. He is still very young to be an expert. I give him that.

[@theflipflapchannelcreatedb8160]

Remember the Y2K millennium bug? Look further 24 years later!

[@pedrosmmc]

DEI fingers on the keyboard?!

[@kennethworkman4740]

Computer says no lol

[@doctorpsylus]

test run ?

[@AfroMuggleKing]

PXE boot?

[@TomTKK]

Bitlocker

[@MiracleEkezie-rb4rh]

*Thanks for keeping us updated! I feel sympathy and empathy for our country. low income people are suffering to survive, and I appreciate Amanda. You've helped my family with your advice. imagine investing $30,000 and receiving $95,460 after 28 days of trading*

[@johntyler7024]

I began investing in stocks and Def earlier this year, and it is the best choice l've ever made. My portfolio is rounding up to almost a million, and I have realized that when a stock makes it to the news. Chances are you're quite late to the party, the idea is to get in early on blue chips before it becomes public. There are lots of life changing opportunities in the market, and maximize it

[@EmmanuelOgu-i8r]

What opportunities are there in the market, and how do I profit from it?

[@MiracleEkezie-rb4rh]

You can make a lot of money from the market regardless of whether it strengthens or crashes. The key is to be well positioned.

[@happinesschidera2293]

I would really like to know how this actually works.

[@MiracleEkezie-rb4rh]

All you need is a good capital, and the service of a professional broker, with those your investment will most certainly produce high yields.

[@samholden4171]

They probably blame the russians for it😔

[@thelastofthehitachi972]

backdoor "π"

[@lenniegodber7805]

This company is clearly incompetent. This kind of problem should have been picked up in pre release testing. The fact that it wasn’t says that anybody using this software needs to look for an alternative.

[@Sloane1_4]

They have been the market leader for years. Which is why everyone has them and this got so big.

[@jeovanneramos5066]

I honestly think they got hacked. But now the billions of dollars in lawsuits coming up.

[@theblackhand6485]

Now Jo Biden is a head in the polls?

[@souvikdas5662]

Win 11 or win 10 ?

[@The_Lost_Subrosian]

Skynet

[@GabroPlayz]

Skynet

[@Crouchy232323]

First the Euros defeat and now this, Starmer must resign

[@dr9gonkid20]

AI CODE AND SUPPORT FOR THE RIGHT WING

[@greenfly1264]

CROWDSTRIKE The clue is in the name guys.

[@QSZXQSZXQSZX]

Windows11 and bug

[@RoanGui]

What about the cultural enrichment in Leeds

[@MrChakyb]

It outage ?! Or Microsoft fak up .

[@rebelm.s4481]

❤

[@ZackYounguk]

BBC doing some of the rep management, instead of demanding better.

[@stabilis8895]

Leeds

[@GabroPlayz]

Leeds

[@codswallop164]

This didn't happen on a Friday by accident, look into crowd strikes founders and who they really work for

[@Mark-kh1ny]

More foil?

[@codswallop164]

@Mark-kh1ny Saved me from a weakened immune system recently and living in a majority dark city when I moved out at the turn of the century... So yes please 😉

[@ivanz8788]

Microsoft OS is to blame. no other OS will fail like this from any type of software as it´s designed well from the beginning.

[@Mark-kh1ny]

Wrong, root kit scanners can and will stop any OS boot including Linux.

[@Kreze202]

If you wanna blabber on about nonsense, at least try to make it seem like you know what you're talking about next time, yeah?

[@metapol118]

crowdstrike strikes the crowd