Crowdstrike chaos: What are the lessons from the world's biggest IT failure?

It’s been an extraordinary day of cancelled flights, disrupted businesses, problems for healthcare and TV stations not being able to get on air (ahem).
And all because of an update for Microsoft Windows.
So what caused one of the biggest IT failures ever seen - and what do we know about Crowdstrike, the company which released the update?
Ali Fortescue’s in for Niall to discuss it all with our science and technology editor Tom Clarke and data and forensics correspondent Tom Cheshire.
For further background from Sky News, you can read Tom Clarke’s analysis on the questions Microsoft now has to answer here (news.sky.com/s...) .
Producers: Soila Apparicio, Rosie Gillott
Editor: Paul Stanworth

[@jrtstrategicapital560]

One lesson learned: CASH REMAINS KING when systems fail....DIGITAL CURRENCY will always be vulnerable....

[@JeffEbe-te2xs]

More and more business don’t take cash and employees don’t know what cash is

[@alienfish8521]

@@JeffEbe-te2xs Ah well, they're missing out on business then.

[@Demons_of_the_night]

Cash and offline paperwork in a non digital office .

[@dodzb7362]

The lesson here is to never push a new update without TESTING. It could be either Crowdstrike or Microsoft fault for not having more time to TEST their queues!!!!

[@jw8160]

@@dodzb7362 Also never update on Friday.

[@DawnofFab]

This mess on a Friday! I work in IT, when stuff like this happens on a Friday it takes longer to resolve as offshore teams are off, it's their weekend already and to get a hold of them becomes an issue and causes delays

[@jw8160]

At the companies I worked at, our policy was never do updates on a Friday. 🤦‍♂️ We did updates Tuesday, Wednesday, or Thursday.

[@Woodzta]

Read-only Friday. It's an unwritten rule for us here.

[@TomNook.]

Yup, never push to prod on Friday. Shocking that a multi billion dollar company that is IT centric doesn't practice this

[@TheMediterano]

I still can't conceive the following points: 1) How were such a widely used product's automatic and manual test processes not able to catch the bug? 2) Don't they adopt canary or blue/green deployment strategies in order to reduce the blast radius in such large roll-outs?

[@rezwhap]

That, and why so many clients allowed themselves to be subject to such a blast radius. This is not a ‘black swan’. Vendors pushing out bad updates is an entirely predictable, even common, event. Plans and mitigations should have been in place.

[@sythelic]

stop hiring cheap QA and have a proper staging lab !

[@Paws4Thought669]

Use cash and do not travel with the sheep

[@TarolinSchofield]

Anyone that has to follow what someone says is a sheep and that includes anyone that follows your comment , make your own mind up cash or card who cares.

[@frankriquelme4623]

And they want to go Digital let that sink in

[@pauleades9037]

Demonstrates the importance of resilient business continuity planning and the dangers of over reliance on digital systems without the analogue backup of critical systems.

[@maneshipocrates2264]

The lesson should be to completely move away from windows in all its forms.

[@benmanutd2]

Not just windows, diversity is key It'll be the same if we all use MacBook

[@maneshipocrates2264]

@@benmanutd2 Any secure open source OS is better than Mac or Windows.

[@Tigrou7777]

Linux or MacOS would have fail the same way if they would have some buggy Crowdstrike driver loaded in their kernel as well. Modern Windows is very stable.

[@Bonjour-World]

@@Tigrou7777 LOL .... "Modern Windows is very stable" ... ROFLMAO !

[@Tigrou7777]

@@Bonjour-World 99% of blue screens came from faulty hardware (eg: bad memory sticks, gpu whatsoever) or bad drivers (as Crowstrike case). I got AMD and Intel platforms since almost 10 years (Windows 10 - 11) use computer everyday and don't remember last time I saw a bluescreen. The Windows 9x era are long time gone.

[@keithbessant]

Where would be without those so-essential updates.

[@TomNook.]

There's such delicious irony in spending thousands using a Cybersecurity company to protect your systems from going down, when it's the reason for your systems going down

[@frankm7707]

CrowdStrike is conducting another round of global production testing.

[@LandGrabbingIndia]

Quality checks.

[@nirfan2020]

As I watching this news, KZbin still distrupted me with its ads...😁

[@WalkiTalki]

Why do so many systems not have redundancy? Its not like Microsoft hasn't been less than good enough since the 80s. Especially healthcare and banking. Those should be tripple redundant. As the world globalizes humans become lazier and less responsible. The movie Idiocracy was not meant to be a goal for the world.

[@frankm7707]

Looks to me they didn't do proper testing before deploying updates.

[@Woodzta]

Redundancy can only do so much. We had our servers go down with all sorts of transfers of data and live customers who lost access. We can't just restore 200 servers from a backup at once with our file systems using shadow copies. Most of them were simply fixed manually. I had to give many users the encryption key before talking them through removing the sensor update, and again redundancy can't fix that any faster than manual intervention.

[@RobRijnbout]

This clearly shows how dependent on technology we are..where is plan B now guys?

[@volvojohn9036]

MICROSOFT LAID OFF THEIR DEI TEAM 5 DAYS AGO, IS DEI HIRES REACTING TO THE WRITING ON THE WALL?

[@LandGrabbingIndia]

Don't put scammers in charge😊, especially those working at the scam centers.

[@JeffEbe-te2xs]

Don’t put all your eggs in one basket

[@jammingj8187]

Just say no to digital currency 💱

[@elye3701]

This is howY2K would have looked like.

[@kevindoom]

another lesson is that IT people are people too , stop only thinking of them when things go wrong

[@gregvisioninfosoft]

between this software upgrade disaster, copilot spyware, forced hardware obselessence with Win11 install requirements, and NO LONGER ACTIVATING fully paid retail packages of Office 2000, 2003, 2007 and 2010 - time to abandon this cr*ppy monopoly.

[@darrenvoysey7734]

another lesson learned...despite his wealth and philantrhopy...cometh the man, cometh the moment....i remember where where i was when i heard that basically, the outernet is down...

[@mjlewis104]

Funny how it didn't effect Russia!!

[@Verdent777]

Biggest lessons. Always have cash. Don’t let the government force digital currency!!

[@MicheleLLOYD-bk2mt]

A bad change that wasn’t tested. Recovery procedures that weren’t tested. Sloppy illdisciplined outfit……….

[@lilpiepie]

This is what happened when IT is outsourced to India. I work in IT and deal with many Indian developers. Their work quality really sucks

[@maneshipocrates2264]

It should not matter. The last time I checked, there is no such thing like an Indian OS, programming language etc

[@Woodzta]

CrowdStrike is as big as it is because it is amongst the best. It also needs low level, real-time access to the machine so anything like this or an anti-virus can cause such issues. It has absolutely nothing at all to do with "outsourced to India".

[@wildandbarefoot]

CROWDSTRIKE picked a fight with china a few years ago .

[@MykhayloMaksymchuk]

As a person working in IT and mainly on Microsoft tech I did noticed that Microsoft becoming worst and worst with QA. I know it's not their fault, but they need to hire back QA testers!

[@jammingj8187]

CASH IS KING !!!

[@synchronistory]

Here's what's going on in the much larger context. There's a remarkably intense planetary configuration in the astrological and astronomical sense, that indicated a series of intense energetic dynamics occurring during around this very period. From seismic activity to technology to overheated tempers to geopolitical and very "head"y leadership dynamics (e.g. "the bullet") ... even to meteoric occurrences (the extraordinary meteor crossing the midday sky in NYC the other day). Pam Gregory is a world-class astrologer and many others have also noted this configuration in the July charts. In other words, it's bigger than just what the media understands. It's actually "heavenly", but also worth noting and not overdramatising. Try to surf these energy waves with as little drama and as much compassion as possible because Earth is alive, and at the affect of cosmic vibes :-).

[@John_Bradbury]

I wish humanity had got beyond medieval superstitious ideas.

[@DawnofFab]

@@synchronistory are we saying Mercury is in retrograde? 😆😆

[@synchronistory]

@@DawnofFab 😅something sure is ...