CrowdStrike The Day the Earth Stood Still

July 19th 2024 is not a day that we will forget. A simple software update caused chaos around the world effecting billions of users and millions of computers, disrupting everything from air travel to financial services and medical care. The big question is how it was allowed to happen? Clearly this is where change management procedures clearly failed causing millions of computers around the globe to blue screen (BSOD). In this thoughtful video I’ll discuss the what happened, how it happened and what you need to do to recover from this issue.
For more on me visit me at www.Andymalone...
Looking for more? Why not sign up to my Patreon page / andymalonemvp

[@wschloss]

Well done Andy! Nice to see how our OLD basic knowledge comes to be seen in action , before going behind a script or App 😊 .

[@ZX48K]

A bit more information on the CrowdStrike BSOD issue: The problematic driver that caused the crash was actually empty; the file contained only zeros. Being a Windows kernel driver, it caused a null pointer crash when Windows attempted to load it at boot time, leading to a shutdown to prevent further damage to the OS. CrowdStrike operates at a low level within the kernel to detect potential threats. However, this incident raises questions about Windows architecture and whether there's a better way to integrate third-party products in a sandbox outside the kernel.

[@AndyMaloneMVP]

You make some great points here and I completely agree agree 👍

[@joem2469]

Informative indeed! I couldn't imagine applying this patch to 100s of servers without an automated patching solution.

[@dragonflysdiscoveries7567]

Excellent, Love it.

[@gardnerfred4]

Thanks Andy! My office was affected by this and we used the same process to recover. Unfortunately, as you stated, we had to do each machine one by one. No script is available that I know of at this time. Thanks for your continued wealth of knowledge.

[@AndyMaloneMVP]

Whew I'm glad you managed to resolve everything :-)

[@e125]

MS said there was 1% win computer that was affected. Ofcourse, it was a lot, but clearly not "world staid still".

[@jasonbevan6275]

After watching a technical overview from Dave Plummer on Dave's Garage I was astonished that the code was able to be inserted directly into Kernel Mode and wonder whether Microsoft were aware of the mechanism for bypassing driver signing. Malware protection is very important, but Microsoft shouldn't allow anti-virus vendors to be able to insert untested code at such a low level.

[@AndyMaloneMVP]

Crazy eh!

[@than10]

All malware authors would totally agree with you! Kick out AVs from the kernel!

[@harrylumsdon6773]

Another case of IPO greed???

[@jacksongarcia8828]

THE DAY THE CROWD WENT ON STRIKE.

[@lc7798]

Loved WinNT, stable brick house

[@AKJordansKids2009]

How many crowdstrike customers uninstalled the crowdstrike app? I’m lucky we don’t use crowdstrike but if we did - we would be a former customer.

[@fbj73]

Thanks for the video, Andy. Our agency was not affected. My only guess on this is that this was caused by human error, meaning that someone at CrowdSource was either off that day and did not have a backup person to do the QA/QC. It was like the story of the single bolt on the Eiffel Tower that came loose and caused the tower to collapse.

[@AndyMaloneMVP]

Absolutley!

[@dschleider]

Hi Andy. My company wasn’t affected but it might as well have been. The solution you showed might work well for a physical computer or even an on-prem hypervisor. For an Azure VM I think your best bet is to recover from the latest snapshot. It would be nice a video showing that, or any proper solution for Azure VMs that you know. Thanks for your great videos!

[@AndyMaloneMVP]

Great point. The article that I showed in the video actually has multiple scenarios of which VM’s are one along with bit locker recovery. View the article I learn.microsoft.com.

[@ying-ym8ut]

2:07 Misinformation! It didn’t it push through windows update! It was a 3rd party update through CrowdStrike. It has nothing to do with windows. Any computer not running CrowdStrike would not get this issue. The update just happened to be a windows version this time. CrowdStrike had been problem with Linux version as well, but just because not that many computers running that combination of products that time, so we don’t hear a lot of report of it.

[@prt2v]

Always worthwile, Thnak you. Father Andy (Your outfit looks like an Anglican Vicar)

[@AndyMaloneMVP]

Hehe that's what my wife said this morning :-D

[@michaelwitzsche3392]

“Microsoft’s security culture was inadequate and requires an overhaul” In a recent report, the Cyber Safety Review Board (CSRB) found Microsoft’s “cascade of security failures” resulted in a catastrophic breach. Choose a provider that delivers without compromise. Choose CrowdStrike. Yes, we can say they delivered without compromise. 😉

[@silvercityweb]

Good video Andy. I'm so glad I'm retired. I'm sure it was "all hand on deck" to patch the few thousand computers where I worked not counting the emergency flights, oh yea no flights, to the other facilities to patch them. When I retired, we used electronic medical systems to know what drugs to dispense when. I can hear the lawyers smacking their lips getting ready for the lawsuits. This ranks right up there with the Boeing neglect with the 737 max. I'm sure in the coming days we will hear of people dying because dispatch systems failed, medical records were not available, etc.

[@AndyMaloneMVP]

100% agree!

[@Kvantum]

I think a lot of big companies are going to look at monolithic AV applications and realize that we can't universally trust them. We might have to have companies start keeping a minimum viable backup running on a different AV solution. If Crowdstrike can even survive this as a company (or if they even should), they have to massively revamp their QA process.

[@AndyMaloneMVP]

I totally agree.

[@prebsi8603]

Thanks Andy... I got a few questions - hope someone can answer😊 1) Guess this only hits Crowdstrike customers, or…? 2) How was this faulty update offered to the world? As a Windows Update, or as a Crowdstrike Update? 3) What about Change Management at those affected customers? Don’t they test ALL updates for a very small number of computers, before they patch on all computers? Thanks for your help everyone

[@than10]

It was a update directly from CroudStrike, and, as i learned, a 'forced' update outside the control of each IT department. According to Microsoft it was about 8.5 millon PCs, I would double it to account PCs that have their telemetry turn off. It's less than 1% of the Windows PCs worldwide, but Croudstrike is an Enterprise solution, hitting hard big Entrerprises and having the biggest impact.

[@prebsi8603]

@@than10 Thanks :)

[@zxrenew5642]

Absolute tripe, What happens if your Hyper-V hosted server is affected? What happens if the device in questions was in the middle of Windows updates. Not as simple as M$ fanboys make out. If the PC has been trying to reboot and needs a disk check. Loads of issues with this around windows hosted visualization is not being discussed. Never run VM's on a windows OS. That is more than likely why GP's cannot make appointments still. A bad windows update can take down all their windows VMs. People need to move away asap.

[@AndyMaloneMVP]

And this is why I use a Mac 😉

[@eji74]

Unfortunately I know what happens when a Hyper-V hosted server is affected. My group alone at the IT Support company I work for had to deal with at least 30 of them, and we still have one that we haven't been able to get onsite yet to fix, therefore that company's employees still cannot access their network shares or authenticate in order to connect to their VPN. It's been an absolute nightmare.

[@joelmamedov404]

Critical systems should not run on windows. Back to Unix, mainframe .

[@AndyMaloneMVP]

The truth is there is no 100% secure system not even even UNIX or Lennox

[@urafishhead]

Audio issue?

[@AndyMaloneMVP]

Where I've just watched entire video and it's fine. Check that you have not muted video.

[@ZzeezZ78]

No issues here

[@urafishhead]

Must have been a glitch in the matrix. I clicked as soon as you released, other videos were fine. I went back and forth several times before commenting. Maybe I was "too quick on the draw" lol. Sorry for the fire drill!

[@ScottiRaffaele]

TYPO --> DIR = DEL

[@AndyMaloneMVP]

The command is correct acording to MSLearn. Once located THEN Del

[@supriyochatterjee4095]

Never ever use any other antivirus or security softwares in any environment whatsoever it is home or office or small office or small businesses or enterprise or big other than Norton or AVG or AVAST or Bitdefender or McAfee or Kaspersky or Checkpoint Zone Alarm or ESET or GDATA that's that.

[@jojolization]

Agreed. how about the SentinelOne? i sometimes received the introduction from the supplier that it is a nice product in recent 1-2 years, but I don't have a chance to use it.

[@supriyochatterjee4095]

@@jojolization Not sure but have heard it triggers lots of false positives not sure though.

[@BDBD16]

ROFL those are all trash.

[@harrylumsdon6773]

Troll much?

[@supriyochatterjee4095]

@@BDBD16 You don't know anything about antivirus or security softwares

[@kabyson]

+

[@raginald7mars408]

I stay with Windows 7 forever! Micro Soft = Self Titanick

[@AndyMaloneMVP]

Only problem with this is it’s extremely vulnerable. If you’re not on the Internet, you’ll be fine but if you use it for browsing, forget it you’re gonna be in real trouble.

[@enadegheeghaghe6369]

You are asking to be hacked. LOL