CrowdStrike Update: Latest News, Lessons Learned from a Retired Microsoft Engineer

Dave brings you up to date on the CrowdStrike IT outage and considers its broader implications. For my book on the spectrum, see: amzn.to/3XLJ8kY
Follow me for updates!
Twitter: @davepl1968 davepl1968
Facebook: davepl
1. Introduction to the CrowdStrike Falcon IT Outage:
• Overview of the recent CrowdStrike Falcon IT outage and its impact on various industries.
2. Technical Details of the Outage:
• Explanation of the faulty sensor configuration update and how it led to system crashes (BSOD) on Windows systems.
• Specifics about the corrupted “Channel File 291.”
3. Impact and Response:
• Description of the scale of the outage, affecting approximately 8.5 million devices worldwide.
• Steps taken by CrowdStrike to deploy a fix and provide mitigation guidance to affected customers.
4. Previous Issues with Linux Systems:
• Recap of earlier incidents where CrowdStrike updates caused crashes on Debian and Rocky Linux systems.
5. CrowdStrike on macOS:
• Discussion about CrowdStrike’s security solutions for macOS and their use of Apple’s System Extensions.
6. Kernel vs. User Mode in Security Software:
• Analysis of why kernel-mode access is used by CrowdStrike and the associated risks.
• Historical context of kernel vs. user mode in Windows drivers.
7. Regulatory Challenges:
• Narrative on Microsoft’s attempt to introduce an API to prevent such issues and the regulatory hurdles faced from the European Union, which deemed it anticompetitive.
8. Conspiracy Theories and Broader Lessons:
• Overview of conspiracy theories that emerged around the outage.
• Lessons to be learned from the incident, drawing a parallel to the Tylenol crisis management.
I'm long since retired, and any opinions are mine alone; not a spokesperson!

[@hotzemusic]

"Standing around with their disks in their hand" was such a great quote lol

[@markzempel]

"Signed by Elvis himself" is a pretty good too 🙂

[@genericasian]

I also liked the statement concerning conspiracy theories... Sheer incompetence is a better explanation than any conspiracy theory. :)

[@juanmacias5922]

@@genericasian Occam's razor stated more eloquently. :D

[@vcv6560]

Of course it's looking for the drive slot Regina.

[@bravodefeated9193]

​@@genericasian wheres the fun in that

[@christopherg2347]

This was a obvious Crowd Strike procedure error. What _should_ have happened: 1. The update is sent to automated testing 2. The tests return with crashes in 2-5 minutes 3. The guy that made all-0 file would be very embarrassed for that mistake. 4. The code that apparently doesn't check the input would have been flagged for improvement 5. Nobody outside the dev company would have heard of it. Just another day in the office, doing Kernel level work responsibly.

[@eliotmansfield]

I’m guessing the original update file was good and contained a valid update - but the process of packaging and distributing it thereafter changed it to contain zeros - which as we heard wasn’t anticipated

[@crapmalls]

What should have happened. Company tests own equipment before imported update sent to machines

[@harryganz1]

As a developer. I think it is very unlikely that this wasn't tested. No developer is committing a file of all zeros and nobody reviewing it will miss it and no system that tests it would pass it. Most likely the all zero file was generated by an automated system from valid source code and the issue was with the automated system. My guess is that there was some sort of configuration drift between the staging and production environment, so that the build was fine in staging and somehow produced the broken file in production.

[@argledotorg]

@@eliotmansfield CI testing catches that by doing a test deploy of the final package into the test environment. But test environments cost money, and generally also involve QA engineers who also cost money, and if there's one thing people are taught in business school it's "don't spend money"

[@davefb]

nearly spat out my lunch at the 'the file was 0' bit.... WHAT????? Feels incredible to not have any check for null input?!

[@spidalack]

"The code just raw dogs it and hopes for the best" You almost caused me to spit out my drink

[@orijimi]

I suppose you're not acquainted with how widely the term has been getting applied as of late, starting with the concept of taking flights and intentionally depriving yourself of comfort while doing so.

[@moffix]

Better than bareback! gotta love Dave

[@pieterdebie4162]

Didnt expect that one 😅

[@Tom-xg3ub]

Same here was not expecting the raw dawg line lol

[@robertcross5794]

"and hoped for the best but it is in life as it is in software you can be lucky sometimes but if you come to rely on luck it will eventually run out. " I also like the rest of it. Wise words.

[@patrickhoveling]

“It choked, turned blue and died“ I love the way you explain these technical items in such a clear and funny way.

[@tepidtuna7450]

Laughed so hard I had to pause the video. Brilliant !

[@InfinitaShop]

@@tepidtuna7450 THIS MAN IS A HORRIBLE SCAMMER. DONT COMMENT AND UNSUBSCRIBE

[@geoffreybassett8343]

Brilliant!

[@stephen.jenkins]

Yeah me too. While this all 0's thing is no laughing matter, I couldn't help but laugh at His explanation of it. Funny.

[@doneill64]

As a retired Microsoft guy - the real veteran developers at MS can be absolutely hilarious. They are over the top intelligent, and there is so much low hanging fruit. Dave is but one great example. And oh god the office pranks, legendary.

[@BFLmouse]

Way back in high school my programming teacher taught us that our user interface code had to be bulletproof. A 'bullet' was then defined as a hyperactive ten year old at the keyboard. One of the first tests he would do is to mash both hands on the keyboard. My program was expected to handle that gracefully. I've been using that philosophy on everything I do ever since.

[@jefferysmith5921]

A-men!!!

[@HamiltonKsos]

Ctrl-q is a bad choice for an exit key combo as it's easily done by a cat on a keyboard

[@Miwna]

@@HamiltonKsos I've done it many times when I wanted to hit ctrl + tab instead.

[@lPlanetarizado]

i remember one professor taught me something similar: "treat all users as idiots"

[@savagesarethebest7251]

Input fuzzing tests are quite common.

[@XH13]

My first C course back in school. The prof asked us to write a program that asked for a date d and a number n, and return the date d+n days. Beginner stuff, but you have to start somewhere. When the first student completed the assignment, the prof went to their computer and when asked for a date, he typed : "did you trust the user input ?" Program crashed, lesson learned.

[@juanmacias5922]

OMEGALUL what a savage.

[@sobolanul96]

We made a simple inventory stock keeper in excel and vba at work. It used a touch screen and a barcode scanner to add/retreive parts. Dumb simple. After the interface was finished we went around the offices with the thing asking people to try and break it. We had to modify the interface and add locks at least six times until it became stable. People always find a way.

[@connormclernon26]

@@sobolanul96the harder you try to idiot proof something, the Universe creates an even bigger idiot just to prove you wrong.

[@scottjanssens9978]

My prof would always type in "hello". We started checking for that and returned, "Hello, Dr. Brubaker".

[@ErazerPT]

@@sobolanul96 Yeah, that happens 'naturally'. I call it a 'subconscious process' where the people making the product subconsciously avoid not doing TheRightThingTM, because they're instinctively afraid to break it. Anyone outside that product team couldn't give two tweets less, so they'll go at it with glee. Fun story, a dev friend of mine used to use his granny for testing, because the old lady had an uncanny knack for doing BadThingsTM to the products :D She was also a great UI/UX tester, because she was totally non tech savvy, so if she could use it, most people could too.

[@roycsinclair]

Most telling is that CrowdStrike has already done the same thing to smaller code bases, i.e. those two Linux versions mentioned and has obviously not fixed the internal processes that allowed corrupt updates to be released. Thank you Dave for getting more of the story.

[@trollol_]

Go Woke Go Broke

[@roycsinclair]

@@trollol_ This really has nothing to do with the political positions they have taken though taking political positions is stupid and an extremely rude slap in the face to their employees who don't hold those positions. That doesn't mean individuals in a company shouldn't be able to participate in politics but it should never be a company direction or policy.

[@anupambiswas140]

To tell the truth it isn’t as easy as it sounds to update the process for some uncommon systems, as it’s ok sorts for the code to break at some very uncommon systems but what I am mainly amazed it that they missed testing in such a major operating system.

[@roycsinclair]

@@anupambiswas140 Linux is not uncommon and should be consider a major operating system. That Crowdstrike broke Linux back in April should have rung alarm bells inside Crowdstrike and among it's customers though. It pointed out a failure withing Crowdstrike that allowed improperly tested software to be released.

[@misham6547]

Linux runs the Internet

[@whatisthis__95]

"I try to never attribute to malice that which can be sufficiently explained by incompetence." Greatest statement ever

[@nickwallette6201]

A rare example of rational thought in a world of unconditional jumps to conclusions.

[@dr.chives]

Hanlon's Razor. There's even a wikipedia article about it. 😀

[@pauldelvigna1823]

No doubt, the go-to explanation should be incompetence until proved otherwise.

[@ruppert5134]

Not in crowdstrike s case.... Dimitri Alparovich crowdstrike used to work at McAfee where he and his team of urkrainian hackers criminals wrote VIRUSSES to sell MORE anti-Viral McAfee software....Yes the rumours are TRUE !!!! Alparovich is STILL at crowdstrike doing bad shit behind the scenes.... He needs to be arrested....

[@blairhoughton7918]

I literally read it here as he said it onscreen...

[@lindoran]

There is so much value added here. You can really tell that Dave has had to present -- "ok, what had happened was" to a design review before. The bit about standing around with their disks in their hands ... that was priceless sir :D

[@uzaiyaro]

We had our own Tylenol scare in Australia. A woman sent threats to Arnott’s, a biscuit company, saying that she had contaminated a line of biscuits in some states (if I remember correctly), and arnott’s response was to recall every line of biscuits in every state immediately. They also published full page ads of the police reports, and any updates that had come from the police. They instructed people to not eat any of their biscuits. When they found the woman and dealt with her, Arnott’s didn’t lose any market share, in fact they gained market share because of their demonstrated trustworthiness. It has also become a case study in how to deal with a crisis in business.

[@alerigali]

We had the worst of all scams in 2021, worldwide, with the experimental products side effects that ended killing more people than the illness supposedly wanted to prevent. To make things even worse, those products didn't prevent the virus spread, complications or anything, at all. The genocide is still unpunished and labs made several tons of money in the process. Australia was one of the worst, but please don't believe me, check the total excess deaths in 2020 and later years

[@robertcruz7898]

Good on them!

[@ChrisCandreva]

Regarding the Linux issues with Crowdstrike: The important difference there is those were problems with the falcon-sensor itself. Those were discovered through the normal server patching process. Since we patch test machines first, we were able to find the problem before it hit any production servers. Having the problem in an automaticly downloaded channel update is a big difference.

[@quietprofessional]

That's my theory as well. I've been on a number of Change Advisory Board meetings run by two different clients and the change sponsors always detail whether an update has been tested in the test environment before it goes to the production environment pending change approval. And to get approval there needs to be some fallback plan should the update encounter errors midway through the change window or is causing the errors to a specific service or application to which a change record number is attached to the master incident ticket after preliminary investigation. Obviously no one in Crowdstrike had considered to put driver file "291" in the testing phase nor was their CAB involved to give the no-go. Funny how Crowdstrike on that July 19th morning took on the meaning of striking down many Windows-operated machines (one of my two corpo laptops was hit with the bug and it was a Windows 11).

[@EwanMarshall]

this still causes questions of whether that should have been caught in the internal testing before sending out and raises warning flags.

[@GoogleDoesEvil]

Actually, with RHEL/Rocky Linux it was because CS Falcon ran into a Linux kernel bug. That's why the fix was an update to the kernel. Still, should've been caught in testing and delayed.

[@mountainshark2388]

TBH, Linux doesn't need any outside help when it comes to crashing and dying.

[@r3pek]

@@mountainshark2388 love how to talk about what you don't know :)

[@Hamiltron_]

“Their code just kind of raw dogged it” 😂😂 One of many quotable moments in this video. Bravo, you 56 year old gem of a nerd.

[@richardc020]

The deadpan matter of fact of every line is just totally enthralling. It brings drama to something folks think isn’t dramatic but easily is, our daily lives rely on it.

[@hepcatliz]

nope, it was just crass

[@Hamiltron_]

@@hepcatliz He’s making light of what realistically was a tough time for a lot of folks. I’m sure a little light hearted joke would be well received considering the chaos they had to endure.

[@utp216]

I liked that part! 🤘

[@m4rt_]

"Never attribute to malice that which is adequately explained by stupidity." - Hanlon's razor

[@KeyError]

“Cockup over conspiracy”, as I’ve heard it

[@TheMontablac]

Any sufficiently advanced incompetence is indistinguishable from malice

[@JLProPhoto]

"Any sufficiently advanced stupidity is indistinguishable from malice." - One line used in an article on the Crowdstrike fiasco (paraphrased from Clarke)

[@robknight9406]

Any sufficiently advanced malice is indistinguishable from incompetence

[@invertedpolarity6890]

Yeah but typically malicious people (politicians) are also stupid. So you get the same result so the intent doesn't matter.

[@WaltScrivens]

Thanks, Dave, especially the excellent review of the Tylenol crisis. I was working for J&J, teaching Mr. Burke to use his new IBM PC at the time. He was a marvelous leader and manager!

[@alexandermonro6768]

It looks like Boeing didn't take the opportunity to learn from Tylenol. Let's hope they're learning now.

[@Dawnjohnston-c3]

Falcon represents a significant burden for developers, acting as corporate security bloatware that penalizes them for basic tasks like compiling code. This results in lost productivity and siphons money from companies without providing a tangible return on investment. Executives often overlook these issues until a major crisis occurs, yet the decision-makers responsible are rarely held accountable. Consulting a financial advisor could help companies evaluate the true cost and financial impact of such security measures on their overall operations and productivity.

[@ErnestZDodson]

They should have licensed that API.

[@ElvisSirinBo]

Rising prices have affected my intention of retiring at 62, working part-time, and building my savings. I'm worried about whether individuals who weathered the 2008 financial crisis found it less challenging than my current situation. The stock market's volatility, coupled with a reduced income, is making me anxious about having enough for retirement.

[@ElvisSirinBo]

@@Victoria-io7qb impressive gains! how can I get your advlsor please, if you dont mind me asking? I could really use a help as of now

[@captianmorgan7627]

And yet more bots.

[@dijoxx]

Financial advisors don't have the level of technical knowledge to understand such problems yet alone propose solutions to fix them.

[@jimtussing]

Hi Dave. I did a lot of Windows development starting back in the NT days and just found your channel. Have been enjoying with much nostalgia. Highly informative, extremely well articulated and fun to listen to! Congrats on a great niche and rock on!

[@IanSinclair77]

My good man, you're amazing. I work with one of the biggest and highest ranked Universities (accoromg to world rankings). I shared your last video to many others in our IT, including my Director. Today, I heard him (essentailly) quoting your video, which I'd sent him a few hours earlier and told him to watch. He sounded very smart to everyone else, and yet I knew he was summarising what he'd learned from you. Some people in this world are smart... And then there is the rest if us... Keep it.uo my man. You're making the rest of us smarter everytime you post a video.

[@thomygoldman324]

My new favorite quote I heard from this situation is "Any sufficiently advanced incompetence is indistinguishable from malice" and I felt that

[@raygunsforronnie847]

Arthur C. Clarke approves... ;)

[@he8535]

Crowd strike, striking the crowd. Who would have guessed

[@jonk1446]

Hanlon's Razor :)

[@Acorn_Anomaly]

@@jonk1446 Sort of. This is Grey's Law, a corollary of Hanlon's Razor. (Hanlon's Razor is "Never attribute to malice that which is adequately explained by stupidity.")

[@jonk1446]

@@Acorn_Anomaly you're right, I was ignorant. Thanks for correcting me

[@marcschafers6295]

Dave, did you write that line: "A whole lot of tech with their 'disc' in their hand waiting...." That was perfect! lol This was a great video, Dave, and was funny in more than a few places. Your coverage of the issue is very astute and I can tell you're "The Pro" in this area. Thanks for the vid.

[@brucewilliams6292]

Came here for a Crowdstrike bash; stayed for the great explanation of how to handle crisis management should be done! Great video.

[@aaronriggan2373]

"While I would never dare to question the wisdom of printer designers..." OMG, the SNARK!!! I'm crying! 🤣🤣🤣

[@LS-jv4uh]

Your 2 videos have been the best explainers on this topic I’ve seen. Thank you!

[@mfgxaldy]

Glad to see a follow-up so quickly. Thanks Dave.

[@bydandie]

The most considered analysis of the Crowdstrike outage I’ve seen yet, thanks 😃

[@floydchusset3143]

This is a very well thought out, presented and useful video!! I recently sold my condo for $400k and i want to invest the money in the stock market. However, it appears the market is at an all-time high. Should I invest elsewhere or wait for a market correction?

[@majidcoper]

The stock market is risky But staying on the sidelines is riskier. Missing the next bull run will be far more costly to your long-term wealth than getting in at the "right price". Consult a financial advisor if you're unsure how to proceed.

[@ryanthompson8256]

You're right, I and a few Neighbours in Bel-Air Area work with an advisor who prefers we DCA across other prospective sectors. Instead of a lump sum purchase, Following this, my portfolio grew 37% in the last quarter

[@grinjohnson6452]

Mind if I look up your advisr please? I've worked in real estate for over 25 years and have neglected a major stock portfolio. This served me well when I was flipping and renting houses, however I need a different plan now

[@ryanthompson8256]

I've stuck with the popularly ‘’Laura Grace Abels” for about five years now, and her performance has been consistently impressive. She’s quite known in her field, look her up

[@grinjohnson6452]

thank you for putting this out, it has rekindled the fire to my goal... was able to spot Laura after inputting her full name on the web, she seems highly professional with over a decades of experience

[@raymarquardt5157]

Nearly all, if not all, of my comp sci professors _constantly_ hammered into us "Never trust input will be what you expect" and "Always check your d**n pointers before use" - either of which would have prevented this. What I find more concerning is they appear to have pushed an update without testing it first, and that they rolled it out everywhere, rather than using _any_ sort of phased rollout.

[@djulianerenbourgh4969]

And, adding an insult to injury, they effectively bypassed the Microsoft safeguard measures

[@kevinmcfarlane2752]

Read their update today. The bug was in the component that did the checking, which had previously been checking successfully.

[@scottrabinow2773]

@@kevinmcfarlane2752 Makes you wonder how well tested their testing tool was, doesn't it?

[@RonJohn63]

@@scottrabinow2773 Quis custodiet ipsos custodes?

[@threeMetreJim]

That isn't what they say on their blog.

[@Notdave29]

The bigger issue is that much IT infrastructure relying on a single application. I ended up going to 6 gas stations that day trying to fill up my car before I even found one that could accept cash. That’s a major national security risk, even without a hostile actor. It will only get worse from here.

[@MichaelDeHaven]

Yep, I'm less worried about AI than I am centralization like this.

[@AntiAtheismIsUnstoppable]

@@MichaelDeHaven This is both. It's centralized AI detection of malware.

[@effexon]

ah that side of story makes me lean to conspiracy side as that would be interesting knowledge beforehand to know.... I dont think state level hackers are interested in your credit card, they try for way bigger fish.(countries usually have access to central bank to print more money)

[@evilj]

8.5 million devices are tiny number but unsurprisingly, ALL are from high-profile organizations. I guess Crowdstrike has a hell of a sales team

[@spvillano]

A single application? One monitoring application brought one vendor's operating system down. That's not a single application per se, but it is a major flaw in the implementation of one monitoring application's coding that allowed that to happen in the first place. After all, do you then condemn Windows as an OS? So, which single other OS do you select and later condemn and eventually have a computer without any OS? Having multiple OS's beyond necessity increases one's infrastructure complexity and support costs. That gets balanced upon enterprise needs on a cost/benefit basis. As for national security risk, let's review some edge cases, since at the extremes, the effects tend to be greater. Russia was largely unimpacted, how many cities did Ukraine lose during the time that the world went dark? A village even? Guess what? By your standard, using electricity is a major national security risk, gotta get rid of it!

[@medea27]

Having worked in risk management & business continuity, it's been disturbing hearing IT managers complaining how the dodgy update _circumvented_ the very mitigants they had in place to address this exact type of event. An update should not be able to override the client's staged release process... the entire purpose of staging is to give a business the ability to quarantine faulty files _before_ they can take down every computer in their network. CrowdStrike managed to turn an inconvenient business disruption into a major business risk event. Unbelievable.

[@januslast2003]

The companies probably decided on direct update to reduce IT headcount. And besides, they probably figured that, if something went wrong, Croudstrike's insurance will ultimately pay.

[@effexon]

in video games that would be uhh congrats achievement for destroying world.

[@jdavis7515]

Spot on! This is where MS is most culpable in the event. MS should be acting as the gatekeeper. Security folks are going to argue against this I imagine, but most corporate IT tests all updates - even those coming from MS. I'll reiterate my previous comment - how did this get past CS internal change management processes? Since I'm very active in our CM processes, this reeks of incompetence to a level of unbelievable.

[@roganl]

CRWD is a rootkit - once you buy in, it owns your systems, remotely. As observed by the Primeagen - it's "pay to lose". That "quarantine faulty files" functionality can only be provided "underneath" 3rd party drivers - i.e. by Redmond itself. Self monitoring is a great aspiration - and it's possible in many circumstances however if CRWD has a bad day or a lapse in judgement or its motives run afoul of the broader user base - the words you need are "defense in depth" and MSFT needs to backstop the failure for the stability and reputation of their install base.

[@effexon]

@@roganl I cant remember so well but were these very invasive rootkits common DRM in pc games required to install to play some 15-20 years ago? then those got smarter by platforms. People complained already back then DRM crashed pc and prevented playing.

[@KeyBorg1]

I am a tech guy, but not a hard-core coder. I have never watched any videos like these, where the explanation is broken down so well that, in my humble opinion, darn near anyone could understand it. Seriously great videos and great explanations of a complex topic. Truly appreciated. I shared your last video with a lot of people, and will be sharing this one as well. Great job!!

[@neondactyl]

Been a fan of the channel for a while, glad to see you getting the views you deserve from covering this

[@projectartichoke]

Another great video on the CrowdStrike debacle. And, man, that is one cool shirt!

[@davidseasons6374]

Incredible! A cogent and coherent report on a major news story without yelling, screeching and F-bombs.

[@HeimirThorSverrisson]

As I heard the EU part of the story it was quite different. They blocked MS because all third party vendors were supposed to use this new API, but MS itself planned to bypass it! That of course is anti-competitive. So MS was not going to eat their own dogfood!

[@alliejr]

THIS!!!

[@Crayonberry1212]

Damn that would be bad. It would mean MS would descend to Apple's level. Wait.

[@crispybatman480]

The EU blocked Microsoft from what? I'm sorry, your sentences are a little vague.

@@crispybatman480 So MacOS say "you all need to use this" and doesn't have competitive product that can bypass it. Microsoft said "you all need to use this" and then would have bypassed the requirement itself by using kernel driver, thus giving itself competitive advantage with its own product. EU simply forced Microsoft to either give all access to kernel drivers or use the API solution itself as well regarding security products. Microsoft chose to allow all use kernel drivers.

[@MrCalldean]

@@Crayonberry1212 Sure that as soon as Apple have a stranglehold on desktop OS, they'll do the exact same to them.

[@tomking1890]

Nothing like running all his words together. I really liked the review he gave. Thank you, Sir.

[@AlexCastelijns]

This video has so many good quote's, thanks Dave for another funny and informative episode, i'm looking forward to the next

[@DavesGarage]

Glad you enjoyed it

[@_lordtra]

Very nice put. There is Postel's law from the 80s also known as the robustness principle in computing. Which is about input validation and that you should never trust input for your code and always validate it. This principle drives the most important protocol on the internet TCP and might be one of the reasons why the internet somehow works.

[@JamesQMurphy]

DevOps guy here. In our organization, code has to pass a successful pull request build with unit tests, a main build with unit tests, and deployment to two environments where the code is tested with other tools. When a bug reaches production, it is the fault of the entire organization, not just one developer. The exception: when some manager with an agenda shortcuts the process and rushes something to prod. Then it's solely their fault.

[@Aser6000]

Apparently systems with the update policy set to N-1 or N-2 were also affected because some types of Crowdstrike updates just ignore that policy.

[@Andy_Panda]

THIS!!!

[@johanneswerner1140]

That is one of the really bad and stupid things. Really stupid.

[@archerkid02]

@@johanneswerner1140 delaying the update might help avoid mistakes like this one, but if Crowdstrike is actually useful at preventing zero day attacks, wouldn't delaying the update also leave those systems vulnerable to attack as long as the attacker could be faster than the update delay? He talked last time about a red/blue option for each Windows install, but what if organizations implemented these instead by having a second Windows installation that's update delayed and warm swappable for when an update fails like this? Like a rollback option that doesn't rely on the existing windows to work? Or just a literal backup closet full of laptops for emergencies?

[@andersjjensen]

@@archerkid02 The problem here is that the CrowdStrike documentation makes exactly zero mention of the fact that they can bypass N-1 setups. Telling people that they are managing their own stability vs vulnerability tradeoff, and then nullifying their policies is considered a dick move in bird culture.

[@mrmattyboy]

Exactly. The preliminary root cause states that the agent itself _does_ follow the N-1 etc. policies. However, the content updates, which are downloaded outside of normal agent updates (which caused the issue), do not. It doesn't directly _ignore_ this policy, but appears to be a completely different release cycle.

[@MrJbipes]

Thank for this as well as the previous CrowdStike video and information. I'm not a programmer, but I managed them. I worked in computer software and applications my whole career, on nearly every platform and OS, mostly CAD/CAE/CAM, PLM and CRM disciplines. I loved your explanations with the touches of satire, irony and comeuppance you spoke with. You comment here near the end, "As for me I try to never attribute to malice that can be sufficiently explained by incompetence", I nearly spit out my beer. That said it all, as I was left with that thought from your first video. I've tried to explain this to others when asked what happened, as they too, commonly blamed MS. I'm no fan of MS but I know it smelled like a CS problem from the get-go. Keep it up, you gained a bigger fan. I watched your other vids also.

[@WaltWhitman_1819]

I have to say...I don't know how I didn't find this channel earlier. I appreciate someone that keeps up and can explain the new collective stupidity in straightforward simple terms. The days of IRC for me are over and that is where we used to talk about these things...but I no longer want to deal with getting a virus because I logged on...lol Old school guys now hang out on KZbin apparently...haha. I appreciate this video...thanks man.

[@randallgreen4084]

Best source of real information that one can understand and relate to, thanks Dave.

[@mbtjr]

Dave, you have the uncanny knack of being able to explain something to mere mortals that they really shouldn't be able to understand. I really enjoyed your deep-dive assessment of what occurred and especially how things could like this could be prevented in the future. Thanks!

[@unsaltedskies]

What I've taken from this incident is we've a good view of who is using CrowdStrike. Useful to know.

[@dmorga1]

This and your previous video are really fantastic explainers on this issue. I work for an organization that works closely with developers but we aren't highly technical people. Your videos could help even my newest team members understand exactly why this happened and hopefully draw some parallels to how this sort of thing can do an end run around "certified" pieces of code'functionality.

[@Whitewingdevil]

Thanks for explaining the situation and how/why it happened, when stuff like this happens mis-and-dis-information always runs rife, it's good to have a clear voice explaining it all while discussing what can be done and what has been done to prevent these kinds of faults.

[@colin2utube]

Thanks for this follow up (and hopefully more in the future). It's great to have an informed and balanced commentator pinning down the details when there's so much other uninformed comment obscuring the situation rather than shedding much needed light on it !

[@Lupinicus1664]

I used to write kernel level code, in IBM assembler, on mainframes in the 80s/90s/00s and what Dave was saying about the rigours of updating kernel (or 'control program' code as we used to call it) were intense. Desk checking, unit testing, system testing and review by peers (who felt a duty to find something wrong with it) was always followed by a careful introduction with robust recovery plans.

[@_Mentat]

I wrote kernel mode drivers, on a multi-user system with production work and customers on it, and I never crashed the system. I always wrote the last chance exception handler first.

[@spvillano]

That's one thing that I actually enjoyed with peer review. The joy peers take in savaging the shoddy work of a peer. Kept my ancient ass on my toes, given most of my peers were literally half my age or less and all were quite bright cookies. We both learned a lot. I learned a lot of the newer book learning that they'd acquired, they learned dirty sysadmin tricks on attacking and protecting systems. In one case, I showed them how EICAR could be buffered into notepad, which was actually being used in an active attack. I'm pretty sure that write-up is still being used in training. Always try to leave it better than how you found it.

[@TheInternetHelpdeskPlays]

I remember coing by business nvq and being told "there will be a problem! You will not be remembered for it. You'll be remembered for how you handle it."

[@RonJohn63]

Coing by business nvq?

[@peterkowalewska778]

This video should hit 1 mil based on the coverage of the topic and clarity. Bravo

[@Bryghtpath]

The breakdown of the CrowdStrike outage and the lessons learned were super insightful. It's wild how one update caused so much chaos.

[@calhoupe]

The driver ingested the blank update, choked, turned blue and died. 😂

[@donaldwert7137]

And kernels popped.

[@canbakr5602]

😂

[@spvillano]

@@donaldwert7137 then, the management turned the most fetching shade of purple ever witnessed by mankind.

[@donaldwert7137]

@@spvillano And looked around for someone tangentially associated with the error to blame. 😁

[@black350Z]

Totally remember that Tylenol incident. It's why everything you buy, now, has a tamper proof seal.

[@gsk_wd8cia]

I heard about it on the TV 2 seconds after i swallowed two Tylenol Extra Strength capsules.... I hadn't even moved from in front of the package.

[@raygunsforronnie847]

Product tampering also changed a whole bunch of details in supply chain accountability that were less obvious to consumers.

[@spvillano]

Or as I call some of them, patient proof seals. I've had some that I've never successfully retrieved from the packaging and the dust literally went into the trashcan. Pity they never caught the turd.

[@ryanphillips8492]

Dave, thanks for the laughs. "Standing around thier disks in thier hands..." and "thier code just decided to raw dog it" had me dying this morning 😂

[@not12listen]

I absolutely love how you eloquently go over the likely chances that someone's code was not vetted properly or maybe at all. "Never attribute to malice that which can be sufficiently explained by incompetence." That aside, my co-workers and I had to deal with quite a few devices that had BitLocker enabled. I am not a fan of typing in BitLocker codes.

[@JezerRamirez-p2o]

I’m not an IT guy and I’m pretty dumb when it comes to code, but somehow I feel like I understood just about everything you said, thank you for explaining.

[@adamchurvis1]

Dave, I hope some classrooms are still being blessed by your most valuable presence, despite your Retired status. Not a single fragment of fat in your words, everything direct and to the point, everything understandable through clear and concise communication, perfect narration cadence, crystal clear diction... you check all the boxes, sir. Thank you for sharing your knowledge with us all.

[@JustinNelsonsProjects]

I do general maintenance at a restaurant owned by a large corporation. Corporate IT is extremely paranoid so naturally they run crowdstrike. I walked in on Monday to find that none of their Windows machines would boot. And I thought cool, I already read about the quick fix involving safe mode. Only to realize, they were also using bitlocker. Nothing I can do at that point except to pass the buck on the corporate IT who is most certainly not going to provide me with the BitLocker key... Luckily our POS system runs Android so we were still able to be open for business. Just couldn't do payroll and accounts payable accounts receivable food orders and you know all the unimportant stuff LOL... And because this corporation owns a ton of hotels, our little restaurant is low on the totem pole...

[@renerpho]

Happy days...

[@grokitall]

this issue of a broken kernel module causing a crash can happen to any non trivial os. the question is why microsoft or crowdstrike did not have anything in place to prevent the boot loop.

[@RaveDJ-BigDave]

Curiously, my IT team did talk me through BitLocker Recovery. Expediency and trust over potential malice (or future stupidity)

[@ShawnFumo]

@@RaveDJ-BigDave I think this is a case where it does make sense. Just give the key to get things up and running and then rotate the key later on to make it more secure again.

[@grokitall]

@@ShawnFumo these systems were mostly locked down, and mostly for good reasons. this is why the boot loop caused so many problems for so many people. the only way to fix this was to turn up, change all sorts of settings so that the system was unlocked and insecure, then do a trivial fix, then secure it and lock it down again, and that is in the cases where access was easy. try doing that to airport flight status screens 20 feet in the air. this is almost impossible to outsource to local staff, unlike the "reboot into safe mode and delete this file" advice from cloudstrike.

[@ArthurOgawa-q9z]

I appreciate the wide scope of your comments, particularly your citing the bug that hit Linux O/S machines and your comment about Mac OS. My perspective is that the malfunction may very well stem, not specifically from the "all zeros" configuration file, but from the nature of the P-code interpreter itself. In short, that engine _must_ be immune from bad input, since it runs at such a high level of trust with the driver insisting on being loaded at boot time. I look forward to reports from people who trace the fault down in a debugger. ~~~~Arthur Ogawa

[@eddiecastillo6047]

Dave, thanks for all your insights! I work for a company that has bitlocker activated using dell 3080's, 3070's etc. Running Windows 10. It is amazing how hard it was getting computers fixed. The big problem was booting to safe mode with networking. Sometimes it could be found. Pressing F8 on boot to windows did nothing. A nightmare at best!

[@bobdeadbeef]

Great coverage! My own suggestion, as a final backstop, is to set a flag w/ any update, and clear it if the system comes fully up. Then if the flag is still set on reboot, reject it. Perhaps only reject after N reboots (2 or 3) to minimize false positives from other causes, such as subsequent drivers. A final layer in the Swiss cheese risk model, mitigating the risk of systems needing manual recovery requiring hands-on IT assistance.

[@An.Individual]

MS have signed CS Falcon. Falcon is an interpreter. For MS to allow an interpreter to run in ring 0 is very risky.

[@brodriguez11000]

Introducing Turing-complete where it didn't use to be is the foundation for a lot of attacks.

[@bereft.of.identity]

and people are saying the "C-00000291*.sys" files are NOT signed. An interpreter running in ring 0 using unsigned content files. What could possibly go wrong?

[@zisaizic4759]

falcon is not an interpreter, the channel files do not contain code, the code that crashed windows was signed and tested by microsoft and was released in feb 2024 with falcon 7.11, it's just the content of the channel file that trigger the bug.

[@ReverendTed]

@@zisaizic4759 And that's really my question - would Microsoft's testing of the kernel-mode driver not normally include tests that might have revealed this input validation vulnerability? That is, "if I feed it zeroes, does it crash?"

[@zisaizic4759]

@@ReverendTed obviously not, microsoft probably doesn’t test the channel file features, that’s in crowdstrike to test and fuzz the parsing code. sometimes these kind of bugs are very hard to catch, shit happens, crowdstrike should’ve tested this file in particular or have some staggered rollouts instead of just assuming this code was bulletproof based on previous testing or previous channel file delyopments

[@BigDaddyWes]

One of your last points about whether or not the entire world should be dependent on a single vendor is honestly the biggest takeaway in my mind. It's a risk to have any system rely on a single point of failure, but to have literally millions of systems depending on the exact same point of failure seems absolutely bonkers. The world will always try to find someone to blame, but it's worth recognizing that we put ourselves in a position where this could happen, which from what I know is totally counter to general IT philosophy. I'm not trying to victim blame at all, it's just crazy to me that so many people and companies rely on so few points of failure that something like this is even possible.

[@ShuRugal]

But having more than one vendor would be communism!?!?!?! /s, for y'all who need it.

[@effexon]

this is where conspiracy theories come to play... some of those international organizations want exactly that centralization and weakpoints for control purposes and "easy access".

[@nomore6167]

"whether or not the entire world should be dependent on a single vendor " - Such as Microsoft? The entire industrialized world is dependent on Microsoft Windows. I completely understand your point, and wholeheartedly agree with it, but if we're going to acknowledge it, then we have to address the fact that Microsoft Windows is the biggest example of the world being dependent on a single vendor. To your point, though, that's one of the reasons I'm against moving things into "the cloud", because that's exactly the same thing -- moving so much of the world's computing into the hands of just a few companies.

[@MyWalk3296]

Agree, if power goes out in one area it shouldn't go out in our whole country & elsewhere. There should be so called circuit breakers also

[@ShuRugal]

@MyWalk3296 circuit breakers do exist, but their job is not to keep your power on, it's to protect the transmission lines and generation equipment.

[@lessevilgoog]

I'd like to know more about this EU decision. If it was something you'd have to pay some extraordinary fee to Microsoft for access to, then I agree, it's anti-competitive. In other words, I smell greed and gatekeeping not arbitrary EU decision making. But I could be wrong. It wouldn't be the first time Microsoft touted a "security feature" that just so happened to make it difficult for others to enter their market as a (side-eye) surely unintended by-product of how they decided to make it.

[@orngjce223]

They made Windows Defender not have to run through it.

[@chairmakerPete]

The EU is a disaster in the tech space. Achievements to date include putting cooking warnings on every. damn. website. as if cookies are some sinister force that can penetrate your cells, and of course assuming that the bad guys are going to comply with their ridiculous GDPR legistlation. Of course, the real triumph of the EU has been to force Apple to put USB C connectors on iPhones for sale in the EU - a decision that immediately send billions of Lightning cables to landfill. The EU exists to create legislation for imaginary problems, all of which apparently justifies the existence of the EU.

[@timepilot2012]

This is an exceptionally well done video. I'm also 56 years old, and was delighted to hear you use the term "bork". I say it all the time, and my young employees have no idea what I'm talking about.

[@smiththers2]

i like using fubar as well as bork lol im only 41, though my step dad was a programmer and IT for Liberty Mutual from the early 80's and has since retired almost a decade ago.

[@philipramsden4975]

41yo IT Manager here and I use "bork" or "borked" all the time

[@savagesarethebest7251]

I am 30 and I would use the word borked, along snafu and fubar.

[@Daekeyas]

Thanks for a clear breakdown of the event and what that means

[@RudysRetroIntel]

Fixing a server with non encrypted disk and safe mode = easy, fixing a server with encrypted disk = nightmare. Thanks for sharing

[@jdavis7515]

I'm curious as to why you think that? Our teams recovered ~17K hosts. One extra step due to encryption.

[@RudysRetroIntel]

@jdavis7515 I had no problems with my servers (122), but people did have issues

[@robertcross5794]

Although I wouldn't classify it as a nightmare I did spend quite a bit of time on hold with corporate IT waiting for a sys admin to pick up and give me the bitlocker key so I could get to the cmd prompt and delete the 291 file. I only had a handful of hosts to fix at my location but I have no idea how many the admins had to take care. They were probably giving out bitlocker keys in their sleep. We are a worldwide hospitality company.

[@RudysRetroIntel]

@@robertcross5794 I'm sure the businesses and people with no ability to get 911 emergency services would call it a nightmare :)

[@dev-mc4jq]

Fixing a server with a dead HBA card & raid arrays spread over several clusters of disks + encrypted partition = ultra nightmare. Choose ZFS instead, too bad windows don't have ZFS.

[@rschiwal]

As an IT security professional facing replacement with AI, this was a very funny "I told you so" moment.

[@Stuxnet1914]

Excellent video. Very helpful and insightful with the tidbits from the good ole days!

[@MrMegaManFan]

I'm already subscribed, and thank YOU for covering this landmark computing issue with such clarity. It's truly a historic time that (hopefully) we won't see the likes of again.

[@rider275]

This was a really informative video for me. If nothing else I have a better understanding of what may cause a BSOD and the difference between kernel model and user mode. Thank you!

[@NoNonsense316]

Best quote: "Never attribute to malice that which can be sufficiently explained by incompetence." Wise words.

[@BaobobMiller]

. Hanlon's statement itself is an absolutism that if followed would result in approximately zero attempts to prosecute the vast majority of crime in the universe and is often argued in various forms by agents of espionage and organized crime... The statement and practice themselves demonstrate poor, if any, application of logic skills, and a fairly astounding lack of wisdom. Not that one lacks logic skills necessarily. People have the tendency not to think through and apply what they know to the things they THINK they already understand. This is particularly prevalent cases where the "things they know" were learned AFTER the "thing they think they understand" was indoctrinated or assimilated into their "knowledge base". In reality we deliberately strive base our systems on the inclusion of all possibilities given the current known set of facts and only eliminate them as verified facts show them to NOT be possible unless finding ones self in the position of having run out of avenues of evidentiary exploration and forced to make a decision based the combination of probability, greatest success, and least harm. It's literally the basis for all areas of scientific learning including computer science.

[@arturoaguilar6002]

@@BaobobMiller More like your interpretation is absolutist. Like, isn't a prosecutor job to search for proof of malice in the first place? I don't know why you go out of your way in trying to apply a general rule into a specialized field.

[@spvillano]

@@arturoaguilar6002 the term in criminal law is mens rea, literally "guilty mind" in Latin. Examples would be prompt attempts to cover up a crime, which effectively displays knowledge that their actions were criminal to begin with - otherwise, why cover up a guiltless and lawful act? And in courts around the world, both concepts are considered. Oddly, without Baobob's problems in determining the difference between the two.

[@johnshortt3006]

A long time ago I was involved in writing a graphics terminal emulator. The input was required to adhere to a strict syntactic structure but a large client tested it by sending it random data to ensure it wouldn't crash. I thought this was a very smart thing to do and it uncovered a few bugs.

[@ReverendTed]

I took a couple of entry-level programming classes in college, back in the 90s. The understanding that you couldn't trust user inputs to be valid was one of the first things we learned. To be fair, they taught us that while explaining we wouldn't have to worry about input validation\vetting in our entry-level projects. Even so, anyone in the periphery of Internet culture is aware of user-input code injection exploits, like the "drop table" memes.

[@Erik_The_Viking]

Great video! Love the shirt. Nice to hear this from an ex-Microsoft engineer familiar with kernel mode and internals.

[@kylereis3639]

From what I’ve read this was an “emergency update” file so it didn’t go through the normal pipelines and process. It also bypassed any channel restrictions so even if your system was set to only get n-1 or n-2 updates you still got this update.

[@ddichny]

Well, that update did turn out to be an emergency, so at least it was labeled accurately.

[@desembrey]

Blue Screened machines are pretty safe from any malware coming along and infecting them. (heh)

[@Daijyobanai]

Having had a test run on Linux and now a full on catastrophe on Windows, it's fair to call for senior management at Crowdstrike to set their linkedIn profiles to Open for Work. Or just give them 2 weeks notice like they do everyone else.

[@arty2k]

Thanks for the "raw dog" reference, I had a laughter buffer overrun.

[@JohnGonzalezFL]

At 30+ years into being a C++ developer, I still have to keep reminding myself that “all input is evil”. I had an instance a few years back where data from a “trusted” source was accepting outside data without validation. That, in turn, caused issues with my program. Doh! It took a while for me to code around this and fully validate what I thought was a trusted input stream.

[@jefferysmith5921]

IMHO there is no such thing as a trusted source. I learned that the hard way too.

[@Razimuth]

Great video, this is the first take that correctly puts the focus on Windows itself as the root cause of the problem. Crowdstrike ultimately caused the outage, but the conditions for which it was possible lies solely on Microsoft.

[@Kyzyl_Tuva]

Dave, your videos are so helpful. Thank you for what you do!

[@JinX0011]

Raw dogging it and hoping for the best. We've all been there.

[@JB-ml6ir]

"Standing around with their disk in their hands" has to be the most amazing thing I've ever heard

[@thelanavishnuorchestra]

Yes, thanks for pointing out that it's not just a developer and a dev ops engineer rolling the thing out. They clearly don't test what they're about to roll out and that they don't have the proper error checking. They have management structures in control of this procedure and they're ultimately at fault. I'd say the CEO and the heads of several departments need to resign and be replaced by people who will put proper controls in place. I had a client looking at ClownStrike as a security vendor, but between the cost and the requirements of the software, we just expanded on things we already had in place. Breathing a sigh of relief.

[@kingjames4886]

it's definitely whoever wrote an update consisting entire of 0s fault...

[@thelanavishnuorchestra]

@@kingjames4886 from my understanding, something went wrong in the process where that was generated, and that's how it got filled with zeros.

[@kingjames4886]

@@thelanavishnuorchestra the only way this makes any sense is if an AI wrote it any no one checked it...

[@ratsalad346]

Thanks Dave. Lots of respect for Dave Cutler and the Win32 kernel team. I wish every Microsoft product had that quality.

[@ratsalad346]

Sorry, *NT kernel team.

[@AaronBrooks1]

I brought this up on the previous video but will again point to the eBPF model used on Linux as being a good way to get userspace logic in the kernel with memory and deadlock safety. I'd love to see this mode used in more places.

[@gregzielinski]

I'm so glad you covered the crowdstrike issues in these videos. I've now found your KZbin channel and have more to dig into.

[@malcolmexton4299]

Check out Dave's amazing interview with Dave Cutler (life-long hero).

[@wirenutt57]

"And Crowdstrike, not so much. Looks like their code just kinda rawdogged it and hoped for the best." Dave, I'm having trouble writing this comment due to the tears still flowing from laughing so hard. Congratulations on the best line on KZbin in a long time. F-ing classic.

[@thomaspripley]

Another 100% substance, 0% fluff video from Dave!

[@hgbugalou]

Preach Dave, preach. I am linking your videos to all these snotty Linux bros who haven't used windows since the 90s.

[@nekoDan]

Very informative and entertaining. “Raw dogged it and hoped for the best” is my favorite line from the video 😀

[@Shlegal123]

I want your SHIRT! Well not yours but my own!

[@donnierussellii4659]

Search "Binary Computer 1s and 0s Blue Button Up Pocket Shirt"

[@snowbatsnowbat7197]

Search for geeksoutfit binary. I wonder if Dave has the matching drawstring shorts 🤔

[@mainbhifunny5913]

@@PTSCZ do a reverse google image search! geeksoutfit

[@PTSCZ]

Binary Computer 1s and 0s Blue Button Up Pocket Shirt

[@foobar4938]

Its not unfair, they couldve still released the api and kept kernel drivers. They wanted to do exactly what the EU righfully blocked.

[@jodosh]

Yeah my memory of the EU decision was that MS was wanting to kill off anyone who didn't use the new API, and they didn't get enough assurances that MS wouldn't abuse the API to force out smaller security companies. The EU was fine with the API, but not at the expense of competition.

[@refl9630]

"I would never dare to question the wisdom of printer designers."

[@RascalCatify]

We had about 1600 endpoints we had to touch on Friday that were affected by this. Earlier in the morning myself and a couple of other co-workers were getting servers that were affected up and running. And yes, Bitlocker on the laptops provided another fun factor on that day!

[@volvo09]

I do not miss the days of having to group up and visit EVERY endpoint in a 1200+ user office after a failed update. Only happened a few times, but it sucked. When I started in 04 we had this old software that reported the computers location back to a database, so if there was a mass move that weekend we had to be on site to power up each computer, wait for it to boot, log in, open the software, and enter the new cubicle number manually, along with changing the computer name to the new floor and area... It was terrible It was so much slower than fixing this crowdstrike blue screen.

[@iskieisketerol5963]

Very well explained. You have raised very interesting points as to the perfect storm that allowed this to occur.

[@thesaintcivillainofficial]

That's so crazy, almost as crazy as the time I ran a company that scammed people out of hundreds of thousands of dollars back in 2006, and ultimately had a state court make me pay it all back . Ain't that crazy, mister Dave Plummer?

[@RobCoops]

Well kind of... the problem here was that though Microsoft offered the API to everyone else they themselves wanted to continue using their kernel driver for their own security tooling the argument being we are Microsoft we would never write bad code as we have the ability to debug this and know the ins and outs of the system better than anyone because we wrote it. Well yes, but also no... This is where a complaint was raised with the EU court that essentially claimed MS would be able to crush the competition as they would be operating on a lower level than everyone else so they would be able to catch more than anyone else. That argument was made because Microsoft intended for all security tooling but their own) the be removed from kernel level access. The EU Court in a reasonable ruling said that this would lead to unfair competition and MS in all their infinite wisdom decided to keep their security solution running in kernel mode and not move to this new API, thus being forced to allow all others to also be playing at the kernel level. So yes in part the Eurocrats are to blame for this, as with so many other things. But if Microsoft had not wanted to give themselves an unfair advantage they would not have had to allow the security firms to play around at the kernel level thus removing the possibility of this type of issue. Then again Crowdstrike has as you rightly pointed out a history of these types of issues as their QA process clearly needs some work. Because anyone who would have deployed this on a windows system in the Crowdstrike office would have noticed that the computer got a little funny after their last update. Yet this was shipped to the world obviously without ever being tested.

[@Monaleenian]

It’s Microsoft’s operating system. They should be able to do what they want with it. That’s not an “unfair advantage”

[@alalalavaladu3670]

Microsoft has to and should use security solution that runs in kernel level. They have to fulfill both the robustness and security requirement for their operating system. this can only done if external code is strictly regulated.

[@lionvillelion]

@@Monaleenian Part of running a business is operating it within the law of the places you intend to sell your product.

[@RobCoops]

@@Monaleenian I guess you do not quite follow the monopoly regulations around d the world. Essentially if you are going to own 80% or more of a market then you are considered as having a monopoly and new regulations apply. Here is an example Netscape was the leader in internet browsers having +90% of the market. Then a competitor released a free version of their own browser which was not nearly as good as Netscape's offer. But this competitor happened to be Microsoft and they released this as part of their operating system making it the default browser back in the day when nearly no regular human being could download software without a browser. This free offer from MS was later judged as illegal. After all they gave away a product to crush all competition using income from a different product to fund the development and maintenance of their browser software. Microsoft argued that the browser is a part of any modern operating system, monopoly watchdogs judged otherwise and forced Microsoft to make a version of windows without the browser and offer a choice of browsers upon installation. Yes it's their product but when you end up controlling the market you open yourself up to additional regulations aimed at protecting consumers.

[@MrCalldean]

@@Monaleenian That works on a small scale - but as soon as you're the 800lb gorilla, you have to accept that you need to play nice with others if you want to play at all.

[@sonny-fn3kn]

man this crowdstrike company kinda sucks imma stick with softwareonline personally

[@philipmasie]

Great 👍 explanation Dave! Thanks for the detailed analysis 😊

[@karthickm7776]

Hi Dave. Nice Explanation from you. you are a Legend.