A Comedy of Errors - An IT infrastructure expert weighs on what went wrong (and how to prevent it!)

www.skool.com/...
🚀 Welcome to the New Era Pathfinders Community! 🌟
Are you feeling overwhelmed by the AI revolution? You're not alone.
But what if you could transform that anxiety into your greatest superpower?
Join us on an exhilarating journey into the future of humanity in the age of AI! 🤖💫
🔥 What is New Era Pathfinders? 🔥
We are a vibrant community of forward-thinkers, innovators, and lifelong learners who are passionate about mastering the AI revolution. From college students to retirees, tech enthusiasts to creative souls - we're all here to navigate this exciting new era together!
🌈 Our Mission 🌈
To empower YOU to thrive in a world transformed by AI. We turn AI anxiety into opportunity, confusion into clarity, and uncertainty into unshakeable confidence.
🧭 The Five-Pillar Pathfinder's Framework 🧭
Our unique approach covers every aspect of life in the AI age:
1. 💻 Become an AI Power-User
Master cutting-edge AI tools and amplify your productivity!
2. 📊 Understand Economic Changes
Navigate the shifting job market with confidence and foresight!
3. 🌿 Back to Basics Lifestyles
Reconnect with your human essence in a digital world!
4. 🧑‍🤝‍🧑 Master People Skills
Enhance the abilities that make us irreplaceably human!
5. 🎯 Radical Alignment
Discover your true purpose in this new era!
🔓 What You'll Unlock 🔓
✅ Weekly Live Webinars: Deep-dive into each pillar with expert guidance
✅ On-Demand Courses: Learn at your own pace, anytime, anywhere
✅ Vibrant Community Forum: Connect, share, and grow with like-minded pathfinders
✅ Exclusive Resources: Cutting-edge tools, frameworks, and insights
✅ Personal Growth: Transform your mindset and skillset for the AI age
🚀 As You Progress 🚀
Unlock even more benefits:
🌟 One-on-One Mentoring Sessions
🌟 Exclusive Masterclasses
🌟 Advanced AI Implementation Strategies
💎 Why Join New Era Pathfinders? 💎
🔹 Expert-Led: Founded by a leading AI thought leader, connected with top researchers and innovators
🔹 Holistic Approach: We don't just teach tech - we prepare you for life in an AI-driven world
🔹 Action-Oriented: Real skills, real strategies, real results
🔹 Community-Driven: Join 300+ members already navigating this new era
🔹 Cutting-Edge Content: Stay ahead of the curve with the latest AI developments and strategies
🔥 Don't just survive the AI revolution - lead it! 🔥

[@kraz007]

Bullish for Crowdstrike... Now everyone knows them.

[@spectralvalkyrie]

I just ordered a hat that says, "Crowdstrike? Who the f*** is Crowdstrike?"

[@artcurious807]

James Comey loves crowdstrike also

[@softwarephil1709]

@@kraz007 including the law firms preparing class action lawsuits.

[@ThioJoe]

I read at least one comment that they pushed the update IGNORING customer staging rules. Haven’t seen confirmation of it though.

[@DaveShap]

Yeah, if that's the case, it's even more hilarious. A security company should know better. Change management for the win

[@brandonzhang5808]

Confirmation isn't really needed when incremental rollouts are meant to prevent exactly this kind of issue by definition.

[@ryzikx]

thiojoe watches dave shap😱

[@tubaguy0]

I have my organization’s production infrastructure on N-1, which is 1 release behind the most current version of the Crowdstrike sensor for this exact reason, as a best practice. We still got hard reboots on our hyper-v cluster, satellite servers, user workstations across 25 sites. I hope Crowdstrike takes the lesson and makes the process changes necessary to make sure this doesn’t happen again.

[@4Fixerdave]

ITIL requires staff. You don't often get the IT staff until things break so bad that even the bean-counters figure out that they need well-run IT. Basic reality in anything but top-tier organizations... when IT is doing its job well, nobody knows they exist. They just get more and more work dumped on them until they can't do a good job. Then, everyone knows they exist. Rinse and repeat with the next batch of managers.

[@LaserGuidedLoogie]

Exactly true.

[@JeanPierreWhite]

Its not possible that all companies have abandoned ITIL. Some may have. SOmething else caused this and it bypassed all change management and release control at Crowdstrike customers. I'd like to know how.

[@boonkiathan]

ITIL won't have prevented any bad things from happening, even if fully staffed. ITIL only helps to run things smoothly when things run on a good day only right people vigilant at the helm of the change management and release control processes it's only if the truth of what happened will ever truly see the light there should be some formal inquiry

[@AAjax]

There's been a bunch of constructive dismissal and actual dismissal at Crowdstrike over their return to office mandates. I'm thinking that the remaining team is being overworked to pick up the slack, with an outside chance of cowboy culture.

[@DaveShap]

Oh that's a good point. The good people can walk and get another job. That's why I left the corporate world.

[@raymond_luxury_yacht]

Voting infra and who knows who is actually responsible Could be a blackhat insider. This is not what it seems.

[@morphrelink]

I used to work at a 911 call center and the thing they would do during major updates and patches to the system was a very simple thing. We had a back up server that employees would switch to. Similar to working in safe mode except our safe mode was a completely different server. I always assumed most business knew to do this.

[@nyps]

oh they do, but there's still 2 issues: a) the employees' devices were also affected and there is usually not much redundancy for end user equipment. b) the backup server most likely was also running crowdstrike (as part of the org's standard security stack) and was thus affected by the same bug.

[@phsycokilr117]

To add onto @nyps, the whole situation arose from an automatic push update that activates on air before booting up windows. Unless the computer was turned on without an internet connection, the system would run into this bug and not work. Most sectors require internet for their backup computers to be of any use.

[@morphrelink]

@@nyps No, the back up servers are private and are not connected to the internet in anyway. So even if they did have Crowdstrike, it would not have been the most recent build. Even the inbuilt cad maps are a bit outdated. In the main servers. When a call comes in, it connects to the phones gps and details of the caller and their location so it can be shown on the map. Autofilling a lot of info as they take the call. The call is then sent to dispatch's computers who relay the info to the officer. The officer also recieves details of the call inside their squad car. (When 911 asks for this information it's usually just to verify that it is accurate which isn't always the case.) Anyway, when logging into the back up server everything has to be entered manually and the call is physically printed out and carried over to the dispatcher so they can radio the call into the sector the call is thought to be located then see which officer is available and how far they are. All things that would be automated with a server that is fully online. The back up server doubles as a training server since it is offline and won't become a permanent record. Usually, the updates happen in the dead of the night when little to no emergency calls are expected to come in as a way to mitigate the down time which usually lasts an hour tops. As an extra failsafe It's also done in increments. That is to say, Only half the callers are in back up and updating while the other half are still online. Then rotated out. To follow that, There is an entirely separate back up call center in a different part of town call takers can go to if the entire building is down, and as a fourth back up, calls can be re routesd to neighboring police agencies from outside the city. To have the full team down, neighboring cities down, and the back up center down unceromoniously in the waking hours of the morning when most people are calling in to report either their car broken into over night, or a fender bender on their way to work. Ontop of the the black out etc. You can imagine why there were issues getting through to 911.

[@Me__Myself__and__I]

They do. But if a 3rd party pushes a broken update to all systems simultaneously then backup servers are useless because the primary and backups all fail simultaneously. IT departments were short sighted (and likely being lazy) in allowing that.

[@morphrelink]

Granted, if the entire operating system is compromised then of course, they won't beable to use that computer to access anything, but I wanna clarify the back up server that we used was on a private server not connected to the internet, for all intents and purposes it's basically a local dummy system primarily used for training. It's never truly online to begin with and the software and its database is several updates behind at best. Even the maps are outdated. When using the main servers, the system is indeed connected to the internet. This is so the call taker can get geolocation information from cell phones and check and communicate with city, county, state, and nationwide databases. When a call comes in, much of the info is automatically populated into the call notes and the calltaker continues to ask questions to verify the information is correct. The call is then sent to a queue within whatever sector the call was placed and a dispatcher radios the call to the nearest available squad car on their map to relay the info while the call taker talks with the caller and simulataneously updates the call. The call info also gets sent to the squad cars onboard computers. When using the back up sever, the cad system used becomes a glorified notepad where everything has to be entered in manually and then physically printed out and walked over to the dispatchers cubicle so they can do a general broadcast to the police in the area over the radio. Some of my old coworkers who still work there said it didn't affect all the computers and they weren't down for very long, that they did in fact go to the back up system for a short while but it happened early enough in the morning that there wasn't a huge call volume anyway.

[@eSKAone-]

Skynet tested a few battle moves

[@Me__Myself__and__I]

On a side note, this is one reason that the world will be extremely vulnerable to AGI/ASI if it isn't fully aligned. Imagine what an intelligence that knows more and thinks faster could do with all the issues like this that exist and haven't yet been exploited.

[@SimonCarpio09]

You're bringing back painful but fond memories for me with this one. P1s on the weekends and P2/3s for breakfast.

[@dab42bridges80]

As an ex-infrastructure engineer myself, I 100% agree with your analysis. This would never happen in a properly managed IT department. This is called learning the hard way, instead of looking for advice from those who've been there and done that, or trying to save money upfront instead of investing in proper practices from the beginning.

[@JeanPierreWhite]

Nope. United, American, NHS, banks didn't all stop doing IT properly. Something or someone at Crowdstrike bypassed all change management at these corporations. How is it possible that a third party vendor has the ability to deploy a change to prod across its entire customer base pretty much simultaneously. On top of that Windows endpoints cannot recover from boot loops automatically, which after more than 30 years is inexcusable. Windows is a bad choice for mission critical endpoints.

[@dab42bridges80]

@@JeanPierreWhite I'd like to see evidence of those corporations having used their own testing before patches were applied to their production environments. I've seen atrocious practices at large and small companies that were jaw-dropping in their negligent configuration (I have a background in contract and full-time IT and I've worked for the entire spectrum of businesses.) However, I do agree that Microsoft operating systems can be poor choices for critical platforms, but even those can sometimes be traced to inadequate patching, lack of experience, and/or old versions that weren't upgraded for financial reasons for example.

[@brianhershey563]

Back in the 90's we (30 person tech company) needed 3 terabytes of Exabyte and DLT tapes to back up all drives, that was leading edge! I loved that job, nobody ever worried when I was around :) Also ran the phone system! 🙏

[@Steve-xh3by]

As a retired software engineer who has released many high impact updates, I simply cannot believe that a phased rollout wasn't followed. Shapiro is right, this is a process problem. The patch should have been released to a small subset of clients first, NOT just blasted to the whole world.

[@DaveShap]

Someone is getting a PIP for this for sure

[@Me__Myself__and__I]

Retired eh? Then maybe you've missed the fun arguing with newer devs that setting software dependiencies to auto-update is actually a bad idea. Many seem to think auto update is a BKM. So you deploy the software to a test server and fully test it. Then when you deploy to prod it deploys with the current versions of dependancies at time of deployment, not necessarily the versions that were tested. If say 6 months later you need to deploy a new prod server, it uses the current versions then. I've had arguments over this because some insist auto updates are better, more secure or whatever. Its truly sad.

[@Steve-xh3by]

@@Me__Myself__and__I I was referring to the process of the deployment to clients, in general (at a high level). When you have such a large client base (global), it was always bad process to release to the entire client base all at once, regardless of how critical the update was. There should be a phased approach with a validation of some sort to ensure that there were no problems to the limited set you deployed to, before opening the flood gates. Granted, I don't have all the details for this outage, but it seems like a catastrophic process failure.

[@DrKaoliN]

@@Me__Myself__and__I Wow, I didn't even think about that scenario. Deploying with newer untested dependencies makes the testing pretty much obsolete. I hope CrowdStrike will be transparent about how this incident was possible.

[@brianWreaves]

As non-engineer/DevOps, I have auto-updates turned OFF on everything. Primarily because it is more difficult for me to repair issues and thus more time. e.g. Browsers, OS, antivirus, etc. I wait a few days to see if there are reported disasters. I often wait months on OS updates.

[@bkreinz]

U r fully right. IT change errors: The original IT risk. People should learn more about OpRisk

[@kccorliss3922]

Non expert here. Couldnt they have found the bug by doing some test roll outs on some test computers? How do they test?

[@DaveShap]

Yep that's the point. This was clearly a process failure. It's easy to test batch

[@Me__Myself__and__I]

Short answer: clearly they failed to adequately test. Plus companies should never have allowed 3rd parties to push updates directly, or at least not to both production and backup systems simultaneously.

[@ticijevish]

They are supposed to "stage" any update. Staging is running a server like the ones the clients use with data, programs and processes like the ones the clients run. The problem is that such servers cost millions of dollars and staging itself also costs time and money and managers loooove cutting such costs, reasoning there is only a tiny chance of anything going wrong when untested updates are sent out (pushed to prod). Of course, nothing bad ever comes from this...until it does. If you ask the managers, they will tell you that staging fosters laziness among the coders and is an unneccessary cost and source of delays. I do dream of a world where whenever a manager says such nonsense, the people around them write Crowdstrike Falcon on a Post-it and staple it to their forehead.

[@Me__Myself__and__I]

@@ticijevish Kinda but no. That is (mostly) how staging works in an enterprise except most of the time the servers are nowhere near that expensive. But that isn't at all how commercial software works. Software companies don't have staging environments configured like their costomers with the same kind of data. That would literally be impossible. Software companies use different types of testing.

[@kccorliss3922]

Could they have asked chat gpt if the code Had a problem for f sake?

[@ChristopherRyans]

Once again condensed truthfull info. I couldn't believe how hard it was hear news anchors literally having no idea what they were talking about. This video was refreshing

[@mickelodiansurname9578]

What even happened to the idea of incremental roll out? I mean pushing to everyone is just lunacy!

[@SciFiMangaGamesAnime]

There is an excellent speech done by Bert Hubert - 'Cyber Security: Pre War Reality Check'. I recommend it to everyone, really, its not overly technical and very insightful. In short, he explains just how vulnerable networks and infrastructure are at this moment and what role outsourcing is playing in this problem. PS: And if you (person in YT comment section, not you, mr. Shapiro) think right now: "stop fearmongering, it wasn't an enemy attack!", I will reply - Exactly. What my 'Exactly' means? Check the speech to find out, its a great read. Or just ask AI to summarize it for you.

[@Me__Myself__and__I]

More relevant to this channel. Imagine what an unaligned AGI / ASI that knows vastly more than any single human and can think / act much faster could do to the world with those same vulnerabilities. Even worse than in war, because there would be no friendlies to protect and no emotions or morals to hold back. For example if human hackers found vulnerabilites that allowed them to cause nuclear meltdowns or make nuclear bombs explode in their silos - humans might be unwilling to do that, if for no other reason than nuclear fallout kills humans. Intelligent machines would be capable of doing things we humans might not even think of because they don't need the same things to survive.

[@NoidoDev]

​@@Me__Myself__and__I That's exactly the argument that this is not an AI problem, it's a computer security problem. These things are well known for quite some time by people interested in the topic. It's not AI that needs regulation, it's big corporations and other bureaucracies.

[@Me__Myself__and__I]

@@NoidoDev That is completely 100% incorrect. Even the limited AI available today is pushing the boundaries. But AGI and ASI are completely different things that don't fit ANY prior patterns or molds. We are working on creating artificial INTELLIGENCE that will be able to think, reason, decide, plan and take actions. It won't be some dumb IT system. It will effectively be an alien species that is smarter than humanity and has completely,pletely different psychology, different motivations, etc.

[@TheGeneticHouse]

Great explanation David! For medical and business yesterday everyone's computer was out. I do not like automatic updates because I am legally blind and when things change the world is over lol I always have that turned off I guess there's other reasons why that's a good thing lol

[@rainstormdesign]

Great video, appreciate the inside scoop. I would watch more of these IT style videos where you relay your insights from your time in this field. Ai will collide with all this IT ‘nuts and bolts’ stuff. That will be a great time to have you provide even more insight.

[@DaveShap]

Gonna have more interviews, but yeah I'm not just your average KZbinr. I was in the trenches

[@AICentralSA]

As a current infrastructure engineer I’m glad you can relate

[@TorgBlitz-qs5gz]

Sam Altman got a house that is a Lemon. The developer does not want the Earth being taken over by machines.

[@jsivonenVR]

I’m on summer vacation, and my employee is the local vendor of Crowdstrike. I bet it’ll be fun to head back to the office in few weeks, listening to panicking customers 😅👌🏻🔥

[@grokitall]

how this happened is now out. first they generated the signature file, which was broken, then they ran it through a validator program which was so broken that it said it was ok, then they shipped it straight to 8.5 million mission critical machines all at once as a live update you cannot override and caused most of them to go into a boot loop. any testing at all could have caught these problems before they even got out of the door.

[@TRXST.ISSUES]

Unrelated to crowd strike but I’ve recently considered how smaller optimized models make human level reasoning more likely. It allows for much cheaper multi level abstraction. E.g. 100 GPT 4o minis working across different layers for the same problem. Or miniature models overseeing computationally simple levels of abstraction before the larger model takes on the big meta problem. Like threads in a processor.

[@QC9B4XK6u]

The problem here is this was pushed out as a 'live update' i.e. even though the majority of CS customers were set to update +1 (the default) - one update behind current, you still got the update. Even if you were set to lag multiple versions behind. It has nothing to do with staging. This is common practice for all EDR vendors, S1 and Microsoft Sentinel included. In fact, you won't find an EDR or AV vendor that *doesn't* do this.

[@retheoff]

Thanks for pointing out the problematic policy of "always get updates automatically" or "you are negligent if you don't auto update instantly". I'm so tired of people pushing that. I don't trust my vendors after being burned many times of the years. Now, I only do tiny small biz updates, not the same thing as all this corporate stuff, I know. But man these updates can cause so many problems if not checked before rollout. I always do that and take my time testing the updates. It keeps my client up and running and managed. What I really don't get why not just a lack of apparent testing from Crowdstrike, but also why all these companies are simply taking the updates by them without testing on their own. Maybe they will now!

[@tubaguy0]

Might want to inform yourself on this one before armchair quarterbacking. Security software needs to be updated frequently, so the best practice that is typically advised is to set the sensor updates to N-1, or run one version behind Crowdstrike’s latest version in production. In this case, it didn’t matter, whatever Crowdstike sent caused even old versions to crash Windows. My organization will be having a discussion with Crowdstike about how this occurred and how it will be mitigated in the future. My point is that you should be careful went painting with such a broad brush.

[@RWBHere]

It must have been one of the biggest onoseconds in history. (Tom Scott reference.)

[@Calphool222]

I'm not sure if Crowdstrike Falcon even provides a way to stagger releases. This was a channel file (a data file that gets read) that was released, not an upgrade to the software itself.

[@meandego]

Now everyone knows what Crowdstrike is.

[@bjrnpoulsen6365]

Woohoo. That was a spot on take on IT infrastructure!! Been there and done that. Most companies (if not all…) is one or other form of chaos. Even a +24 months old automation can cause chaos due to author is not there anymore or forgot most of the setup

[@jonogrimmer6013]

Very interesting view into this, thanks!

[@briandoe5746]

As a Lennox daily driver user. This video had my full attention. What distro do you run?

[@latestAiHacks]

Hopefully we can generate our own operating system in the future

[@GabrielRUrbina]

My first take on the outage was, WHY THE HELL were they releasing everything all at once?

[@JeanPierreWhite]

And why was one company able to update of all their customers endpoints?

[@robhannah2794]

A good idea that Apple already does (learned years ago that bricking devices was **bad**), but that end-users of Apple products should also do - don’t install i*OS updates for a day or more.

[@CaedenV]

Yeah... You are a little out of date. Im totally with you on the ideal, but there are so many layers of systems and security software, and insurance requires that certain types of updates are applied within days if not hours. Unless we are moving to a <a href="#" class="seekto" data-time="61">1:1</a> of it staff to production staff, most places just aren't going to have the ability to fully vet and test every little update that comes in every couple hours. That all being said, if you had your keys handy, it wasnt too hard to fix this issue, and most of my friends and I were back up and running at the infrastructure level in a couple hours. End user devices took a little longer, but even then we had them back up and running in less than half an hour of when they walked in the door. The real magic to me was aad and laps... Being able to give remote users the local admin password and bit locker key, wak them through the fix, reboot, and then remotely reset laps so that the local admin creds were no longer known by the end user... That was way easier than I had expected. Saved a lot of driving and shipping!

[@Matt2010]

Ok sir yes you're right, however it did have a deep impact on the Economy, which no needed backend should be allowed to cause ever, IMO.

[@the.blue.raven7777]

If C levels had ears to listen

[@keithrice9281]

Is this perhaps a win for Windows Defender? Being as though it’s native to Microsoft? I understand the same risk applies, but seems to be a lower risk vs Crowdstrike and S1

[@genx7006]

If I was the head infrastructure guy at CrowdStrike, I would be calling in sick until I exhausted all my PTO.

[@weredragon1447]

It was...Aliens!!! 😂😂

[@JeanPierreWhite]

you know it.

[@JeanPierreWhite]

All of these corporations didn't suddenly get amnesia and forget to do change management and release control. These companies must have abdicated change management to Crowdstrike. They may not have realized they had abdicated this responsibility, but they are now lol.

[@user-nj5bx8py4h]

Basically 1 person at crowdstrike clicked a button. 1 person. Thats all it took.

[@bananadane]

Solid vid, No major notes

[@raymond_luxury_yacht]

Us voting infra on this?

[@F.a797]

Why is software able to cause kernel panics in the first place? Why are operating systems not designed to be resilient to this kind of faults?

[@SergioBallestrero]

Because this piece of software, like an antivirus, practically becomes a piece of the OS kernel itself. A dangerous design choice by the software. "Monolithic" kernels (windows,Linux) cannot protect from a piece of their own code. Read about "microkernels" .

[@JeanPierreWhite]

Because Windows sucks.

[@chartingwithliv]

Literally bought long term options, this was a firesale. Ez money

[@krunkle5136]

"it's not that biggadeal"

[@PhrontDoor]

I said the same thing and some clueless wannabe scrubs push back with "oh yeah, and when pii is compromised and ransomware invades your system then who's gonna pay"...

[@6lack5ushi]

yes a process error. Sandbox and iterative deployment. No brainer but having a driver at system level that is a boot start, and can push new code to a billion machines... isn't that just a backdoor into a billion machines?

[@wisedonkey_]

Bingo!

[@sleepwalker6825]

Run Corporate machine Updates ( Isolated ) in a contanier .. Test then Run

[@vmura]

ClownStrike are a joke. also why cant OS be anti fragile

[@Me__Myself__and__I]

Almost all software is very fragile. Largely because software is extremely complicated and almost no one has the skill to design and write robust software. Making software less fragile is a skill and the vast majority of devs spend zero time trying to learn or improve that way of doing things.

[@ticijevish]

I bet the bad update that they pushed to prod without testing or staging... was coded by LLM AI! 😆

[@casualuser5527]

Billion dollar ad for Rust 🦀😂

[@eSKAone-]

But what if not updating compromises your main systems security? Because often times they close security wholes. You can't put all your main systems offline. And in the event of a security breach your insurance will make a no no face when seeing you operated with non actualized systems.

[@boonkiathan]

its a wakeup call for all the automated updates, system privileges we allow 3rd party softwares to have, and over reliance on a single vendor including Microsoft a single Windows Home user can be bricked any day an enterprise running hundreds of business-critical systems need have more resilience, concern for outages and fallback systems(an independent 2nd level system for continuity). you hang your entire airline business on your 100% Windows computers and Azure cloud and expect to pray for no rainy days? sue for your losses??

[@brianhershey563]

Ahh this makes me miss daily scrums... well wait hehe

[@ydmoskow]

So, will crowdstrike be sued to death?

[@DaveShap]

Probably not. Vendors are insulated here, generally speaking. A bunch of account execs will have a bunch of angry calls though

[@Me__Myself__and__I]

They will likely lose customers. I believe Elon said his staff removed Crowdstrike from all systems, and I highly doubt companies who uninstall it will reinstall later.

[@azamai]

Go build testing soft for corporations

[@jamespong6588]

You are uninformed on the issue

[@mroctober3657]

A good advertisement for Apple.

[@vi6ddarkking]

And Linux

[@Voorhees94sg]

@@vi6ddarkking Maybe it won't be the year of AGI, but the year of Linux :D

[@aliasgur3342]

No it's not. Falcon runs on Linux and MacOS too and there is no reason the code for those platforms would not be equally susceptible.

[@aliasgur3342]

​@@Voorhees94sgI've been hearing for 20 years how Linux will rule. It's an OS for people who like to tinker and avoid real work (edit - I'm referring to desktop)

[@erkinalp]

@@aliasgur3342 driver architecture is different there,

[@jackkendall6420]

Great advertisement for Rust lmao.

[@drmarioschannel]

BSOD is such an american tradition.

[@shooteru]

Follow the money. Someone was shorting the company the day before. J Bravo channel detected with markers the day before and let everyone know to short it cause is gonna go down. Next day we had the crash. We live in a simulation lol

[@BunnyOfThunder]

"Every company has lots of legacy code." One of the best things AI can do for code is increasing productivity enough that companies can address this more.

[@Me__Myself__and__I]

That wouldn't make any difference. The legacy code is often there because "it works", often no one knows how or why it works and changing it both costs money and is considered risky.

[@BunnyOfThunder]

@@Me__Myself__and__I So imagine you knew how it worked, the cost of changing it was near zero, it had robust pipelines for testing and deployment, and you had the resources to keep it up-to-date and flexible. It isn't safe to just leave it. It needs security patches and the infrastructure that it runs on may stop supporting it. You ideally want to keep it modern and healthy and well-exercised, but companies can't afford the time it would take to do this. If AI could amplify productivity, companies could tackle these problems rather than ignoring them.

[@Me__Myself__and__I]

@@BunnyOfThunder Wow, in that world everyone must also be billionaires, all girls get ponies on their birthdays etc. That is unrealistic, even with AI the cost is never zero. Also, not everything needs security updates. Security updates are required when there is a problem. Truth is a lot of older code was simpler, often had zero dependicies and was on average much more secure than modern code. Plus just because the code is old that doesn't necessatily mean it can't be deployed on modern servers, For example most code written to run on Windows 30 years ago still runs without change today.

[@BunnyOfThunder]

@@Me__Myself__and__I First of all, there's no need to be rude. Second of all, patching legacy software isn't a theoretical problem. In my work, it's a pressing reality every day. It's not a question of if your legacy code needs updating, it's when. They all get problems sooner or later, and when your code is ancient and nothing supports it, the fix gets very expensive and time consuming. Thirdly, the entire premise of my comment is that AI makes the cost of this more feasible. No, the cost is never zero, but it can go from "too expensive to be worth the benefit" to "cheap enough to be worth the benefit." With AI, the reduction in cost may end up being *very* significant. It may (likely) end up being multiple orders of magnitude cheaper than it is now. If it becomes possible to update a legacy system to a modern, well-exercised one for a few hundred bucks worth of AI tokens, every company is going to take that deal. It's more secure and cheaper in the long run.

[@robertbutsch1802]

Something is not quite right with this explanation. It requires that all the airlines, banks, hospitals, grocery stores, etc.,etc. that got blue screened failed their due diligence to properly vet this “patch” before fully deploying it across the enterprise at the same time that CrowdStrike failed to test it properly before distributing. The odds of all this happening at once are minute. Somewhere I saw a tweet from somebody at CloudStrike saying specifically that this was not an update of any sort. Apparently it was some sort of content delivery. I’ve seen Windows “channel files” mentioned somewhere. I think this may have been analogous to distribution of new antivirus signature files.

[@DaveShap]

Lol, no, like I said, the vast majority of IT departments are poorly run. No one knows what best practices are.

[@Me__Myself__and__I]

I'm guessing you've never worked in corporate IT. IT departments were doing a lot of stupid things 25 years ago. Everything has gotten more complicated since then and the average IT employee is less intelligent and skilled since then. Add to that an ever increasing pressure to do everything faster, spend less and reduce headcount...

[@ben_kendall]

@@DaveShap But think of the money they save (not)