Delta Air Lines CEO on CrowdStrike outage: Cost us half a billion dollars in five days

Рет қаралды 29,677

Күн бұрын

Пікірлер: 154

@potato2941 4 ай бұрын

As an IT professional, it was 100% CrowdStrike fault for Delta being down for day 1. However Delta being down day 2 to day 4, is 100% Delta IT operation fault. Most IT departments including mine was up and running 99%+ within 24 hours.

@BrightsBrewery 3 ай бұрын

Chief Excuse Officer - 100's of millions of dollars in redundancies, eh?

@grokitall 3 ай бұрын

but you did not have to fix 40,000 mostly locked down servers, then spend extra time recovering from the logistics problems of pilots and aircraft not being where they were planned to be.

@MattMiller211 4 ай бұрын

This global internet outage is insane! All airlines grounded and i was stock the airport and even banks, media, and offices from the U.S. to Australia. How can CrowdStrike have such a monopoly that could help restore such a massive amount of tech?

@VictoriaAllen-ml8kx 4 ай бұрын

It's pretty concerning. If they can fix this, what other control do they have over our infrastructure? or are we truly in the matrix?

@JoyceMuller-xv6kh 4 ай бұрын

Right? It makes you think about the stability of our systems. But hey, I barely spend time online. When I checked my portfolio with Desiree Ruth Hoffman, we were still in the greens. That’s been the case for 16 months straight!

@AlexYoung-21 4 ай бұрын

Wow, really? I've seen the name Desiree Ruth Hoffman before but can't figure out where.

@VictoriaAllen-ml8kx 4 ай бұрын

Probably from her forecast on Nvidia before the pump. But how are you in the greens with all the fluctuations due to the election and everything else? Can you share her strategy?

@JoyceMuller-xv6kh 4 ай бұрын

Honestly, just schedule a call with her. She has vast knowledge in finance and really knows how to navigate these times. I handed over my portfolio to her so I can focus on my family. These days, things just get scarier and scarier.

@CH-qt4hr 4 ай бұрын

Microsoft should have offered Delta a $10 meal voucher

@TheBalkenende 3 ай бұрын

🤣🤣🤣 nailed it

@user-tm1oy6ck4t 3 ай бұрын

Ooooo. Brilliant! 👏👏👏

@user-tm1oy6ck4t 3 ай бұрын

Delta is a mere shadow of its former self. When i worked for Disney that's all we flew, and it was a stellar experience. NOOOOT anymore.

@christianamaral-c3l 4 ай бұрын

Missed opportunity for the Squawk crowd this morning. Instead of pressing Bastian on why Delta took several days longer than any other carrier to recover, here they are fawning over the Delta One lounge experience. The problem was not merely external.

@NihilSineDeo09 4 ай бұрын

The CEO hinted when he said that, within the industry, Delta is the most heavily invested with Microsoft and CrowdStrike

@HZADAMS 4 ай бұрын

it is a mistake to think that this was delta issue. Reboot had to be done manually for many systems due to the bricking by crowd. Delta servers were spread all over which had to be manually rebooted one at a time for 40000 times.

@Phineas1626 4 ай бұрын

@@HZADAMSYeah, but he had no answer to what the solution is if it happened again next week.

@DavidKen878 4 ай бұрын

⁠⁠⁠⁠⁠@@Phineas1626Of course he didn't. I would presume this isn't something that you just decide on a whim. Has any other CEO gave an answer to that what if?

@Phineas1626 4 ай бұрын

@@DavidKen878 It’s PRECISELY his job to consider those “what if?” scenarios. Yes, I guarantee you other CEOs (or their designees) think about those scenarios constantly. Risk management is a big part of any big company.

@ericp4573 4 ай бұрын

They should get treated how they treat their customers

@vidademikey9790 4 ай бұрын

Cry about it.

@theoldmanofthenorth 4 ай бұрын

Stop using CrowdStrike. They had a similar issue on Linux not that long ago. Or, don't allow updates to go straight through to your production systems before internal testing. Best of luck.

@opmedia_1 4 ай бұрын

Blame game as a CEO? Interesting approach

@jakebeyerle 4 ай бұрын

Lmao. The airline industry has some of the worst treatment toward the public and customers. Pot calling the kettle black.

@Phineas1626 4 ай бұрын

Glad to see someone gets the irony. Maybe Delta should consider suing themselves…

@jpdstan 4 ай бұрын

The outage happening was Crowdstrikes fault, sure. The outage lasting for a whole week is absolutely Deltas fault. When pilots and FAs show up to the gate ready to go and they can't because of an "internal scheduling error", that is absolutely Deltas fault. Delta infra is clearly a house of cards, and i think this money grab from them will make any future vendor wary of doing business with them, it'd be like selling life insurance to a cancer patient

@blev360 3 ай бұрын

Recovery is paramount. They should have been using Rubrik instead of legacy backup tooling.

@mattbrown5949 4 ай бұрын

Free consulting? Wow….ill assume that’s sarcasm.

@tomsalem6458 4 ай бұрын

I would like to respectfully disagree with the Delta CEO, our outgoing flight on July 19, was canceled by their partner WestJet. Moreover our return trip on their partner WestJet was canceled on July 29. The Delta call center could not help us after an hour on the phone. Told us to call WestJet. We called WestJet and waited on the phone for over 2 hours on hold and no assistance. Finally, we had to drive to the airport to rebook our flight that was cancelled by the software that was supposedly fixed. These are the facts and the CEO is either unaware or smoothing over the problem. I personally will avoid Delta in the future.

@HY-hm9gr 4 ай бұрын

You bought a codeshare flight ticket. The operator is still Westjet, not Delta. You have to contact Westjet.

@bryant4422 3 ай бұрын

@@HY-hm9grno, if the ticket was issued by Delta, if delta is down, westjet will not fly the passenger for free if westjet cannot confirm or settle payment

@H3H3official 4 ай бұрын

"When was the last time you heard of a big outage at Apple?" Bruh 😂 How you gonna make $34,214,328 a year and say some ish like that 😭

@DaniEles-rc7ij 4 ай бұрын

when was the last one?

@shastam9590 4 ай бұрын

@@DaniEles-rc7ij I think the point they were trying to make is that Apple and Microsoft operate in fundamentally different spaces. Microsoft is a major player in cloud infrastructure to other businesses, while Apple primarily focuses on consumer products and services. That difference makes any direct comparison irrelevant

@atg2269 4 ай бұрын

different business, Apple is much more consumer based in comparison to Microsoft. cant go out comparing apples to oranges

@DaniEles-rc7ij 4 ай бұрын

@@shastam9590 LINUX is the base for Apple OS... and LINUX is scalable to the enterprise level... Delta needs to invest in internal systems.. like the did in the 70's

@ripplecutter233 4 ай бұрын

wdym, he can just migrate to Apple Cloud Services which totally exists, ez /s

@brandon889 4 ай бұрын

Yall need to team up With HP enterprise and Apple and IBM

@ProfessionalBirdWatcher 4 ай бұрын

Sad state of affairs when the only people holding CrowdStrike accountable are the shareholders.

@aroundandround 4 ай бұрын

You buy software at your own risk. All software will have bugs. Surely Delta read the fine print in Crowdstrike’s terms and conditions while contracting with them, didn’t they?

@chrisriley7613 4 ай бұрын

Crowdstike pushed a forced update out without testing it. They didnt follow their own QA process. Cant blame delta.

@Holly_Unleashed 4 ай бұрын

No, this was not a normal bug. It was an untested, un-phased updated to the kernel level. Inexcusable on Crowdstrike's part. This was completely avoidable.

@Phineas1626 4 ай бұрын

@@Holly_UnleashedMost business problems are completely avoidable.

@joaocardoso7880 4 ай бұрын

An unprepared CEO and company crying about not being prepared ……….. they should let him go

@Groaznic 4 ай бұрын

You have no idea what you're talking about.

@joaocardoso7880 4 ай бұрын

@@Groaznic but I am sure you do 😴😴😴😂😂😂😂 get out of here troll

@joaocardoso7880 4 ай бұрын

@@Groaznic now you really embarrassed me. What can I say back to this response 😅😅😅😅 I am sure you are the one that knows everything 😅😅😅😅😅🥱🥱🥱🥱🥱.

@MamacitaBonita 3 ай бұрын

@@joaocardoso7880What kind of name is Joao? Is that even a real name?

@grokitall 3 ай бұрын

most companies cannot cope with cowboy developers doing things which actively undermine your resiliency planning. lots of people got caught after 9 11 as their backup was in the other tower. same happened with the big european data center fire, when the provider had the backup literally next door, so it caught fire as well.

@LEVELGAZANOW 3 ай бұрын

It’s YOUR failure Delta. Stop blaming others for your ineptness.

@morock1n 4 ай бұрын

So let me get this straight, you install an app in Windows and then that app misbehaves and it's Windows' fault? How is this guy a CEO? no wonder Delta sucks so hard (leg space and most uncomfortable seats). No thank you

@grokitall 3 ай бұрын

when it triggers a well known pre existing bug in the kernel which has not been fixed after a decade, yes microsoft has some share in the blame. it was crowdstrike which crashed the machines, but it was microsofts failure to fix the bug which kept them down and required on site repair and remediation from tech support.

@jayjordan833 4 ай бұрын

I’ve never had a bad experience on Delta Air

@karlking4980 4 ай бұрын

This seems to also be a failure on Delta’s IT department for not testing the CrowdStrike file and for pushing it to all production servers rather than a gradual rollout which can be ramped up after there is confidence in the file. Also, why 40,000 servers? There are better alternatives that don’t require the huge army of developers and systems people to maintain. And to apparently recover after an outage. The raised floor space and data center environmentals like redundant power, etc. must be very costly. Your IT leadership built an expensive empire.

@HZADAMS 4 ай бұрын

you have to understand how endpoint security with CRWD works. it doesnt give administrators a right to accept and schedule a rollout.

@karlking4980 4 ай бұрын

@@HZADAMScould Delta have tested the change before moving it to production? This process for accepting CS changes seems to fly in the face of best practices. No testing and an all-at-once promotion to every production server. Hence the result we all witnessed. Thank you for the CS information!

@katscandance 4 ай бұрын

No literally every company using crowdstrike got hit with it

@karlking4980 4 ай бұрын

@@katscandance Thanks! Shouldn't the CS file have first been distributed to their customers non-production servers? (Rather than directly to production.) Very interesting relationship between CS and their customers that no testing is performed before production. I wonder how CS tested this file (that resides in the Kernel) without experiencing the BSoD. More likely, CS did experience the BSoD but did not react properly to it. Just my two cents. Thanks for the reply!

@mikkpal 4 ай бұрын

@@karlking4980 I believe I read somewhere that CrowdedStrike actually had a bug in their automated tests, which resulted in them not noticing it before the rollout. However, I think that big companies should be able to test it in a non-production environment before deploying any updates on all their servers, as that is considered good practice.

@matt.stevick 4 ай бұрын

He was not a happy customer … at all. 😬

@aerialee3950 4 ай бұрын

Why are the other two reporters there when only one is talking?? It's a weird visual.

@Aviator526 4 ай бұрын

What does Joe do on this show? Honest question.

@CamiloSanchez1979 4 ай бұрын

I hope they sue them to oblivion. This overreliance on vendors with single point of failure is insane.

@lordgarion514 4 ай бұрын

It was Delta, and everyone else, that was being cheap and only relying on one system. An intelligent person would say an airline should have at least one backup system....... And so should every other large corporation.

@Phineas1626 4 ай бұрын

@@lordgarion514Or maybe at least ask ‘what if…?’

@grokitall 3 ай бұрын

@@lordgarion514but lots of people who were hit used release n-1 on their production systems and n-2 on their backups. unfortunately the crowdstrike rapid release process gives the impression that this applies to that as well, when it does not, so what was expected by their customers was that the release process would be engineered to quickly stop a problem and recover, and that this would be spotted on the customers test machines. this then would not be applied to the n-1 systems in production, and if it took long enough to hit them, it would not hit the n-2 backus machines. what actually happened is that this flag only applied to the core kernel update, not the two files in the channel update, so when it was shipped untested, it kept shipping for 90 minutes, taking down the test machines and production machines at the same time, and then when it shifted to the backups it killed them too. the only thing that crowdstrike got right about this was saying yes it was us.

@FoundingFathersUSA 4 ай бұрын

Free lunch at Redmond HQ cafeteria for 100 years.

@xingxing85 4 ай бұрын

So the rule of thumb is to avoid Microsoft. I hope people learn from the lessons haha

@DaniEles-rc7ij 4 ай бұрын

LINUX is out there.

@katieadams5860 4 ай бұрын

They published a bad update last month too on Linux, a bad software update for any OS can take it down

@grokitall 3 ай бұрын

@@katieadams5860but the boot loop problem is specifically a windows bug, and some linux systems are build using capabilities which just roll back to an earlier install on power cycling, which can be done by unskilled people. it was the boot loop problem which kept machines down for so long.

@danielking104 3 ай бұрын

Don't tell me crowdstike doesn't have insurance.

@grokitall 3 ай бұрын

most insurance has exclusions for negligence, as do the liability exclusions in eulas. this means that when it gets to court, the get out of jail free card won't work, and the insurance won't pay.

@yukinun311 4 ай бұрын

this should happen more when you think about how wrongly people decided that brutal rich companies made peoples suffer by not paying their taxes. Ago 1/8/24

@vincentjiang2727 4 ай бұрын

Should have say its caused by another country's hacker.

@LivesMatter-t7z 4 ай бұрын

Kudos to Delta!!!! There needs to be a congressional hearing. Microsoft needs to pony up as well because if you're going to let certified security companies into your core OS processes, you need to make sure they simply "tested" this. A 15 min test would have caught this. Instead, they dumped it out to hundreds of million machines globally. I've delivered software all levels from actual developer, tester, support engineer, PM, and I have NEVER seen anything like this. When you deploy, you deploy to a handful of low-risk clients first (NOT aviation, hospitals, emergency services... etc !!!!), wait a week, then deploy to the rest of the world. STUNNED!!!!! Multiple failure across all organizations of CrowdStrike. They need to go out of business.

@benjamink8448 4 ай бұрын

Wait A WEEK? These were continous security updates, not a new program version. There may be multiple updates per day due to emerging threats. Crowdstrike did not test properly, and rolled it out to everyone at the same time. They are at fault. Microsoft does not have responsibility here. Delta chose to install Crowdstrike.

@grokitall 3 ай бұрын

@@benjamink8448on the contrary, microsoft has had multiple instances of boot loop problems over the last decade, and have not done enough to prevent it. yes crowdstrike took 8.5 million mostly mission critical systems down, but it was microsoft not fixing the boot loop problem for a decade which kept them down.

@rogersmith1203 3 ай бұрын

I wouldn't trust this Texas based cybersecurity company, I'll stick to Kaspersky thanks.

@industrialist2002 3 ай бұрын

Delta is no better than American, JetBlue and United. They want to charge a premium more meager service. Whilst Bastian teavels on a private jet, catered with 5 star meals and expensive alcohol.

@lordecaroline 4 ай бұрын

The interviewer kept interrupting and it was so rude

@lali_sanchez_blog 3 ай бұрын

I totally agree with this guy, even if they didn't have the most and adequate IT equipment, they pay millions of dollars on support from MS and Crowdstrike. Crowdstrike at least admit what they did was wrong MS didn't, they are just like oh well sorry this happened, it was a minor bug, like seriously? Delta pay for your services too, so you need to work with Crowdstrike on how to prevent this in the future.

@loganwells9950 4 ай бұрын

CTO’s always go with vendors who are convenient

@HelloWorld-hb7yt 4 ай бұрын

i like this guy.. he speaks the truth.

@NoLMAOnade 4 ай бұрын

I like him as well! He's honest, really good-looking, and an all-around amazing person! 😍😍😍

@christianamaral-c3l 4 ай бұрын

Actually no. He’s just deflecting. Delta took several days longer than anyone else to recover from the same event.

@MamacitaBonita 3 ай бұрын

The best of the best!!!! Y’all just jealous

@grokitall 3 ай бұрын

@@christianamaral-c3ldelta took longer because they had a wider exposure than everyone else. because this required hands on fixes by skilled tech guys to fix, it took a while.

@damiensanders1804 4 ай бұрын

Lotta glazing going on in the comments here 😂 People bein like "LOVE THIS GUY!!!" gtfo he don't care about you or your family

@Auspiciousblackcat 4 ай бұрын

Those 4 guys want age 77 so bad

@darrendent8288 4 ай бұрын

$PANW FTW!

@sirloin869 3 ай бұрын

yea,u aint use enuf paper straws & that made internet go sad

@VinothDevakumar 4 ай бұрын

CRWD had one job - To prevent IT outages!

@7mmLongRanger 4 ай бұрын

I, for one, sleep better at night knowing the Ukrainian gangsters at CrowdStrike have access to the kernel on my laptop.

@thomasmerrill3585 4 ай бұрын

Palantir???

@goodmushroom 4 ай бұрын

what about palantir?

@MSalman1 4 ай бұрын

This guy seems salty on how rich the top tech companies are lol

@ozgurinsan 3 ай бұрын

Goodluck with non existant apple server.

@amatuer2 4 ай бұрын

I got an uber $10 voucher😂

@EU.N.ChaoZhiChi 3 ай бұрын

😮👍👍

@SAT0SHI.NAKAM0T0 4 ай бұрын

™️ 🥊 KO

@brandon889 4 ай бұрын

Apple would never

@redscorpion9325 4 ай бұрын

I don’t overlook Apple either,it could happen to them too,usually Updates is not always good if not properly tested

@gberlucchi 4 ай бұрын

An honest and intelligent CEO….rare indeed.

@NoLMAOnade 4 ай бұрын

He's brilliant... incredibly handsome... and an all-around fantastic person... A true rarity! 😍😍😍

@christianamaral-c3l 4 ай бұрын

Remember, Delta took several days longer than any other carrier to recover from the same event. Just deflecting here.

@grokitall 3 ай бұрын

@@christianamaral-c3lthe guy from delta pointed out that they had some of the highest exposure, with 40,000 machines affected. if you ignore traveling time, and assume there are enough techs on site, and a repair time of ten minutes per machine, working 24 hours per day, you are still talking about 277 man days to get them all back up, even before you add in the time to fix machines which were not designed to cope with being crashed. when you add in the restore time for broken machines, travel time, and not being able to get techs in the right place to fix machines, those numbers can go up very fast.