Bankrupt In 45 Minutes From DevOps | Prime Reacts

[@darylphuah]

Man, the amount of things that went wrong here - Not removing dead code - Re-using old feature flag (wtf were they thinking) - No deployment review and validation that proper code was deployed on all servers - No post-deployment monitoring - No observability /traceability metrics? (They couldn't immediately pinpoint that one server making way more trades than it was suppose to?) - No Kill switch Any one of these in place would have prevented the whole thing or minimised the damage

[@nnnik3595]

No rollbacks?

[@adriangodoy4610]

​@@nnnik3595when the stated deploy process is copy paste the new binaries into the server... I guess rollback it's not even a possibility

[@rachitpulhani3478]

they did rollback, only the deployments that were working correctly though @@nnnik3595

[@monad_tcp]

@@nnnik3595 no staging ? no tests ? no automated anything ?

[@darekmistrz4364]

Don't blame them on reusing old feature flag. Do you know how much the new one would cost? Their price is so high that it's so obvious they reused an old one. Look how much they saved on it! /s

[@wlockuz4467]

This is why you need a physical kill switch, something that blows up your server room.

[@morganjonasson2947]

OR you could just make sure your web app is made out of flask as a development server without WSGI. That way the server will immediately crash as soon as you reach thousands of users in a short span of time. Usually one doesn't want this to happen, but if you are scared of high volume and don't want to deal with it, than that solution is perfect lol.

[@Its.all.goodman]

​@@morganjonasson2947haha 😅

[@smtp_yurzx]

They laughed at me and said I was crazy! Who's laughing now?!

[@awmy3109]

😂

[@Tekner436]

just put a small incendiary device on your main fiber line

[@spacemonky2000]

Imagine hundreds of millions of dollars are being dumped every 10 minutes while you try to debug your deployment in prod. jesus christ

[@ninocraft1]

the devs need alot of therapy after that xD

[@FourOneNineOneFourOne]

I worked at MS and I saw a team showing us a demo of a stock trading API. The dev forgot to switch the account to the test one and used real company account to execute the trade worth $30m of IBM stock. He started getting calls, the trade couldn't be reversed, but the trading floor closed the position in small profit, so it now just a funny story, nobody got in trouble. Couldn't imagine if it went the other way.

[@BrandonSorenson-fb3gg]

This is why robust testing is very important....and a staging environment as identical as you can get without actually making trades is crucial

[@1adamuk]

I'd just run out of the door tbh can't deal with stress like that.

[@rdil]

Imagine getting an email saying something is wrong and seeing this much happen and not immediately killing the system

[@doresearchstopwhining]

Imagine relying on someone outside of your company emailing you to tell you your shit is broken and now your company is bankrupt.

[@vikramkrishnan6414]

Email is not an adequate alerting mechanism for shit like this.

[@crtune]

The people building this software are not themselves traders, nor are they regulators, and there are some budgetary problems in seeking to write every possible screw up and related protections into even the most heavy budgeted trading software. This install and uninstall problem clearly shows the problem with a design for the "parent order/child orders" design and the way set up did not provide for network wide deactivation of this capability all without removing the PEG software. A better design would have provided for ability to turn off PEG entirely without uninstalling.

[@FrankJonen]

@@vikramkrishnan6414IMAP Push is just as immediate as any ping.

[@Rexhunterj]

@@vikramkrishnan6414 I'm amazed they didnt have have an app and push notifications for this lol.

[@shashantr.9380]

I myself work as a System Engineer in a HFT company. Whenever there are any discussions regarding Risk Management, almost every time we get to hear about Knight Capital's incident

[@harleyspeedthrust4013]

risk management. why do they call it ovaltine

[@boredphilosopher4254]

@@harleyspeedthrust4013why not roundtine

[@harleyspeedthrust4013]

@@boredphilosopher4254 the mug is round

[@steffenbendel6031]

I remember the Knight incident. At that time, we traded FX (currencies) with there Hotspot subsidiary. Vibes out the pensions of the people working there. I myself caused a similar incident in our company: When a client changes the amount they wanted to trade with us, that created some adjustment trades. Either selling or buying more. After a change there got something switched. Execution for sell trades where added to the amount to buy and visa versa. So the number of shares you still need to trade got bigger and bigger. Luckily, I always insistent on keeping the trading slow and not very aggressiv. So it only traded back and forth for a few hours in the night, before we disabled and fixed it. Still had to file and incident report with the regulators.

[@neoplumes]

Article: "DevOps is broken!" Also Article: "Nothing about this is DevOps!"

[@composerkris2935]

It's actually is a perfect example of why DevOps is needed.

[@monad_tcp]

its like people that say the windows firewall don't work (its turned off most of the time, or someone put a too broad rule there)

[@Waitwhat469]

@@composerkris2935 Right! Like engineering sent that shit over the wall and they deployed it. No one seemed responsible for it in prod, it's just was allowed to be. No one monitoring their new feature, no one designing automated deployments, backout plans, rollouts, etc. Just one big buerocratic process shoving new code in one end, and getting these results on the other.

[@TosiakiS]

It's actually "lack of devops" when they say "because of devops." Yes it's a confusing title.

[@hbp_]

I'm still not convinced that an automated deployment was what would have prevented this disaster. Surely if it was built perfectly, perhaps written in Haskell.

[@alexandrustefanmiron7723]

U never ever ever reuse the same opcode for a new functionality ever.. never ever ever ever ever!!!!!! If you remove and deprecate you throw an error of sorts... But never ever ever repurpose an API endpoint!!!!!!

[@tryfinally]

their code was probably cursed and adding a new flag was too difficult

[@astronemir]

How cursed of a codebase do you think they had to reuse the same flag code? I bet they only removed the dead code for the express purpose of reusing its code, but they didn’t even separate the deployment into two. Remove first, make sure everything is working, then reuse it (don’t ever do it), but if you’re doing it, don’t do it in the same step.

[@grimsas]

As a bounty hunter I love when there's some obsolete API leftovers accessible in a codebase. Makes life much more exciting 😉

[@kneesnap1041]

Unfortunately, it's not considered bad practice to use single character command line flags. Most likely it was something like reusing -p. I can point to tons of unix programs that are using pretty much any single letter command line flag they could possibly use. I think it's a lesson that in critical applications, --power-peg should probably be used instead of just -p

[@rain_deer]

I worked at a bank a few years ago, and the team I was on had a completely manual deployment process. We had a round robin table of who would be in charge that week and they would have to go around through all the sub teams and collect a list of manual instructions. And this was never prepared ahead, you really had to fight tooth and nail to get those instructions. We'd then wrap all up and send it off to the 1 and only 1 prod support guy that had been on the team longer than I was in the industry. Eventually that guy turned 56 and it scared the hell out of management, and they blocked the entire team from deploying anything. I was leaving the bank at that time so I don't know how it worked out, but every now and then I wake up in a cold sweet thinking about it.

[@Nil-js4bf]

Somewhat surprised that banks don't have better automation for this stuff. But then again, banks still rely on ancient code doing batch processing on csv files... If it ain't broke, don't fix it. The problem is, when it does break, it can be catastrophic.

[@monad_tcp]

That guy must have won a very sweet severance package when he retired

[@monad_tcp]

@@Nil-js4bf "If it ain't broke, don't fix it. The problem is, when it does break, it can be catastrophic." If you think like that you deserve everything that happens because of complacency.

[@WinterMute_df]

Man, it sounds almost like the bank I'm working at (one of the biggest). I'm getting a deployment to do every other month without even a clear understanding of how all components of the platform work (because of 'need to know' and such). Without me calling this ONE GUY when something seems odd, everything would go in flames on Monday twice already.

[@Tekner436]

@@Nil-js4bf "oh shit it broke, call ted" "uhhh ted died 6 years ago."

[@nexovec]

What's the worst thing that could happen... *Siphons the entire AWS account

[@awillingham]

I thought that’s where this was going. “Our load balance was configured incorrectly and we allocated 500,000 instances which logged 5,000,000,000 errors and crashed cloudwatch and our s3”

[@monad_tcp]

​@@awillingham I though they did something wrong and fired 1Billion instances and that cost $500M USD of AWS stupid charges. But it was even more amusing. At least if that was AWS you could immediately cancel your entire bank account, claim hack, make a public PR storm and never pay it back.

[@FourOneNineOneFourOne]

@@monad_tcp Believe it or not, AWS has built in triggers to prevent anything like that from happening.

[@monad_tcp]

@@FourOneNineOneFourOne That means it already happened at least once, someone did exactly that and they implemented stoppers to prevent it.

[@vikramkrishnan6414]

At that stage, I would just walk into the server room with a flame thrower and burn it down after around 10 minutes of this

[@ea_naseer]

remove the plug from the wall😂😂😂

[@vikramkrishnan6414]

@@ea_naseer I want to be extra sure. Also, are there some copper ingots you would like to sell me?

[@ardnys35]

i was thinking sledgehammer. or a stick explosive or flooding it for the extreme measures.

[@genxer1824]

@@ardnys35 Just nuke the entire site from orbit.

[@monad_tcp]

First 5 minutes I would go to the server room and press the red button and kill the power

[@demolazer]

I have a close family member who works high up in data architecture at a major bank. You would not believe how batshit their dev processes are.

[@larbiishak1974]

surely they heard and learned from this story

[@mcspud]

I have been there. Can confirm.

[@emptydata-xf7ps]

Blame it on the CIO. The stakeholders are the pushers and the CIO needs to make it known to them how costly some mistakes are.

[@adriangodoy4610]

The only way to improve a bank system is: create a completely new bank, make it grow until it's bigger than the original, absorb the original. You can't really touch that COBOL in. Meaningful way

[@stevezelaznik5872]

I used to work at an insurance company. However bad you think it is, it's worse.

[@Gruby7C1h]

I remember reading a story about it few years ago, the real lesson here is: never "repurpose a flag".

[@harrytsang1501]

I think it's ok as long as you space out the deployment between dead code removal and feature flag on But yes, lack of separate feature flag, lack of kill switch, lack of knowledge to even kill the server

[@mennol3885]

Why is this not in the lessons learnt? In would not pass my review. NEVER repurpose a flag. There is zero cost to a new flag, and if there is not then that is a problem by itself.

[@tylerbreau4544]

I think it's foolish to only take a single lesson. There's also killswitch, automated deployments, accurate rollbacks (how did the rollback not stop the power peg system from running?), etc.

[@arcanernz]

Infinite loop + high speed trading, what could go wrong? I think the problem was they didn’t have any anomaly detection and mitigation. A bad deployment caused this but the rollback made it worst, sometimes you can’t test for every scenario hence why you need anomaly detection and kill switches.

[@andrewyork3869]

HFT is so fast I challenge what good a anomaly detector would actually be.

[@darekmistrz4364]

@@andrewyork3869number of trades per minute? number of type of trades per minute?

[@arcanernz]

Better than nothing

[@CoderDBF]

@@andrewyork3869 About 400 MIllion worth at this point. The system would have halted a lot sooner.

[@ab-nb5xg]

Normally you do have to pay good money for a power pegging but $400 million is probably a bit steep

[@_Lumiere_]

Entering findom territory there

[@SaHaRaSquad]

Very low pegging on investment. Especially as it only ran on 1/8 of the capacity.

[@NoComGaming-uq1oq]

Steep?! it's effing vertical

[@anthonyisensee]

As a DevOps engineer, this story shows exactly why you need good DevOps, or at the very least good engineers that can do good DevOps.

[@Veri7a]

if not, you auto deploy PegOps

[@mlc1610]

DevOooooops

[@wormius51]

I never worked on something this big but when I work on something, the guy just wants the thing to do a thing so I am pretty much doing all the front back ends, devops, testing ect.. Do you have dedicated people for each thing in the stuff you worked on?

[@cornoc]

@@wormius51 yes that's how things usually work when a company gets big enough.

[@ugh_dad]

for sure, this story seems like the title should be Bankrupt in 45 minutes from Lack of DevOps

[@chrisalexthomas]

I remember hearing about this when I was working at a finance company back in the day and I couldn't believe it. Every time I see this article I still read it, despite knowing the history already because it's just so damn funny. Who doesn't love a story with a protagonist called power peg?

[@raulsantana163]

"The code that that was updated repurposed an old flag that was used to activate the Power Peg functionality" The article covers the deployment part but this is it's craziest thing. For such impactful functionality, they should have just deleted everything and reused nothing.

[@khatdubell]

The last place I worked I made deleting unused code my religion. I deleted millions of lines of code. When I left there were plenty of still unused code that needed deleting.

[@catgirl_works]

This one really stuck out to me too. Do not EVER reuse flags. If, for whatever reason, you absolutely have to reuse a flag, do not repurpose that flag in the same release that removes the old code. That is a disaster waiting to happen. The old code should be completely removed from the system long before you even think of reusing a flag.

[@jasondoe2596]

​@@catgirl_worksExactly, I'm surprised Prime didn't mention this.

[@oleksiistri8429]

​@@khatdubellyou shite a lot, sir😊

[@tymak_cz]

Yeah...what could go possibly wrong, if you try to repurpose code, that was 8 years dead.

[@danielvaughn4551]

DevOps is shorthand for "job security"

[@theangelofspace155]

It's t that SecOps? As in DevSecOps?

[@cornoc]

@@theangelofspace155 it's all just words in the ether

[@TheEVEInspiration]

Automating deployment can also automatically deploy errors, introduce new errors, or be done in an environment who's state no longer represents the tested state in a critical way. Mistakes anywhere along the process can always happen and human supervision is always required to make sure things are going right and if not, to react to the unforeseen/unhandled situation promptly.

[@autohmae]

Yeah, automated deployment of the wrong thing is DEFINITELY a huge problem, but part of the idea of DevOps, especially GitOps, is can you made it a pull-/mrge-request and have review of it.

[@salvatoreshiggerino6810]

I knew it was a scam when I was on a team where they had hired a DevOps specialist who didn't know how to code so nothing was automated, deploying just meant copying individual files to the server and restarting.

[@adriangodoy4610]

Devops -> devs do operations. Companies so I will hire a team of non devs and put them in operations and will call them devops .

[@randomdude5430]

I would never hire an devops who has not been a dev before he switched. I saw a lot of ops guys jumping on the devops train and having no clue about what they are doing.

[@salvatoreshiggerino6810]

@@randomdude5430 You're not using your brain. You hire a rando off the street who vaguely knows how to turn on a computer, pay him accordingly, then you sell him to your clients at full [meme role du jour] rates and then you laugh all the way to the bank.

[@georgerogers1166]

@@salvatoreshiggerino6810 It's called having ethics.

[@georgerogers1166]

@@salvatoreshiggerino6810 But in that case you aren't hiring, your clients are.

[@faithful451]

Realistically their most inexcusable failing was not having a post deployment review to make sure everything was good (all servers in expected state), etc. There are always gonna be suboptimal processes, and things that are manual that shouldn't be, and sometimes not enough staff on the team, or management won't pay for X tool, or whatever, but the one thing you can ALWAYS do is a proper checklist of what was supposed to be done, and making sure it got done.

[@mattymerr701]

I bet they didn't even have a list of all the servers

[@Refresh5406]

"Bankrupt In 45 minutes from every single solitary individual in our company being a monumental idiot"

[@adriangodoy4610]

I bet a lot of people involved were saying openly to management that it was a bad idea. But management wasn't having any of that complaints

[@shoooozzzz]

Best outro yet. "The name is.... The PowerPegeagen"

[@sorcdk2880]

Having worked with software where mistakes could potentially cause similar sized losses, I was always a bit amazed at how small the team was (3 people) and how little management cared to take extra precautions. At least I had pushed to get some good automated tests, and we did end up putting some other procedures in place over time, but it really felt like we were just lucky that nothing too bad ended up happening before we got a more safe setup in place.

[@tylerbreau4544]

It also is apart of the developer's job to inform management of risk and what can be done to address the risk. Any manager who refuse to invest in risk countermeasures within reason does not have the company's best interest in mind. With that said, it is important to note that risk management is a balance, hence the "within reason". Just because a potential problem exists doesn't necessarily mean it's justifiable to spend 6 months of development time fixing it. And it is the team leaders and manager's jobs to weight the cost and risk and determine the best course of action - Devs explain the risk and managers decide if it's worth the cost to fix. If you keep a paper trail at least you can cover your own behind.

[@pohjoisenvanhus]

Being able to roll back on a moment's notice seems to be pretty important, huh.

[@khatdubell]

The rollback fucked them harder.

[@fulconandroadcone9488]

Let alone roll back, just shut the thing down would be impressive.

[@katrinabryce]

But in this case, the roll-back made things worse.

[@vikramkrishnan6414]

Rollback made this worse

[@pohjoisenvanhus]

@@katrinabryce Yes, a rollback as badly botched up as the roll out was. It sounds like their fundamental problem was having both poorly understood legacy code and a legacy server in the mix.

[@JackDespero]

There is always a kill switch. It is called forcefully unplugging the 8th computer with Power Peg from the electricity net.

[@careymcmanus]

At my workplace we have a replication of our production environment (sandpit) which as devs we deploy to and tested before devops deploys the same changes to production. Last year the person who did the deploys to sandpit left and I took over. The process was a list of different steps that all needed to be done correctly and as someone with adhd I can't get that right all the time/often. As it was a sandpit environment the only harm it caused was the ridiculous amount of delays in getting it all working but it drove me up the wall. I was able to convince my boss to give me the time to completely overhaul the process so that it is now just a simple one line command. We haven't had a single deploy issue since and also the DevOps team loves me now because it made their lives easier.

[@agusaris5031]

Imagine blaming “DevOps” when you still copying those files manually which is against DevOps’s principle itself

[@Tekner436]

but... the article is about why you need good devops practices... lol

[@KulKulKula]

"Terraform deez nuts" After dealing with this M$ piece of .... every day, I cannot agree more

[@carlsjr7975]

Terraform is ass.

[@levifig]

The irony here is that the issue was caused precisely because of a lack of DevOps procedures…

[@Eagledelta3]

I fail to see how the title of the article matches what was discussed. DevOps has always pushed for automated deployment processes (or at least as automated as humanly possible) to limit human error. In other words, the idea has been to apply some Dev processes to the Operations process. NOT to replace operations with developers NOR to is it to make the operations team into a development team. Like Agile, the original ideas behind DevOps have been hijacked by managers and companies to get what they want from it rather than actually apply the benefits within those ideas. Nor have either of those ideas ever meant to be "This is the only way to do this" kind of attitude.

[@composerkris2935]

Yeah, everything they did wrong goes against everything I have ever been taught about DevOps. Just one giant oopsie after the next. If anything, this tale demonstrates why good DevOps practices are needed.

[@Tony-dp1rl]

"repurposed a flag" ... WTF would you do that!? lol

[@dmurvihill]

I worked on a high speed ad bidder around ten years ago. The kill switch was literally the first thing we built.

[@vikramkrishnan6414]

Cash equivalents = LIBOR bonds and short term US bonds (typically < 1yr), i.e. bonds of AAA rated countries with near to no interest rate risk

[@karmatraining]

I love this new "Humorous Energetic Sports Commentator, But For Obscure Coding Topics" genre

[@bluecup25]

Imagine being the poor dude who forgot to copy paste the new files to the 8th server.

[@fulconandroadcone9488]

Worse yet, coping it 8 times, and twice on one server

[@tomkatdev]

This is a new record for how hard i've laughed with prime. I can't even type...I may die of laughter whilst typing this on my inadequate keyboard.

[@lashlarue7924]

slow clapping 👏 on this one for sure 😂

[@user-qr4jf4tv2x]

even if you automate the automation can also create its own problem. its like using triggers in a database where you forget it eventually.

[@mariano.pualiu]

Even at Pixar they were able to call the TI department and ask them to unplug the servers right now! to prevent continuously deleting of the assets, didn't help much, but they were able to do that

[@mattymerr701]

Every company has a Texas Instruments department, right?

[@nikarmotte]

2:13 that made me get a laughter for a few minutes. Man you made my day

[@TheOtherNEO]

Can't recall when and who, but there was some broker who's software developers didn't realize that bid and offer means the reverse in stonk trading.

[@BlazingMagpie]

Buy high, sell low

[@Alex_Cevi]

The tism is really firing in this video .. I love it

[@TobiMetalsFab]

I remember once asking "What's the worst that can happen?", and then proceeded to be in pain for the next two years

[@Phaceial]

Extremely high level - Market making are the people that exchange stocks for cash. They give the market liquidity.

[@khatdubell]

“Market makers connect sellers with buyers” is probably a better description.

[@DaveThomson]

@@khatdubell Its both. There has to be liquitity in order to sustain the market.

[@darshanchaluvaraju]

Automation is the core principle of DevOps and The statement "Copying the code to the 8th server (Seems manual to me)" itself kills the concept of DevOps principles. The symbol of DevOps "Infinity loop" itself shouts "Automation!". Guess the "Tech"nician failed to understand that. The article seems to have been written in 2014. I won't be surprised if that's what people understood by DevOps at that time. And if observability part is not taken care, it doesn't matter if the deployments are manual / automated, it is just a ticking time bomb.

[@garciajero]

100% , the only mention of Devops in this article was in the title , anyways we call it Platform Engineering now.

[@mrchubbles]

What's mind-boggling to me is how Knight was still able to be acquired at $1.4 billion despite this fiasco.

[@lashlarue7924]

Well, their configuration of assets in the value chain was such that someone was still willing to pay for those assets. Future value matters. Temporary insolvency can be remediated. Also, $400 million isn't that much money on Wall St.

[@ChrisOfSDUB]

The "term of art" is change control.

[@kiranmkota]

Change management? Version control? Never heard of change control

[@JesseGilbride]

"PowerPegAgen" might be the best one yet. 😂

[@dabbopabblo]

Modulo on an incremental user id is such a genius way to select a subset of deterministic experiment subjects. My grug brain would of just picked a random value and stored it for every hit of a common endpoint if the user hadn't been either selected or not selected prior.

[@NebucadLaVey]

CI/CD is one Part of DevOps. mainly stand DevOps for automatisation, thats why CI/CD often will be said. But it a little bit more. Like the goal to reduce human action to minimum or test to make sure you don't deoploy crap Program who developer fucked it up. kind like those things...

[@Waitwhat469]

Right DevOps is an org and process idea, CI/CD is how you enable that at scale.

[@talkohavy]

Hey ThePrimegen, I don't know if you read the comments... But we would definitely love a talk about how the implemented the kill switch, what it means, and how it would work in case of a real code-red situation. Love your content !

[@isaackoz]

"No written procedures that required such a review." I'm sorry, but having no procedures for somebody replacing code on servers is just asking for an Office Space 2.0. The amount of power those "technicians" had....

[@sylarfx]

this flash crash story is well known in the finance/trading circles, I think there was also a book written about it and similar cases of flash crashes

[@Tips-r-us]

In Dairy Farm in Asia, we own most of the 7Eleven stores, and when we release updates to the 7Eleven Servers and POS, we do the slow clap, 1 store, then 5 stores, then stores in a region, then global. That is our procedure. we don't want to stop all the stores taking money in a single moment, and if crowdstrike happened, would be a nightmare, luckily, all out servers and pos terminals are Linux.

[@kizhissery]

market making is done by HFT(high frequency trading) hence provide liquidy. These system trades billions of time in a day hence they buy low and sell high more like 1cent per trade hence they make money.

[@e_rebus]

There is always a kill-switch. It's called pulling the plug. If there's no plug, grab the scissors.

[@my-curiosity]

I'm a DevOps working for big company. This article is actually good example why you have to use CI/CD / DevOps practices (call it whatever you like, but in the end it's just engineers with specific mindset and responsibilities). Also, I worked for companies that had a low quality DevOps teams consistent of folks from India (in Bay Area, CA)...and that was a disaster. They were doing everything manually on Windows servers...So, even if you got a DevOps team it doesn't mean you have a right people with right skills. You want to follow all Automation, DevOps, CI/CD best practices, not just have a useless DevOps team

[@complexity5545]

Its rare for a financial brokerage system not to have a Halt (or kill switch). ....really rare (and not to have a cluster backup). P.S. Don't forget that no matter how much you plan, even a robot can be told to do deployment wrong. You need a kill switch and a backup cluster (for rolling back).

[@asdfxyz0]

This sounds like a lack of devops to me. Super poor observability around deployments, too many manual steps to deploy, insufficient monitoring of the system, sounds like this was inevitable tbh.

[@MiguelFelipeCalo]

We're just publishing new pages from a CMS and we have checkboxes all the way through making sure nothing broke. One would think a mission critical system like this would have a deployment session similar to a rocket launch.

[@gbb1983]

"TerraForm deez nutz" right off the bat lmao

[@doceddie]

This is low-key your best video yet. 😂

[@akam9919]

0:19 A wild arch user appears! Wild arch user used "I USE ARCH BTW!"... It was completely expected!

[@khatdubell]

"the only blame on the engineers is that they didnt push hard enough" no. Obviously i have no inside knowledge here, but lets assume the worst case scenario. In that case: They are to blame for not removing unused code. They are to blame for repurposing an old feature flag instead of using a new one. They are to blame for not building in a kill switch into either of the features.

[@freezingcicada6852]

Well, initially it seemed like just a lack of communication to avoid deploying the older version. Ultimately I do put the blame on the managers or w/e. Wtf where they doing when such an exaggerated amount of volume in the market was being pumped out. The FIRST 15 mins after opening bell in the market is the most important. They didnt see if it was just themselves pumping the market?

[@khatdubell]

​@@freezingcicada6852 Yes, that too. I forgot about that. where was the monitoring?

[@catcatcatcatcatcatcatcatcatca]

In my view, the price-tag alone can determine the blame. If the company goes bankrupt, the person at fault is by definition the CEO: the colossal failure was organisational risk management, as there seemingly was none at all in this case. While blame is never a zero-sum game, it is not very useful to blame developers or technicians for a disaster like this. While every level has lessons to learn, the lessons for leadership are much more important than the lessons for technicians and developers. Any harm, loss or damage disproportionate to the technicians or developers position and salary is a risk their supervisor failed to mitigate. All the way up to CEO.

[@naniyotaka]

@@catcatcatcatcatcatcatcatcatca Just like Chernobyl.

[@nekoill]

I like how every other day Prime seems to wake up and choose violence against me specifically

[@PCGamesAndTek]

This is a perfect example of why DevOps and following strict release procedures is crucial.

[@madumlao]

This article is kinda sus. The whole incident is being used to "sell" the concept of devops and continuous delivery, as if it was the "technicians" (classic sysops practitioners) fault that the company went bankrupt. "hey guys, if only you automated your delivery, you wouldnt have gone 400 million dollars into debt!" The reality is its obvious that the software culture of that organization was painfully unhealthy. No amount of devops practices would save it, because they'd just end up making reliable repeatable deployments that lost hundreds of millions of dollars. It is not sysops fault that - they used bad practice in repurposing an old flag - they refactor their code halfway - they have no ability to sanity check their deployment - they have no ability to verify the deployed version - they have no ability to turn off the said system The biggest red flag really is that this is a trading company with hundreds of millions in cash and apparently the people budget of their CORE SERVICE isnt big enough for a second guy. I wouldn't be surprised if these trader techbros assigned the intern to do it. Their whole job is to game the money market, and when their game is based on bad (company) mechanics its the "technicians" fault because they didnt "follow latest devops" or something. Yeah, no.

[@kairon156]

Like the kill switch. I bought a plunger for my toilet 3 years ago. Haven't used it yet but I'm ever so thankful that it's there as an emergency option.

[@avocadoarmadillo7031]

*Unloads 10 rounds from workplace carry into servers* "You're welcome"

[@alvinxyz7419]

i dont get it, there is nothing to do with devops in the article

[@jason-anthonygaskin7867]

I definitely could be slow on this one but I feel like they may have had a kill switch. There was a few hundred mil traded by 9:32 and it seems they ignored the emails. I feel like HFT is just so fast the kill switch would have to be automated based on a trigger they had probably built into the other 7 parts. Once you push $3B in notional value up 10% you're kind of screwed even if you get out immediately.

[@RenanTraba]

I add a second issue, dont repurpose flags, the old code wouldnt trigger , or even cause a crash

[@jglaab]

The devastation when it when from a one server pegging to a all hands on deck 8 server pegging

[@Olodus]

The real heroes of this story is the previous engineers that named the replaced software Power Peg, thereby setting up the perfect cherry to this masterpiece of ridiculous f-ups.

[@carltongannett]

NDC conferences has a good talk on this I believe

[@carltongannett]

kzbin.info/www/bejne/p3TCmqKAhtWpaqcsi=gVqxyI8naR8g-AOr

[@carltongannett]

Found it

[@robertluong3024]

can you link it? please 🙏

[@sonicfind]

@@robertluong3024 unfortunately no

[@avishjha4030]

@@robertluong3024 +1

[@Gilligan128]

The title ks very misleading: most of the way Knight handked things is counter to DevOps orinciples.

[@GaryFerrao]

0:43 associating Continuous Delivery with Dave Farley was the best joke i’ve heard so far. 😂 but be careful, you are becoming associated with regex licensing and some Rust things 😂

[@yaksher]

@1:30 Amusingly, market making is neither particularly risky nor illegal.

[@cmoullasnet]

If you’re not confident enough to delete vestigial deprecated code, it means you don’t know how your code works and you’re too cheap/lazy to do anything about it. It’s not surprising that blackbox code did something unexpected. If it’s your personal blog then fine. But high frequency trading systems subject to the wrath of angry rich Wall Street folk?

[@teejaded]

Manually deploying code to servers isn't devops. It's just ops. Write some shit to automate that and run tests on results then we can talk.

[@vikramkrishnan6414]

Manually deploying code in 2012 isn't even ops, it is pure mental illness. At this time, CFEngine, Puppet and Chef were all being widely used by "2 guys and a laptop" style startups

[@hg-sx5nk]

Agreed. Also. I find it really hard to believe they were using the term "DevOps" at the mentioned company - neither most of the finance IT - back in 2012.

[@Machtyn]

I became an SRE last year. Never heard of the position before. It didn't take me a month to hate it. It took me 9 months to finally get moved back to SDET.

[@BeamMonsterZeus]

That's one of the best opening salvoes I've seen aimed at the amalgam known as YT comments

[@arsvi123]

Imagine having your company destroyed by something called the 'power peg'

[@thewiirocks]

Oh good grief. You know how there's Development and Operations? DevOps is the practice of Development and Operations working together. Techniques like CI/CD are methods by which Development and Operations can work together. *Anything else* is FAKE. Especially if you have a separate "DevOps Team". This isn't even a secret. Most companies are doing some bullshit they "call" DevOps but has nothing to do with DevOps. Of course it's FAKE!

[@chris3079]

i traded that day, we were getting information from floor traders about what was happening, it was one of the most crazy day i ever traded. Started even before open. We had guys locked cause mad margin calls on the system. $50 stocks showing for dollars etc. It was just like constant selling or constant buying and watching it on level 2 was crazy, only other time seen that was on purpose when MS tried to support FB ipo, but it was supporting, this algo was selling or buying it. but in 75 stocks and any direction, just crazy. It was notable cause it was same mm and same amount usually, just blocks of stock for sale or buying, same amounts.

[@vikramkrishnan6414]

Question: how did this impact other traders? Did you guys have to trigger internal circuit breakers due to random price changes?

[@dami970]

​@@vikramkrishnan6414I'm curious too

[@chris3079]

@@vikramkrishnan6414 caused one guy to have a 2 or bigger million dollar profit, down to $400k about. A lot of system trade, trade outside normal trading bands stratagies, so they did VERY well. But lots of those trades were canceled, so they didnt net what they first thought. We had human compliance, that was more confused, why is every account showing a huge margin negative balance lol but usually margin gets sold by end of day, so for retail, it nothing, as it was over quickly. But it was crazy. I had good intel, so were able to profit. Intel as we knew it wasn't geo political etc, we knew who and whyish. Intel was real time, not something in advance. Took us maybe 5 minutes or less to know what was happening, which was very quick.

[@steffenbendel6031]

And this was slow enough that you could watch it. Most flash crashes happen in seconds.

[@chris3079]

@@vikramkrishnan6414the trader who use to have a strategies of out of market orders killed it. But a lot of trades were just canceled.

[@earthling_parth]

This was one of the funniest thing I have come across in 2023. Hilarious and scary.

[@grimsas]

That was probably the best bed time story I've ever heard.

[@patrickhighspeed]

That is the greatest Story of all time!!!!

[@RajinderYadav]

I remember when this happened, a brand new High Frequency Trading shop that blew up on day 1.

[@baburik]

0:00 DevOps doesn't exist 6:05 this is why you don't manually deploy at that level if you have hundreds of millions of dollars of assets figure out how to **automate** it pay somebody pay two people

[@Huntertje13]

This seems related to a lack of controls like alerts / logging. Automated deployment will fix copy paste errors but not necessarily mitigate. Code will go wrong, it matters how quickly you can fix it.

[@WalterMiller]

Whenever you find yourself saying "What could possibly go wrong?" instead say "Let's just do it and be legends." You'll feel better about the results.

[@patrickrobertshaw7020]

This development process was an obvious ticking time bomb. It went off in 2012, but if it didn't then, it would have sometime between then and now

[@datmesay]

cash equivalent is something that is very liquid, high demand and hence you can sell instantly for cash, usually it is short term assets.

[@stevezelaznik5872]

I'm seeing a common theme in the comments. "Why couldn't they just walk in and kill the servers?". If you're a high frequency trader, you can't afford to host your own servers. All the high frequency traders lease space in a shared computer room inside the NYSE. This is because high frequency traders want to get their trading information as instantaneously as possible. If somebody else hosts their servers in an office a mile away, it's the equivalent of getting the newspaper a day late. They're trading on old information. That being said, there should have been some remotely activated shutdown command.

[@ninjaasmoke]

this is why you shadow test THIS IS WHY YOU SHADOW TEST

[@avi7278]

"pay somebody to automate it!" -- you mean like a devops engineer? 😂😂😂😂

[@packrat-y7j]

This isn't a failure of the DevOps philosophy. This is just sheer stupidity.

[@TueChristensenDK]

Come on a couple of things here: 1. Deployment day and no one cares to check that everything seems ok (I mean its High Frequency Trading - not your personal myspace page) 2. They took down the servers that were deployed correctly, but not the faulty one - how? 3. Delete old code 4. They re-used the flag that was used in the power peg code - why and couldn't you find a new better name?! 5. You didn't check that a counter for the parent order existed? That seems like a reasonable edge case unittest

[@debasishraychawdhuri]

They changed the code in all the servers, nobody said like: “why don't we just stop every server and diagnose the issue in offline mode?”