“Who TF have multi cloud back up?” “The guy that saved this company”
@yuhyi01225 ай бұрын
Trueee
@ephermalinsights5 ай бұрын
yeah i am amazed
@ceoatcrystalsoft49425 ай бұрын
Opportunity costs. What's cheaper: having a zero-fail backup, or losing everything?
@evian89765 ай бұрын
@tripplefives1402 so your own server can be destroyed by natural disaster if we go this way "if you don't back it up yourself then it won't be backed up." the story of this video literally refutes your point
@oneoranota5 ай бұрын
Yeah, one of the main drive of the Internet is seamless replication of the information, would be a shame to store some critical data in one single spot.
@ImTom43105 ай бұрын
Was so shocked that the company followed the 3-2-1 rule.
@MidnightMidas5 ай бұрын
well its better to pay more for backups than pay for the law suits from 600k people...
@mktomchan5 ай бұрын
@@MidnightMidas many managers are just so short-sighted, if their company fxxked up, they just move to another company. So they dont want to spend extra for long run
@yuhyi01225 ай бұрын
That was a good decision
@guacaprole5 ай бұрын
Whoever architected that is earning their salary.
@Pravculear5 ай бұрын
@@guacaprole and also deserves a raise for that one.
@aenguswright73365 ай бұрын
Nice to know that a company was actually saved from this situation because they had actually followed best practices about IT... sadly rare
@promero145 ай бұрын
If I had to guess it's probably mandatory by law since they are handling retirement funds. Not sure though
@ChineseKiwi5 ай бұрын
@@promero14and you would be correct. Lots of compliance.
@hoangnguyenvuhuy55355 ай бұрын
@@promero14 Yep. IT finance here. Required by law to keep back up, back up of back up, and a whole ass DRC server
@kaseyboles305 ай бұрын
Never Ever forget that "the cloud" is just a fancy name for someone else's computer and you're their mercy.
@YS_Production5 ай бұрын
Somehow for companies nowadays having anything is seen as the biggest sin (having own engineers, own infrastructure, own cleaning ladies and doormen, own furniture, own cars...).
@TelemarketerOO75 ай бұрын
@@YS_Production yeah man, economies of scale have advantages
@stevebabiak69975 ай бұрын
Unless your name is Kamala, in which case you believe it’s something that is floating above you ;)
@YS_Production5 ай бұрын
@@TelemarketerOO7 lmao, yes that is the pretence under which outsourcing gets into companies, but reality is the employees that stay make up for the shortcomings of the external workers or services. And speaking of cloud services, those are almost always more expensive than running on-site servers, for example in a manufacturing plant that does not need outside access to its IT services. Yet cloud gets forced into such usecases. Ridiculous.
@YS_Production5 ай бұрын
@@TelemarketerOO7 one more on how people think these days, I'll just throw approx. numbers as this story 2 yrs old: A user needed Adobe Acrobat license. I said I will buy it for them for 120 USD and their boss needs to approve the purchase. The boss said "Why the heck are you buying something for 120 USD if you can get this for 10 USD a month?" And sent me a link to Adobe Acrobat subsription. I laid out to him that in 12 months, the subscription will have cost more than the licence. This person I had to teach the basic maths and economy was more experienced, educated and much better paid than me. Yet he was blind to this! All he saw was "10
@demonslime5 ай бұрын
whoever at unisuper pushed for double backups need an 8 figure check rn
@pureabsolute46185 ай бұрын
Yep - definitively should be a percentage of the funds saved :)
@caliden4 ай бұрын
@@pureabsolute4618 bean counters definitely won't be asking why those other backups are needed
@BatkoNashBandera7744 ай бұрын
make them king/queen of Australia while you're at it, I guarantee they could not possibly do a worse job at governing than the "elected" aussie govt. shoutout to TheJuiceMedia
@edwardmacnab3543 ай бұрын
whoever ?wow , it's really that bad out there that you gotta rely on a maverick to do something sensible ?
@sgtjonzo3 ай бұрын
@@BatkoNashBandera774 what don't you like about the current government, what policies do you not agree with
@xFlRSTx5 ай бұрын
at least they had sensible defaults like deleting everything with no warning after 1 year
@askjacob5 ай бұрын
It probably messaged some dev that left 2 years ago
@Antleredangelbun5 ай бұрын
@@askjacob haha just imagining someone who got fired a few months ago, on the toilet, getting a notification "It's been 1 year since [Google] backup was created, delete it now to free up space?" and just going "hehe 🫣😗😘😜 :3" and pressing "Yes" 😭🤣
@WilcoVerhoef5 ай бұрын
Well if by internal tool they mean "a local dev clone of the repository" (just used to develop and test)... Who cares. Then they just should have never used it for production in the first case
@arduinoguru72335 ай бұрын
@@WilcoVerhoef That was unprofessional from their side, it not single engineer to handle that task, sure others were with him when hes issued that command, and they didn't notice it, and how the F, google use experimental tool in a working environment, I'm single web developer and I test my web apps for weeks before send it to the client
@blackrifle67365 ай бұрын
@@arduinoguru7233 *Because you have a strong ethical sense. OTOH, Google...*
@mista_jamaican5 ай бұрын
Typically we Australians don't exclaim "Crikey" during a major tech outage. We use more traditional technical language including words like "shit" and "fuck".
@Elitrian4 ай бұрын
Next time I have a major outage at work, I need to remember to yell out "Crikey" now - after I've included all the swears!
@Aneliuse2 ай бұрын
Shicrifuck
@maisjaki2 ай бұрын
Good to hear their grounded approach, despite not having any boots on the ground 😉
@dustinbragg19212 ай бұрын
How bad does it have to get before Australians have to resort to the technical expression of punching through their monitor?
@derpdiggler74132 ай бұрын
why not a "crikey-shit-fuck" for good measure?
@glitchy_weasel5 ай бұрын
The outro slaps harder than the Crowdstrike outage
@CentreMetre5 ай бұрын
Cant wait for the kevin fang video on that
@yuhyi01225 ай бұрын
yeah it slaps damn
@wallyrogers23715 ай бұрын
Too soon
@mr_voron5 ай бұрын
Pure fire 🔥
@NobodyisAnybody5 ай бұрын
@@wallyrogers2371 ratio
@CrimeBeanus5 ай бұрын
why reverse a linked list when you can reverse a lifetimes work
@akshaypendyala5 ай бұрын
😂
@justinliu77885 ай бұрын
in rust! (rust lifetimes reference)
@linuxguy11995 ай бұрын
underrated comment
@BatkoNashBandera7744 ай бұрын
@@justinliu7788 Camomo says hi and also that you're still not welcome back to that server.
@Auchura4 ай бұрын
@@BatkoNashBandera774 nobody cares shut up
@lieutenantbites57393 ай бұрын
I love the redditor commenting "I'm a cloud engineer, it's 100% impossible to be google's fault" and was entirely incorrect. Classic reddit behavior
@Monotof15 ай бұрын
To be fair i can easily see how this happens. Its unfortunately very common for engineers to build some tooling for testing or debugging in narrow engineering use cases. Then sales or another department comes along with a problem they need fixed WiTh tHe HigHeSt pRioRiTy and "borrow" the tool "just for this once". A year later the entire department ist misusing the tool or even some bastardized version of it for production use on actual customer data. No one fully understands what the tool does or what it was even meant to do, but it saves some time so everyone just uses it without a second thought.
@emmah14085 ай бұрын
Sadly true
@Daweim05 ай бұрын
Used to call this "Move Fast (into walls)"
@vinny1425 ай бұрын
" Its unfortunately very common for engineers to build some tooling for testing or debugging in narrow engineering use cases" True, sadly. I've seen a company trying to make their backups faster by backup less data. More importantly: to only backup production databases. Fair enough, but they used a naming convention to determine that it was a production database. The problm is that they did not communicate this new requirement to the rest of the company and just assumed they already knew. So a few months go by where a couple of "misnamed" databases are not backed up, old backups expire and are erased, and then the server starts running out of diskspace. Instead of giving the server more space they decide that "hey, we know which databases are production, so we can simply drop everything else". And they nuked whatever did not follow the new, unknown naming convention. Four customers lost their entire database, forever. Then basically the entire company was assigned to help with manual data entry to help get those customers at least to some state of operation again. And all because they didn't want to buy a $50 harddisk.
@ns32425 ай бұрын
common in saas
@kelly41875 ай бұрын
Nothing is ever being used only “temporarily”
@nickkyg5 ай бұрын
Outro had no business being as fire as it was😂😂
@khaelkugler5 ай бұрын
It was hella fire, but I think the singing was from Suno ai
@Blox1175 ай бұрын
@@khaelkugler dang how did you spot that
@khaelkugler5 ай бұрын
@@Blox117 I've used it a bunch. It's really sick, but once you've heard some you can kinda recognize the scratchier sounding voice and chord sequences!
@fixedd_d5 ай бұрын
Thank you i was going to close the video before the outro ❤
@A.I.rchist5 ай бұрын
I also agree with khaelkugler, however I'd sat the line at 8:59 is what gave it away for me, if you use suno a bit you'll notice a lot of lyrics come out like this. I'm up to being proven wrong, but I'm pretty sure it's AI generated using Suno.
@somekindofhmm2 ай бұрын
I thought this was a good video until 8:18 and then it became a great video. Then the chorus kicked in and it became the best video of all time
@NomenNescio995 ай бұрын
I am actually impressed that they managed to get back up and running in two weeks time after having their entire environment deleted, that's actually a great job by the unisuper it guys, hats off, kudos and respect for a job well done.
@darrennew82115 ай бұрын
Having worked at Google, I'm a bit surprised that the DB didn't get moved to (essentially) the recycle bin for a couple weeks before the space was recovered. The group I was in dropped production DBs a couple of times and had them restored within minutes.
@WhoWatchesVideos5 ай бұрын
@@darrennew8211Yeah, it lends more credence to this being an internal development tool's failsafe cleanup routine.
@oliversmith21295 ай бұрын
@@darrennew8211 No delete protection probably.
@WoodlandTrotter4 ай бұрын
This guy is meatriding for real
@WoodlandTrotter4 ай бұрын
This guy is meatriding for real
@dannyvoid5 ай бұрын
We all know how this would have been resolved had the deletion been to something worth substantially less than $135B
@everyhandletaken5 ай бұрын
$10 gift card? 😅
@NotEvents5 ай бұрын
@@everyhandletaken 🤣🤣
@midgetwars15 ай бұрын
I like to think that some of the google employees that provisioned this were in Australia and they were with unisuper so they were extra motivated
@Alexrocksdude_5 ай бұрын
@@everyhandletaken Nope, not even a response to the ticket. Google doesn't talk to peasants (people with a net worth less than 1B USD).
@mina865 ай бұрын
In that case Google wouldn't setup unique infrastructure just for you a year ago and nothing would have happened.
@OtterSeaborne89015 ай бұрын
The thing about Australians is that black swans are just regular swans here. Jokes aside, I'm not even surprised that UniSuper had that many redundant backups. Universities take their pension money seriously.
@ChineseKiwi5 ай бұрын
It was originally for university staff (thus the name and thus why a subsection of funds are called ‘industry funds’), but these days, it is open to anyone to join.
@brydenquirk11765 ай бұрын
I was genuinely surprised when I learnt that white swans exist..
@Hebdomad75 ай бұрын
@@brydenquirk1176 pffftt... White swans... That's gotta be like a one a billion mutation.
@JACKHARRINGTON5 ай бұрын
Yea it must be albino or something
@ADR695 ай бұрын
That outro track is a 10
@Real_Halo_MC5 ай бұрын
Return of the king Can't wait for the future cloudstrike video
@alex_zetsu5 ай бұрын
When I saw a new video came out for Kevin, I was surprised Cloudstrike Falcon was not the next video. Perhaps he wants to wait for more information to come out so he gets the whole picture. To me I think it is pretty clear, the Falcoln driver had code which essentially said "execute stuff here" and the patch had a bad file full of zeros. if this had been tested by anyone at the company, their computer would have failed to boot, then they would have looked at it, realized their driver didn't validate what it was trying to run, and then have a laugh before fixing its validation. The fact that it accepted a null file suggests to me there is no validation at all and bad actors can put in whatever they want there and the driver will blindly execute it, so the problem isn't just the bad file bricking computer but that the driver is fundamentally not robust. So I think I get it. But maybe Kevin is waiting just in case there is more to the story. Who knows maybe there were major mistakes that led up to the big one. So Kevin is waiting to be sure he doesn't miss anything even if it means other people will cover it first.
@reapimuhs5 ай бұрын
@@alex_zetsu it's also possible that it *was* tested before being pushed to production and there was no issues, meaning something went wrong during whatever process actually distributes those patches and that's why it wasn't caught. my question would be why they weren't distributing new patches in waves instead of to all customers at once.
@alex_zetsu5 ай бұрын
@@reapimuhs I don't think it was tested. We know the driver itself isn't being updated in that patch. And unlike some bugs which only display aberrant behavior on certain hardware, their driver doesn't work if it tries to execute a null file on any machine. Maybe it will later be revealed that the test environment in fact can run a null file, but I doubt that.
@reapimuhs5 ай бұрын
@@alex_zetsu what I was saying is that it's very possible that the file was *perfectly fine* and not null during testing, but that something went wrong during their process of *distributing* that file to production devices that caused it to become nulled. In which case, it would have looked perfectly fine during testing and the issue only would have appeared during production.
@TrakoZG5 ай бұрын
more explosions than a Michael Bay trilogy
@fang-penlin44825 ай бұрын
Can't wait for your CrowdStrike episode, it's gonna to be awesome 🤣
@LiuWoods3 ай бұрын
hahahaha
@uhhhwhatdoipick5 ай бұрын
that outro is fire kevin's kicking off his music career :O
@75hilmar4 ай бұрын
"I have a backup." "THAT WAS OUR BACKUP!!!" "Yes but I have another backup. Can't trust humans."
@lateralus65123 ай бұрын
'Can't trust humans' basically sums up my experience so far in this world.
@nadtz5 ай бұрын
I remember when this happened and finding out they actually had good backups made me feel vindicated for all the clients I've had over the years who I've beat up over backups. Only problem (which you wouldn't expect in a cloud setup) is they probably never tested them which is probably why it took 2 weeks to get everything restored. But at least they didn't lose their data. Also that outro =D
@advanceringnewholder5 ай бұрын
Better than entire Indonesian government that have all their data lost because of ransomware attack. and it looks like no backup
@ChineseKiwi5 ай бұрын
DR tests are done often in Super IT. It depends how much data needs to be restored and how data feeds into one another. And of course observing that everything works as intended
@pureabsolute46185 ай бұрын
Keeping in mind just evaluating the loss of resources can take days, and that the backups had to come from a totally different system...
@Harsh-df9vp5 ай бұрын
Bro's a full time engineer, part time rapper.
@Daniel15au5 ай бұрын
You're right that super is similar to a 401k - the main difference is that it's mandatory for all employers to contribute. Similar to the USA, the money is either invested in a mutual fund of some sort, or it's "self managed" where the employee chooses where to invest the money through a brokerage account.
@francistaylor18225 ай бұрын
Exactly. Its huge amounts of money and critical for peoples retirement as we have had this scheme for decades - $3.5 trillion (AUD) is under management. To give you an idea of how big this is, its double the countries GDP.
@ahmoin5 ай бұрын
the rap at the end was hilarious 😂
@supercyclone83425 ай бұрын
Very funny, but was that AI? Towards the end it sounded slightly fried lol. Kinda scary that I'm even questioning this 😅
@yuhyi01225 ай бұрын
@@supercyclone8342 Either that or autotune
@Xamy-5 ай бұрын
@@supercyclone8342why does it matter. It’s a gag/skit
@supercyclone83425 ай бұрын
@@Xamy- I'm not really bothered especially cause this is the perfect use case. I was just curious if I finally caught an AI song in the wild lol.
@mthia5 ай бұрын
@@supercyclone8342 i believe it's suno ai
@Sawta5 ай бұрын
Can you even imagine how loud the "I _TOLD_ YOU SO!"'s must have been when the root cause was discovered? I've been in Google's side of the issue for stuff like this (on a much lesser, but critical scale) it's never fun. I can only hope that whoever ran the command a year prior was already working for some other company by the time this happened, and got a fat consultancy payment for taking the time to review with them.
@bosshog88445 ай бұрын
The blame game is childish. Whoever ran the command was under pressure and had 1000 other things going on. The fault lies with the design of the system, which a ton of people contribute to. Running a command missing a parameter should not result in the cloud being deleted.
@chunkygiblets46783 ай бұрын
@@bosshog8844 It was probably run by someone far detached from the smelly dev basement it was ill-conceived in.
@YoshiAsk2 ай бұрын
@@bosshog8844I agree that blaming a single person isn't accurate or helpful for anyone, but Google is squarely at fault here
@BoloH.5 ай бұрын
"As a cloud engineer myself, I can confidently say this is complete bullshit, it is 100% UniSuper's fault"
@Zxv9755 ай бұрын
😂 That was my favourite part
@Zxv9755 ай бұрын
@@DespOIcito with all due respect, if you don't have the full context then maybe you shouldn't make statements like "with 100% confidence" 😂
@generalardi5 ай бұрын
I guess one could argue that this particular issue couldn't have occurred if they hadn't moved to Google, but it's not really a good-faith argument, I think.
@daedalus50705 ай бұрын
"My Dad works at Nintendo"
@syte_y5 ай бұрын
@@Zxv975 agree speculation when you have no idea is asinine. The person was so confident too.
@Chickenbreadlp5 ай бұрын
GitLab almost having 0 functioning backups will never get old xD
@theprantadutta5 ай бұрын
Man, that outro caught me off guard, it didn't have any right to be as good as it was.
@rcytray5 ай бұрын
bro cooked with that rap, goddamn
@Alphabeter5 ай бұрын
Hm
@benamende58975 ай бұрын
🤔
@NickHamilton885 ай бұрын
Fr
@justinfalc82135 ай бұрын
holy shit when the autotune came in i shat my pants, bars
@ign1fy5 ай бұрын
This autotune drop is now my alarm tone for UniSuper project meetings. The office is gonna be lit.
@tvwatch96694 ай бұрын
Sounds like Suno AI but might be wrong there
@Silly-s8n3 ай бұрын
6:00 is really representative of the average software "expert".
@sleet090th5 ай бұрын
Solid google moment
@Ps5prolite5 ай бұрын
Well, Google is American. Never trust these pigs. Better off using China based cloud providers
@Cmanorange5 ай бұрын
judging by these videos, what they really need is missile and explosion protection
@N3G4T35 ай бұрын
It's refreshing seeing a company having backups on separate systems. Good management of Unisuper's tech department.
@MHX115 ай бұрын
didn't expect you to spit bars at the end 🔥
@jimkats15 ай бұрын
*Everything on cloud*, they said. *It will be much better for everyone*, they said.
@michaelwills19265 ай бұрын
Yep “off prem” is just someone else’s prem.
@aa-lp1ho4 ай бұрын
Amazing that you actually did a bad Aussie accent instead of a British one. Big props.
@cask2cask5 ай бұрын
outro is legit
@brian.westersauceАй бұрын
9:00 goes hard
@Georgggg5 ай бұрын
"When one of disaster strikes" typical insurance scamvider policy don't cover your case on page 25/60 fine print
@samueljett78075 ай бұрын
At the end of the day, all the "cloud" is is just some other random guy's computer. Count on no guarantees of availability.
@TrakoZG5 ай бұрын
this is what I've said for ages as well. I've always wondered if there could be a coordinated attack from malware that could break out of containerization or virtualization to hit everything on the metal.
@NekoiscoolII5 ай бұрын
Google:"We know everything." Also Google:"Fack i lost money by accident."
@djdeetsdroppingthosefunkyb12365 ай бұрын
I just stumbled onto this video in my recommends and now I'm left with the fire outro stuck in my head. Subbed.
@azraelgm5 ай бұрын
This channel makes me feel better about the mistakes I've personally made and those I've seen at my workplaces.
@Edgodful5 ай бұрын
This channel also makes me fell better on my boring ass it support pos
@fusrosandvich37385 ай бұрын
@@Edgodful It might seem boring for you but definitely remember that it's the people like you that keep everything from crashing and burning. People never think of how important the maintenence crew is until the building's on fire.
@Flupe24 ай бұрын
Came from a review(?) of your video on another channel to leave this comment on the outro. It's wonderful.
@ttominable5 ай бұрын
“Kyle, why are we paying for multiple backups?”
@anilparmar1955 ай бұрын
Never have I ever waited so eagerly for KZbin videos
@midgetwars15 ай бұрын
I find it amazing that this was 100% google's fault. If for example Unisuper set it up incorrectly, blame still can be attributed to Google for having weird deletion priorities and unable to restore deleted cloud environments. But seeing as this was an internal test, there was no "recycling bin" and they even pressed the button to deploy.
@w3w3w34 ай бұрын
DEI
@vosechu5 ай бұрын
Yeah, this isn’t a backup thing, it’s disaster recovery. We do yearly gamedays where we try to restore our entire company to operations in another cloud in a week (which usually turns to two). Super impressive that they got it up and running in 2 weeks!
@spacegamedevsoftware5 ай бұрын
That rap goes hard
@Electrink5 ай бұрын
crowdstrike video when?
@Hopgop15 ай бұрын
Be a long time before all the details come out about that one
@misham65475 ай бұрын
@@Hopgop1considering the lawsuits its going to be a long time
@nojomyth5 ай бұрын
@@Hopgop1 The official explanation was released today with the exact reasons as to how this happened :D
@caduhidalgo49965 ай бұрын
Soon™️
@TrakoZG5 ай бұрын
would need a huge render farm for all the explosions, could rival a Michael Bay trilogy
@Brunn01214 ай бұрын
i wonder if this incident lowered the cloud service price for UniSuper...
@therealdoc5 ай бұрын
The difference between making a backup and insurance is that making a backup is fast, cheap, and easy. However, insurance is designed to milk every penny out of you, provide generational debt the moment disaster strikes, and do nothing to actually help you in your time of need. Insurance is literally a scam and companies know it. That's why they send ludicrously sized fees for medical to insurance providers so they can make a bit extra on each service.
@lateralus65123 ай бұрын
I figered this out quite a few years ago and never used any type of personal insurance again. They money I save over time just goes into an index fund which keeps growing in size.
@rovhalt66503 ай бұрын
The fact that all of our money is just relying on the internet not crashing for whatever reason is pretty horrific.
@electric74872 ай бұрын
The Internet was supposed to be a Web where the failure of one thing wouldn't cause catastrophe, but now, ironically, it relies very heavily on a mere handful of major companies just to not implode.
@danieln.2590Ай бұрын
the dude that did the due diligence of backing things up deserves a raise
@braddeicide5 ай бұрын
Very good having backups with another provider. From my observations that is very rare.
@TrakoZG5 ай бұрын
I briefly worked for a company that had the off site backup. I'm not sure if they followed the 321 rule but they at least ran 21.
@stevecracknell27014 ай бұрын
Bro that back up free style at the end was tha bomb!
@m_the_pig5 ай бұрын
Great video and very informative. Cheers from the UK! 🎉
@anonydun82fgoog354 ай бұрын
The funny part will be when Google points them to the End User License Agreement they clicked on that says Google is not liable for any damages from this kind of thing. But people are still in love with "the cloud" for some reason.
@martenkahr33653 ай бұрын
You'd be surprised how much better the EULAs that companies big enough to have their own legal departments can negotiate compared to the boilerplate take-it-or-leave-it agreements "regular" businesses and individual consumers have to agree to.
@everyhandletaken5 ай бұрын
I'm glad I wasn't the only one here that loved the outro on this one. Great job on both the outro & the content (as always). My wife actually had super with UniSuper up until about 5 months ago, small world.
@avegamers5 ай бұрын
Lovely Video. Time to show this to my company here and restructure some backups
@Jussoparkours5 ай бұрын
As an Aussie I can confirm this is how we talk when dealing with a sev 1
@umeruddin34575 ай бұрын
Amazing video mate and loved the outro😂💖
@Messiah_Abel5 ай бұрын
he needs to release the full version of Backups on spotify fr!! 🔥🔥🔥
@IOSALive4 ай бұрын
Kevin Fang, awesome video keep up the good content
@bamgm145 ай бұрын
Man the Outro Bro 11/10
@matheuscabral96185 ай бұрын
this video was absolutely hilarious, I did not expect to laughter on video about this, the outro was also fire, great job
@FindSimpleGuides5 ай бұрын
Even for tech giants such as Google, there is potential impact when unnoticed small details snowball into big issues.
@TrakoZG5 ай бұрын
when you run prod with test code, excitement is guaranteed
@pauldirc..4 ай бұрын
@@TrakoZG Means
@catcoder124 ай бұрын
I'm surprised that a pension fund followed better data storage practices than some IT companies.
@knightsljx5 ай бұрын
I'm stealing that closing song for awareness training on backups
@josephgonzalez93424 ай бұрын
Those are some sick beats you dropping sir. I'm dancing in my chair here
@undergrounder5 ай бұрын
Already liked and subbed. When Crowdstrike? :D
@4citi4 ай бұрын
This backup song goes hard. Gonna play the chorus before every change.
@brad.fuller5 ай бұрын
not a "backups" problem here so much as "disaster recovery". A salutary lesson......
@thechrisdo5 ай бұрын
Nice video Kevin, you really put a lot of work into this and it shows. I also feel like it was explained in an understandable way, so keep 'em coming!
@henrikstorch42755 ай бұрын
Lyrics: May 2020, GCVE took flight Seamless VMware migration to the cloud was in sight They had dashboards, metrics, operational tools Got it all figured out, Google devs ain't fools Then billion-dollar UniSuper comes around Needs a VMware cloud but their specs weren't found So then went and asked Google ops to spin that thing up Except there was a slight catch: It would soon blow up Engineers scrambled, their logs blank as the dawn Their vital infrastructure was just suddenly gone It was truly an unprecedented black swan event There was nothing UniSuper could do to prevent Backups are our lifeline, in backups we trust When prod is deleted, backups are a must Three copies, two media, and one off-site No backups, I'm leaving 'til you make it right
@realjoshuaW2 ай бұрын
first video ive watched from this channel but that random song at the end caught me so off guard i had to subscribe😂👍
@themirrazz5 ай бұрын
3:53 First it’s PowerShell for Linux, now Notepad for Linux? Next we’re gonna end up getting Windows Explorer for Linux 💀
@YeloPartyHat3 ай бұрын
The song at the end 🔥 🔥 🔥 Such high quality content!!!
@doyouwantsli96804 ай бұрын
I need a longer version of this song
@tzisoreyАй бұрын
An old employer of mine had a poster up - "Backups are like Parachutes - you only know their value when you need to use them"
@Electracion5 ай бұрын
Good thing they weren't on AWS, Amazon would have just denied it was their fault for as long as humany possible, did the bare minimum to fix it once they couldn't deny it any longer, and then forced Unisuper to sign an NDA in order to learn what went wrong.
@P-39_Airacobra4 ай бұрын
The backup song goes SO hard
@chawrx35 ай бұрын
a kevin fang upload day is a great day
@j1d7s5 ай бұрын
I am so glad they had backups! Having a destructive option as the default was a major f-up. I wonder if the people involved in writing the tool were still around when it was released, hard to imagine how that issue could have gone unnoticed during review or at release time.
@kangalio4 ай бұрын
0:26 is it just me or does this sound like AI text to speech?
@marcothepucci4 ай бұрын
It is just you
@electric74874 ай бұрын
It sounds like text to speech at first but as the video goes on it sounds more and more like an actual person.
@leoldqv4 ай бұрын
it does completely
@grey.78283 ай бұрын
It definitely does
@Cbp8463 ай бұрын
No, just sounds like FireShip
@orbatos4 ай бұрын
Please release a radio or full length cut of that song, it's great. Nice coverage of this incident all around.
@Sumilum3 ай бұрын
the guy bashing UniSuper just be wrong the whole time was funny as hell
@3then4 ай бұрын
the song at the end was lit! new subscriber xD
@pav4315 ай бұрын
Backups: On Fire, non-existent Gitlab: Meh. Whatever These videos are pure gold, omg 😂
@cs512trАй бұрын
big props to Unisuper for taking backup seriously
@Saru-Dono5 ай бұрын
Eminem has been really quiet since this banger has dropped
@videofandude994 ай бұрын
I.T.-HipHop= new genre because of this channel
@weiSane5 ай бұрын
That Outro track kinda went hard 😂
@strigido-yt4 ай бұрын
We need that outro as a full song on Spotify.
@SkandiaAUS5 ай бұрын
I'm a UniSuper customer. Although it wasn't their fault they still did a shit job communicating this out. And when it was resolved they sent an email squarely blaming Google and even had them put out a statement.
@Allie-w1l4 ай бұрын
The notion of doing anything financially important online is just plain stupid. I computerized my spouse's business, years ago, and our company computers ran Quickbooks OFFLINE. Always offline. The program was installed from a disc. NO ONE had any way to get into our computers. Invoices were printed out and mailed to customers. Assuming computers have to be on the internet while in use is like assuming you can only drive a car on expressways and never back roads.