The 2020 Google Outage (Detailed Analysis)

[@vikranttyagiRN]

I recently found your channel. Its pure quality stuff here. Thank You

[@hnasr]

Thank you Vikrant

[@mrinalsingh8943]

At 20:14 , maybe every time we login, the Paxos leader logs that info (location, time, date, etc of user login) in the database and that may have exceeded the quota.

[@hnasr]

Good thinking ! That might explain it yes 👍

[@marslogics]

Great content 👍 Thank you for zooming in this time, it was easier to read.

[@hnasr]

Thanks for the tip 👍

[@m0llux]

For the Kubernetes topic: you can use GCP accounts as service accounts for RBAC in Kubernetes Clusters in GCP.

[@devyetii]

The awaited ❤️ Thanks Hussein, watched the summary and listed the video in Watch Later cuz college projects is taking all my time 😂. Keep up the great work 👏🏻

[@avi12]

What's funny is that I started experiencing this outage a moment after starting to watch a video on this channel

[@ddonddon2430]

hi Hussain, Is my below assumption right, i) QuotaManagementSystem(QMS) service running in db nodes is the faulty one. ii) It sends disk usage to QMS controlplane v1 and asks for usage from QMS v2 is this right?

[@valirjonathan2161]

Isn’t there any minimum quota at all? Why would allow the system reduce the quota all the way to zero?

[@t395delta]

nice tldr at the begining thanks

[@VIGNESHPRAJAPATI]

Thanks Hussein for well explained summary.👍

[@KartikMakker]

Seems the leader server logs the user login info that may have caused all the available quota to be used over a period of time

[@indiansoftwareengineer4899]

I thought google is GOOGLE and it won't make mistakes but at the end we are humans and we tend to make mistakes.... Loved your all videos, definitely it's helping a lot for new developers like me to gather a lots and lots of knowledge....

[@user-r1g5i]

Please, make a playlist for post-mortems 🙏🏻

[@hnasr]

great idea!

[@hnasr]

kzbin.info/aero/PLQnljOFTspQXdkZLiYCCh_5RBP1-T-Rnx

[@user-r1g5i]

Thank you! 🎉🙇🏻‍♂️ your reading-outs are hilarious, I wish you had access to internal post-mortems at least of google, they have dozens each year

[@soul67]

I clik on like just because you said 47 in french, really good job!

[@ahsannasir6214]

You should speak more arabic :D It sounded amazing. Thanks for great insight as always though! :)

[@ganeshannt74]

Informative 🙏

[@zainabalhaidary]

How come you were so hard on Amazon but very lenient with Google? :P I think both have done an amazing job to improve the tech industry over the years.

[@hnasr]

I try to put myself in place of engineers working at critical infrastructure, its a tough job really. My tune changed in recent videos becoming more understanding when things happen

[@robertstewart9325]

Google does not use Paxos, even if their PR says they do. Sure they have some nodes talking Paxos in dev, but never in prod.

[@HarshKapadia]

How do you know? Just asking.

[@mishasawangwan6652]

wrong. the mayans predicted this in 2000BCE. facts.

[@Belliger1991]

So the fact google went half a day after publication of the SolarWinds breach was just a coincidence?... With a ssl certificate error? (I have a screenshot xD) ^^ its to easy an explenation in my oppinion. But then again one misshap is seldom alone ^^

[@nsambataufeeq1748]

Google was probably not affected but reacted anyway

[@classical-bit]

20 minutes 😆