It’s always Microservices - The Spotify Outage Explained (March 8 2022)

[@hnasr]

Get my Fundamentals of Networking for Effective Backends udemy course Head to network.husseinnasser.com (link redirects to udemy with coupon)

[@grmancool]

now every time a service I use is down I think "man the Hussain video is gonna be fire!"

[@Stoney_Eagle]

I have managed to dodge all the (outtages) bullets so far but this one I couldn't dodge 😂😂 I was working with Spotify's api and communicating on discord.

[@Chubbywubbysandwich]

Lmao shots fired from both the directions

[@HK-sw3vi]

google cloud said "fuck you" in particular.

[@fanciestbanana4653]

Lol sounds like a target attack. Cripple the infrastructure and destroy communications to delay the response.

[@benjaminsaine9]

This is quality! Keep it up. Thank you

[@moylababa8196]

I enrolled your Fundamentals of Database Engineering course and it is one of the best course on this topic

[@dmitrychernivetsky5876]

They didn't have a plan. What had happened here is they happened to be in the process of migrating to traffic director. When that croaked they simply redeployed the previous version of software that fell back to DNS discovery (aka config file with a bunch of dns hosts for each microservice) Been there done that, hello istio.

[@x4FLJZQw8Zrv]

2:20 To put it into perspective, whenever I have an interview and they tell me about cloud services (which I have no experience of, I am still trying to get into the industry), whenever I mention that there should be a workaround on interviews or to some professors, they are either confused or tell me "there's no way to do that" or "it can't be afforded". Only once I heard there was a workaround set up I wish I knew more on the topic to at least know the details on how would you set up a workaround, the obstacles, and if there are scenarios or configurations where it would not be possible or it would be too time consuming, but I am trying to learn and dedicate more time into the topic

[@JoHeN1990]

From my experience, it’s just a matter of trade-off. Creating a high availability system with “all” the possible problems covered with failover mechanism is expensive so you usually try to cover only the most probable one depending on the cloud provider SLA

[@SamuelHauptmannvanDam]

The fact that this channel exists and is so succesful, is AMAZING but also kinda blowing my mind.

[@mihaiii6720]

I have to tell you the truh : your youtube channel is a genuine gold mine

[@Att4ni]

Fascinating video as always! What a great deep dive into an issue that impacted (and confused) so many of us directly

[@PenguinCrayon269]

Can you make coverage on node-ipc malware npm package?

[@Hirnlappen]

Sick audio processing, and your voice sounds epic, actually!

[@abdulrahmanalhamali1707]

Great video, and impressive management by Spotify!

[@eudesduarte5114]

I was hoping to see this here, subscribed x

[@adnana6773]

"Our services have fully recovered, I'm not recovered though!" 😂😂

[@joerivde]

This channel is pure gold

[@mohit.kunjir]

Spotify has great engineers, this is beyond shadow of doubt. However, the issue here was in service discovery and according to Spotify's report they were using DNS based service disccovery and very recently had moved to xDS based Service Discovery [ Traffic Director ] . In short, they had a backup mechanism place for this unlike Slack whose real issue was in AWS networking. Had Spotify experienced similar issue to Slack even they would have to wait for the service to come back up

[@DevDungeon]

Maybe they included the node-ipc package XD

[@autohmae]

Finally got around to watching this one. So they really needed better metrics and/or better understanding of their metrics to know if all customers had migrated.

[@hnasr]

Thanks for your insights and comments dude.

[@leeyoung]

nice video. but xDS not xSD >.

[@hnasr]

😢 i noticed during editing haha

[@leeyoung]

​@@hnasr sry, it's such a small mistake anyone can make, you are our hero still, Thank you.

[@ushiocheng]

Just imagine if they add the code to delete the old format file there 👀 (and that is why you should never delete things on production environment unless you are *absolutely* sure)

[@nazikhuq]

thats a radio voice that keeps people listening

[@RahulGupta-wd3lf]

can u make a video on Google maps outage happened on March 18.

[@taigopedrosa]

Which browser is this?

[@HarshKapadia]

Nice video! Thank you Hussein!

[@insomniac_coder]

Please make video on Okta Hacked incident.

[@PetHavic]

You had the flew too? I did at the same time as you

[@susmitvengurlekar]

You know what happens when we change formats, I was expecting, "Things go down"

[@ahmedelzubair6907]

Alsalam aliykum brother Hussain. Did you read Designing Data-Intensive Applications Book? If yes please make make a video about it.

[@ramymousa4189]

thank you for that quality video! learned a lot.

[@arif8434]

i didnt know google also ignores and leaves their known failures tests like that...hahaha

[@ericcoleman2288]

Dope explanation 👌🏾👌🏾

[@MJ-xg2ow]

Love your videos keep it up bro

[@poulain1721]

Great !!! We were expecting this one !!!

[@magfal]

Iso 8601 for all date-related operations

[@DMSBrian24]

The era of centralized web everyone

[@addiegupta]

Thanks for the amazing videos

[@murtuzaalisurti]

GitHub outage video please

[@morgengabe1]

when did this happen?

[@TheSpacecraftX]

Omegastar needs to get their shit together.

[@CYXXYC]

he kept reading "xSD" 😭

[@siya.abc123]

Thank you bro

[@arunkumarshanmugavel7385]

Keep some animations the things which you explains, Viewers cant see web page for 20 minutes

[@carlesg0n]

what i wonder is, how didn't they realise they had a bunch of configurations erroring when being read as the new format..? 😅 guessing they missed some alerting in there..

[@prhasn]

Please reupload to Odysee, it is has been almost a year since you updated your channel there.

[@tony-go-code]

Good lesson learn

[@esquilo_atomico]

amazing, as allways

[@plusone.network]

legal jargon for sorry we pushed beta to brod but it was nobody's fault, have a good day

[@therandomguyinworld]

رائع رائع انت راااااااااائع انت والمحتوى

[@matrixkavi]

Dog woke up the baby 😂

[@junkahoolik]

I thought it was always the DNS

[@TheTran-tf5ri]

It was DNS 😅

[@not2day646]

DNS is always the problem!

[@princepines]

its always the GCP

[@KangJangkrik]

I don't know why but Spotify music selection is kinda boring compared to youtube music

[@rick-kv1gl]

ur amazing.

[@abhisheksurve4059]

Hi

[@jakubbuzuk3450]

Damn that video is boring. This could be fit in like 5 minutes instead of 20...

[@hnasr]

Head to database.husseinnasser.com for a discount coupon to my Introduction to Database Engineering course. Link redirects to udemy with coupon applied.

[@testtest9273]

Have you visited psychiatrist ?

[@voxs_wav]

Please EQ ur microphone it's pretty irritating to ear