21: Distributed Locking | Systems Design Interview Questions With Ex-Google SWE

There seems to be some sort of distributed consensus that people lock their car doors whenever they see me walking near them

[@AR-be9rd]

I am a software engineer at Amazon, and your videos have helped me a lot. I was recently able to get senior and principal level offers from Oracle, Microsoft, and a few others. Thanks again. Appreciate it.

[@jordanhasnolife5163]

Holy moly that's awesome!! Congrats on your offers!!

[@ganesansanthanam-5896]

Hey, can we connect on LinkedIn?

[@TheSdl79]

The intro is hilarious😂p.s. congrats on the 30k milestone!

[@pratyushkumarsingh6161]

Once you read Designing Data-Intensive Applications all the videos becomes a great source to revisions. Great quality videos, keep up the good work!!!!

[@adrian333dev]

Thanks for the content! Preparing for an upcoming Mid level role, hopefully your videos will help me

[@jordanhasnolife5163]

Good luck!

[@stormShadow64]

You are doing awesome work

[@divijsharma5610]

Why this channel name man , you are giving life to so many. Rename the channel to Jordan gives life

[@jordanhasnolife5163]

That's not the only thing I give

[@hazardousharmonies]

Excellent job Sir

[@subhamcoder]

congrats, whta introo

[@SohailKhan-gu2du]

Hey , I love the way you teach . Can you also do this concepts in hands-on project in spring boot or something , so that we can improve our coding and also learn how to test such scenario in real Life

[@jordanhasnolife5163]

This is something I've thought about, but realistically would take me a very long time to do haha. In my current state, it's unlikely I can, but maybe if I have some significant life changes

[@oren23198]

hi jordan thanks for those vidoes very helpful, but something it's clear for me how does s3 handles the fencing tokens? doesnt the appliation requires another layer for solving this out?

[@jordanhasnolife5163]

S3 is a bit of a bad example because we don't own s3. But imagine you own whatever data source you're sinking to, you can just build this into the logic there.

[@MallardDuck77]

LE'S GO KNICKS!

[@visheshchanana5658]

In the Distributed Conses, we had 5 nodes(1 leader, 4 followers). When leader went down, we chose the follower that was up to date. What if as soon the follower was chosen as a leader, it went down. Now we have 2 nodes with old data and 1 with new. What happens in this case?

[@jordanhasnolife5163]

The one with new data must become the leader. If it goes down, we can't proceed as we can no longer reach a a majority of nodes.

[@rishabhsaxena8096]

Hey Jordan , Could you please create a video on stock exchange system design, that would majorly focus on the users getting a notification on the stocks they have subscribed if the stock values go up or down based on some parameters in real time. Thanks

[@jordanhasnolife5163]

Will do eventually!

[@ashutoshshukla6242]

How can anyone through the notes you have made during these videos? Is it present in any GitHub repository or somewhere else?

[@jordanhasnolife5163]

See channel description

[@theJeet8]

Your intros are always weirdly funny :) Question: in your final design, is the queue also written to followers? i.e. If Leader were to go down, would followers know that B is waiting for A? How would the websocket be restored by A?

[@jordanhasnolife5163]

Yep, the queue is written to followers. If the leader goes down, the clients will notice it, and reach out to the other links of nodes in the zookeeper cluster to get the address of the new leader and connect there.

[@nguyentrunghieu6200]

I'm kinda curious about the fencing token, since the destination write (in your video it's S3) it has to know/store the value of used fencing token so far, is that possible? Since I think that we might have to communicate with many 3rd parties which we do not have the "right" to do that check. How can we resolve it?

[@jordanhasnolife5163]

I'm not sure what you mean here. A fencing token is just a number that you can pass to an external service, as long as the external service allows it in their API. Then the external service will only accept writes in an increasing order of that number.

[@nguyentrunghieu6200]

@@jordanhasnolife5163 " as long as the external service allows it in their API" - that's my point. I meant, we can't be sure that the external service we want to use will always support fencing token (or have similar thing), what should we do in that case?

[@jordanhasnolife5163]

@@nguyentrunghieu6200 Use a different external service or add a stateful proxy in front of it

[@traveling_cruiser]

Awesome video.. one question.. what is the leader/follower nodes here: application server/ redis cache/ database?

[@jordanhasnolife5163]

Probably just a normal SQL db

[@codr0514]

What editor are you using for drawing? Do you also use any pen based device?

[@jordanhasnolife5163]

Apple pencil + oneNote

[@codr0514]

@@jordanhasnolife5163 thanks for your response 🫡

[@shobhitarya1637]

Nice Video but i have one query. In case of distributed consensus, how reads are done for lock ?. If it is read from replica which is not upto date, it can lead to a problem. I have also watched your raft videos, which gives me impression that distributed consensus provides lineraliziability but not strong consistency in reading.

[@jordanhasnolife5163]

You read from the leader. This is slow, but it's still fault tolerant, because we have the ability to perform a fail over if the leader goes down.

[@ananth11]

Well when you feel you are better you don’t have to prove it to anyone, just relax and see ahead and I am sure you will find a much better girl than her !!

[@jordanhasnolife5163]

Oh she's fine this was just a joke lol

[@mcee311]

what if you have a large amount of connections on the leader node? How do you deal with that situation?

[@jordanhasnolife5163]

I'm assuming this is for many different locks, you basically have to partition them across many zookeeper clusters.

[@user-kf1ul3uh2w]

@@jordanhasnolife5163 does that mean this leader + several followers just act like one zookeeper node. And for horizontal scale up we need more zookeeper nodes with sharding(and every node have their own leader and followers)?

[@jordanhasnolife5163]

@@user-kf1ul3uh2w You are correct

[@fallencheeto4762]

Is linearizable similar to causal consistency?

[@jordanhasnolife5163]

Causal consistency just implies that if a write B happened because a user first saw write A, we should never be able to read B without also having access to the A write Linearizable databases are causally consistent, but not all causally consistent databases are linearizable.

[@fallencheeto4762]

@@jordanhasnolife5163 interesting, we learn something new everyday! Great video man

[@jhonsen9842]

I rejected in System Design Round LoL I took it lightly and didn't prepare

[@jordanhasnolife5163]

Welcome

[@michaelv2555]

No Flink and CDC used? Jordan, are you ok?

[@jordanhasnolife5163]

No someone said my videos were too unrealistic for interviews on reddit and now I'm in a deep state of depression

[@Ynno2]

@@jordanhasnolife5163 I think that's true, but I don't necessarily think it's bad. Your videos aren't really structured in an interview style. You go into a lot of depth in your designs. It would be unrealistic to draw the sometimes huge designs your show at the end of your videos in the space of an interview - especially somewhere like Meta where you realistically only have 35 minutes of design time, but it's beneficial to see everything as inspiration for how you might deep dive in different areas. In a real interview you may only deep dive into a couple of the areas you show.

[@vipulspartacus7771]

Hi Jordan, really appreciate the content, is it possible for you to share your ipad notes. It is difficult to follow and revise your content without the notes and making the entire notes while following the video is time consuming. It would be really helpful if you could share your hand written notes from ipad (maybe it is not perfect but still a better reference than nothing) which we could keep as reference to follow your content. As we go through the video, we could add our own comments or notes on it to make it more clear. Please consider.

[@jordanhasnolife5163]

Planning on doing this in bulk after finishing my current series, this will be in the next 1-3 months.

[@helperclass8710]

Thanks for the great video. I need your help with one of my task. I will become your patreon if you help. In my current company I have received one task in which I have to execute queries in the order they were originally executed. I have a list of queries and their original start and end times. So to execute them again in the same order we need to build dependency graph. How we can build this dependency graph. Query 1: start time 1 end time 3 Query 2 start time 2 and end time 5 Query 3 start time 4 end time. Qry 2 can start after qry 1 has started. Query 3 can be started after 1 finished and 2 started

[@helperclass8710]

My implementation is not efficient as I am checking for every query all the query started before it and storing the dependencies in list

[@jordanhasnolife5163]

Doesn't really make much sense to me considering the start times and end times. But look up topological sorting. Make a graph of the dependency relationships, and run a topological sort. This will tell you when you can schedule a given task, at which point you can run a second job that looks at currently scheduled tasks and whether their start time has passed. I don't have a patreon, send it to charity.

[@rafaelarantes4804]

The content is great, but I always come for the golden nugget in the description

[@jordanhasnolife5163]

I churn out nuggets in the description and on the toilet

[@sid4579]

What is the source of truth for all this?

[@jordanhasnolife5163]

I'm not sure what you're asking - do you mean my sources?

[@sid4579]

@@jordanhasnolife5163 Thanks for replying! I meant where did you learn about all this? Is there a comprehensive resource or is this just result of your years of experience in tech?

[@jordanhasnolife5163]

@@sid4579 Well considering that I don't have many years of experience in tech, I'm going to say that I did not learn anything that way. I'm simply just aggregating any information that I can find across anywhere on the internet. If there was a comprehensive resource for it, I don't think I'd be making these videos in the first place, as I myself am attempting to be a comprehensive resource for it.

[@Ynno2]

@@sid4579 Martin Kleppmann's book and KZbin videos cover a lot of this.