Data Consistency and Tradeoffs in Distributed Systems

This is a detailed video on consistency in distributed systems.
00:00 What is consistency?
00:36 The simplest case
01:32 Single node problems
03:35 Splitting the data
04:23 Problems with disjoint data
07:10 Data Copies
12:01 The two generals problem
13:56 Leader Assignment
15:24 Consistency Tradeoffs
15:56 Two phase commit
25:00 Eventual Consistency
Eventual Consistency: interviewready.io/learn/syste...
You can follow me on:
Instagram: / interviewready_
LinkedIn: / interview-ready
Github: github.com/coding-parrot/
Twitter: / gkcs_
#SystemDesign #DistributedSystems #Consistency

[@MsSierra9]

Your system design videos helped me get a job at Amazon! Thank you so much for all the great work you are doing! :)

[@gkcs]

Congratulations 🍻😁

[@Jag144]

Congrats Say!!!!

[@zhizhou6443]

@@gkcs Nice!

[@zhizhou6443]

And thanks!

[@shubhamsingh6929]

For new readers, I think it should be explicitly pointed out that Consistency in ACID is very different from Consistency in CAP. ACID consistency means that the values in DB should be valid i.e. if we are making two transactions both should complete before committing to disk. CAP consistency means read requests to different nodes of a distributed system return the exact same data. A system can exist without CAP consistency but a system should never exist without ACID consistency. ACID consistency is must-have for any RDBMS to be reliable/useful while CAP consistency can be a trade-off as per requirement. To be honest C in ACID doesn't add any proper value. If a transaction is Atomic, Isolated, and Durable, it automatically becomes consistent.

[@firoufirou3161]

" C in ACID doesn't add any proper value", I don't think so, we don't have the concept of keeping the db in a valid state in Atomic, Isolated, and Durable. None of these mean that we should have the constraints respected.

[@madacsg]

"automatically becomes consistent" - if "C" already told you how to utilize "AID" properly...

[@ildar5184]

" if we are making two transactions both should complete before committing to disk." What does that mean? How I understood consistency in ACID is that constraints like foreign keys, data types etc. must always be fulfilled, i.e. you can't commit a transaction if you're inserting a foreign key that doesn't exist in a referenced table.

[@zod1ack_csgo_clips425]

@@ildar5184 That's referential integrity but in one way it ensures consistency

[@rizwan050587]

Atomicity is you are talking about

[@samarthtandale9121]

The way you teach concepts from scratch/ bottom-up way is the most favourite thing in this channel !

[@gkcs]

Thank you 😁

[@nameunknown007]

One thing I learnt from this video today is you can’t get super computers in super market.

[@jedaademola]

😂😂😂

[@eduardolacerda1557]

Thanks a lot for sharing this content! this is very helpful. Although I was aware of a lot of concepts there, your samples help me to figure them out a lot better. :)

[@srikanthmaganty524]

I like ur energy and the selfless appetite to share with community...

[@benparker8000]

Amazing video. So easy to understand complicated topics. I have very limited experience with this and the pace and the content was perfect.

[@benparker8000]

Did the eventual consistency video ever come out?

[@yt-hq6cu]

You said facebook is down and it was down after a week. what a coincident!

[@sankalparora9374]

Amazing explanation! Thanks a lot! I feel more and more confident in system design daily... Thanks to you, sir!

[@gkcs]

Thanks Sankalp!

[@akhilguptagunturu3048]

Great explanation. Thanks for creating this video :)

[@bimakumkum277]

Thank you for this great explanation

[@robotempire]

Thanks bro I have been doing front-end for the past 5 years. I’m just putting myself back on the market now and this video is a great refresher on the fundamentals consistency in distributed systems.

[@thisalma]

Very well explained ❤️

[@amit_dwivedi]

The Tshirt Gaurav! 😍 NITS ❣️

[@cengizandak4241]

Great explanation. However 2 phase commit is not usually recommended in the current industry due to a possible failure on the coordinator or the microservies will mix the things up by keeping locking the resources, timing out etc.. Therefore there are other options like 3 phase commit or SAGA which are widely used in fintech as well as in big companies.

[@gokuls9929]

That 'Yeah! I made a joke!' look at 8:32 is priceless!

[@sophianachiba663]

very clear as always... what does the prepare statement phase offer that the commit doesn't in the 2 phase commit? since they can both fail and both may need to be rolledback.

[@aniketsingh9956]

That 'C++' expression was priceless 😁

[@gkcs]

Hahaha!

[@RAJU9622]

Excellent Gaurav

[@hamsalekhavenkatesh3440]

Nice, its also worth knowing more than 2PC, we can also use TC/C (Try Cancel/commit ) which does not explicitly lock the resources; rather it works by issuing compensating tx in case of failures..

[@mukunthsenthilkumar8822]

This video is pure gold! Great stuff, Gaurav!

[@raj_kundalia]

Thanks for the video

[@stephensmithwick7502]

I loved the pause on the C++ joke >_< as if to say yes I’m going there

[@aravindkumaresan2747]

Hi Gaurav, thanks for making such great contents regarding consistency in distributed systems. But in case of 2 Phase commit, if one of the follower node machine goes down, then the leader who got commit response from other follwers, also can't commit it and may need to rollup the transactions. So until and unless the single follower comes back, the system doesn't accept any write operations, so it still puts in a single point of failure right ?

[@harshkatkade4206]

In the 2 phase protocol, consider a scenario. If server C receives a begin transaction order from A, and further fails to receive a commit order, what if we lock the read requests for some amount of time from C, until a "begin" transaction orders isn't committed yet, so that the leader can keep trying to commit the change in C. Latency is increased but I guess consistency can be maintained. Is this a correct? Please let me know

[@Thrillseeker419]

Here are my concerns: how do you track the prepare statements sent? How will server B differentiate two duplicate prepare statements? Also, as part of the prepare statement execution can we not update a status column of a row to indicate it is in process of being updated? Also how does the leader keep track of all of the data that needs to be sent out? What if there is just too much data to be sent out for one leader to handle?

[@DejaimeNeto]

"Maybe C was updated here; maybe C++", shots fired

[@shivprakashy]

Don't know if you have a video on the books/blogs you follow. 👏👌#goodcontent for #systemdesign

[@kalpakHere]

One of my favourite topics in this domain. Up next: Consensus ??

[@pallavgurha]

Hi Gaurav, just wondering whether high water mark index will help here and the client can be assured that the commit is acknowledged as anyways everything noways a async in nature and once all the nodes in the cluster are up to the speed we can start showing the change to the client.

[@hasanulislam3112]

Could you recommend any good system design concepts apart from your interview-focused course?

[@stIncMale]

'C' in CAP is what 'I' in ACID. 'C' in ACID has no analogy in CAP and means that there are constraints and the DBMS makes sure they are not violated. More specifically, 'C' in CAP is linearizability, while 'I' in ACID is serializability (and not even strict serializability, which would be more similar to linearizability). While both are what is called "consistency model" or "correctness condition" (a similar concept in programming languages and in processor instruction set architectures is called "memory model"), they are not the same.

[@gkcs]

That's a good point. The "Consistency" term lines up a lot more with Isolation in ACID. I think Martin Kleppmann has spoken about this a few times.

[@nodirbeksharipov458]

you're a genious man

[@kajalpareek8291]

Hi Gaurav ,Thanks for such informative content. Looking forward for a series on design pattern

[@sumitkumarmallick6040]

Can you please share the link to the video where I can find the idempotency concept?

[@zaleel]

Have a question. If all the commit/prepares fail after multiple retries. Should we roll back on master as well? And in in 2 PC, instead of prepare and commit. Why can't we just: First send the update in phase 1 i.e. commit and then in phase 2 just compare A /master to slaves. If if matches it's done. If it fails we mark the state as disputed or eventual roll back

[@stalera]

Great work. Thanks for the knowledge. However, in the explanation there's 1 server with write which makes it again a single point failure and if we increase the write servers then we're back to square one. Can you address this?

[@TheNayanava]

in one of my previous organizations we solved this by enabling multi-master, or multiple primary. We had three primary nodes but all of the write requests would only come to one primary node. there was automatic failover in this scenario, there a linux virtual server which takes care of routing the traffic, the LVS was updated with the public IP of the node which was the then primary. and every time the call would land on the LVS it would resolve to the IP which was the primary. there was automatic failover in this scenario. But as you see every write would be successfully ack-ed only when all three servers had the latest commit.

[@cengizandak4241]

2 phase is not recommended usually. There is a better option like 3 phase commit or SAGA. l would recommend you to check them out

[@TheIndianGam3r]

We cant have more than 1 write servers in all types of DBs. For example, postgreSQL doesn't allow multi-master setup. This issue is resolved by implementing a High availability cluster management tool along with your database. For example , Galera Cluster for Mysql and Patroni for postgreSQL. Here, in case a master fails, one of the read replica servers is chosen as a new master , and a proxy on top handles the change in route. Multi-master setup can have many problems regarding duplicate entries if not handled well.

[@TheNayanava]

@@TheIndianGam3r yeah we were using a gallera cluster on top of Maria db, but although it was a multi master, the writes are accepted only on a single master, the others were exact copies of the master accepting the writes. This helped in automatic failover instead of manual failovers. And true enabling writes on multiple masters actually increased the chatter, that's why we disabled writes on multiple masters.. this was more of a multiple replicas of the same master

[@mukeshstorge7384]

In this cluster each slave will have the knowledge of Master's health by checking the heart beat. Once the Master goes down leader election will takes place and one of the slave becomes master. There are some algorithms involved here like RAFT or Consensus for leader election.

[@sahilzainuddin7134]

Just finished watching Social Network, and right in the next video click, I see an example of the same.

[@Amritanjali]

again learned a lot

[@gkcs]

Thanks Amritanjali! Good morning 😁

[@Amritanjali]

@@gkcs good afternoon ☺☺

[@jeelanyelidandla2477]

Gaurav, do the UK, US, and other countries' governments agree to store their data across other regions? what do you think?

[@ThePujjwal]

Isn't 2PC used for maintaining atomicity in case of a distributed transaction ? Got me a bit confused about we leveraging it to achieve consistency.

[@generationgap416]

Atomicity is all or nothing. Consistency is data copies should be the same on different on different bot in a distributed environment

[@theSDE2]

Liked the new edit. Kudos to your techie video editor 🤓

[@gkcs]

Yes, it's amazing 😁

[@marcpaguilar]

The very quick smile you gave the camera when you said "c++" 🤣🤣🤣🤣

[@praveensg]

Wow, nice and bright. New camera?:)

[@Lima3578user]

KZbin recommend your video at 6am today ..ever since im hooked ... prepping for technical program manager role .. hope you would make some program centric videos too

[@suchismitagoswami5609]

Great explanation. I am having one questions as followed. Ideally, the followers or the replicas should reside in different regions or data centers to ensure availability. So, this introduces latency in every write operation in transforming the data synchronously from leader to the followers if we follow 2 phase locking. How to handle this ?

[@sidderverse]

I believe you will have to go for eventual consistency.

[@generationgap416]

That is why there are tradeoffs between consistency, availability and performance. There are mutually exclusive. Eventual consistency is the middle ground.

[@Varunshrivastava007]

How did you solve the latency in a consistent system. Still servers have to send request from UK to EU for db updates (to remain consistent).

[@Bibhaw]

What is the use of kafka if all the communication been hapening directly from Leader to follower ?

[@imperfecto7734]

08:30 he realises how c++ language got its name. its intuitive :D

[@tylerbenton4495]

Would be great if the people who disliked the video commented on why they disliked it. Did you disagree with something he said? If so, write it in the comments to educate us on why you disagree.

[@praveensg]

Insecurity

[@kayodeadechinan5928]

A very wise comment! Thanks

[@manveersingh5822]

That C++ smile and expression at 8:30 .. haha

[@saiprasad84]

Isn't the leader a SPOF here? Just curious.. may be if the leader fails, one of the followers become the leader? Wonder how that might happen and if there are databases that make it happen automatically.

[@darkstudio3170]

He is wearing NITS Hacks T-shirt. I still remember when he came to NIT Silchar as a judge for NIT Hacks and we had a photo with him.

[@gkcs]

I'm proud to wear it 🤠

[@sagararora2830]

Now I get to know When we went to banks and they are like servers are not responding, but actually in the backend they are keeping themselves consistent. :D

[@praveenkurapati7300]

While rollback sent by the master, does it send to all slaves ?

[@vickyanand5898]

Rollback sent by the leader can fail as well right ? Near to 23rd minute

[@surenderthakran6622]

Loved the video. Just to nitpick a little, Kafka belongs more to the pub/sub family rather than the message queue family.

[@generationgap416]

It does both of those thing well. Dont nikpik

[@aj-loves-tech]

8:32 that smile hits me every time 😂😂😂

[@gkcs]

C++ 😛

[@BarHemo32]

Why not use kafka or rabbitmq and store in them all of the updates, and each database will subscribe and make the changes according to the queue. The data will be eventually consistence, is this a good idea?

[@AnPham-uz3td]

But kafka / rabbitmq will need a way to send update to those follower dbs, and to guarantee consistency between those follower dbs it will just like the 2 phase commit

[@akashtadwai9471]

Hi Guarav, In a two-phase commit, do the transactions of all the servers be rolled back if one of the servers fails to acknowledge?

[@pste22]

yes it should! 2 phase commit is all about establishing consistency between all the related services!

[@generationgap416]

Good question. There should be a thing called Almost Consistent. Use reference counting. There should also a thing called Regional of Consistency. Data sharding should necessitate this.

[@DebajyotiDev]

Is C in ACID properties o SQL same like CAP theorem? What u said is for CAP. But if u check the oracle Docs Consistency in ACID means Data should not violate any DB constraints FK, UK, Col types etc…correct me if am wrong.

[@generationgap416]

You are right. C in acid relate more to atomicity and idempotency. While the C in CAP refers to data copies in a distribuuted environment must be the same. Hash value of each copy must equal if there are no collision in the hash function.

[@lakeman4101]

When i started listening to your videos ... i had no idea other e than you know your stuff, right now I can keep up quite alright...i now appreciate the level of your skill...and the chronological argument you give and use to explain. I love this man

[@mrrishiraj88]

🙏🙏🙏

[@n4naveeen]

Hi Gaurav, can you pls make a video on lifecycle of code? i e, what happens after the coding is done?

[@gkcs]

I have spoken about deploying code and versioning client libraries at InterviewReady. You can check out the first design of "Design an email service" here: get.interviewready.io

[@sagararora2830]

Hi Gaurav Sen, when the video on Eventual consistency will come. Waiting for it

[@gkcs]

Added it here: get.interviewready.io/learn/system-design-course/consistency-in-distributed-systems/Eventual-Consistency

[@basitahmad8067]

Next Hotstar 🙏... High in demand

[@gkcs]

Check out the free course preview at InterviewReady 😎

[@Tarsemsingh-rd8lx]

Thanks Gaurav, I have question, which tool support 2 phase commits? Kafka can you please suggest open source tool

[@vinays2493]

What does GKCS stand for

[@ParthPatel-jn6io]

At 1:24 there's a mis-communication and writing on board for Get profile A

[@sankalparora9374]

That gaze 8:32.

[@gkcs]

😛

[@sureshbabu8794]

What about partitioning? You didn't explain or I might have missed..

[@arulantony2137]

25:00 SAGA pattern ??

[@shakeelahmed8015]

So we can roll back messages from Kafka 🤔

[@sorandom9452]

In case of single leader, there are high chance of single point failure.

[@TheNayanava]

in one of my previous organizations we solved this by enabling multi-master, or multiple primary. We had three primary nodes but all of the write requests would only come to one primary node. there was automatic failover in this scenario, there a linux virtual server which takes care of routing the traffic, the LVS was updated with the public IP of the node which was the then primary. and every time the call would land on the LVS it would resolve to the IP which was the primary. there was automatic failover in this scenario. But as you see every write would be successfully ack-ed only when all three servers had the latest commit.

[@eatcodegame4952]

Incase for example you have 3 nodes in a cluster, incase of any write you can get it ack from more than 1 node so you have more than one node than master which does have the updated copy and it’s gonna take place of master incase of master goes down.

[@generationgap416]

Use a replica set of masters and use a kubernetes to orchestrate the environment.

[@Raja-kl4op]

Hi Gaurav, Great video! But I have a doubt. Why does communication over a long distance lead to latency? Don't servers use electromagnetic waves(speed of 3 x 10^8 m/s) to communicate with each other. So, the distance of a few 1000KM's will only cost few milliseconds. Is that enough to cause latency? Or did I miss something?

[@gkcs]

There are more routers to hop on.

[@generationgap416]

The last mile is the bottle neck

[@rajveersingh2056]

Tsunami in Harvard☺️

[@vaibhavmehta36]

2PC is used in which Industry?

[@gkcs]

Textiles. And also some financial transaction systems.

[@rajvadheraju3568]

It seems like you mixed Consistency and Distributed Transactions when you talk 2PC

[@yashwanthd1998]

We are trying to solve single point of failure and consistency with distributed databases but ending up with the same . What am i missing here

[@generationgap416]

If you are refering to the seemingly single point of failure of the master, note that the master would be a member of a set of master. A service kubernetes would orchestrate the master. If one down kuberntes would start a copy of the master. You would an avalaible problem but that is point of the video that perfect consistency and perfect availability is not possible

[@zuowang9881]

I don’t think he released the next eventual consistency video

[@gkcs]

interviewready.io/learn/system-design-course/consistency-in-distributed-systems/Eventual-Consistency

[@harshshah2791]

What happens if transaction is to add 100 rs and leader retried 100 times, so 100 rs added 100 times?

[@gkcs]

The transaction id will make the server ignore 99 of the later requests.

[@harshshah2791]

@@gkcs I would like to ask one more thing that I see your SD course on IR includes many videos which are already there on KZbin, so can you give an idea what I will get extra if I buy the course? And also what is the source of your SD knowledge? Any goto books, sites, pages?

[@gkcs]

@@harshshah2791 All the lessons in the "additional free resources" have been picked from this KZbin playlist. The other three sections (Fundamentals, High level design and Low level design) are only available at InterviewReady. I use various books, blogs and my personal experience as a software engineer in the content. The course is designed to help you get better at software engineering (to get better at design discussions, coding, etc...).

[@harshshah2791]

@@gkcs I will surely buy it

[@adityagoel1738]

Retrying every 5 seconds doesn't make sense if the system is down because it's not gonna be up and running soon and hitting it at every 5 seconds leads to wastage of resources.

[@swastikjainsj]

how Inertnet work ? GB MA how jio airtel control net speed ? Any new info ? apart from underground wires 😐

[@liatris69]

Data sharing shyness is real.

[@mohammadhemel6412]

While I am watching this videos, Facebook, Instagram and WhatsApp are down

[@swastikjainsj]

one more question When we delete anything it store in our computer only in hard disk just remove the link so that we cannot find that file And now we can store another data on this data it will overlap. but how can we transfer the deleted storage files to cloud that means our disc will can work for more years

[@generationgap416]

You may able to find using an hex editor

[@imperfecto7734]

13:38 "I wont explain everything here, I'm lazy too "

[@samarthagrawal3415]

Sir i am very much inspired from your english conversation, will u make a video on how to do coding problems means how to build logic for that please sir it's my humble request, please make video on that

[@Amritanjali]

he already made one video on this check-in his channel

[@samarthagrawal3415]

@@Amritanjali ok thanks...

[@tze-ven]

Do you know you don't need supercomputer to scale? You can scale with cloud computing. The best thing is that you don't need physical shopping. You just click and pay. 😜

[@generationgap416]

He was contrasting virtical scaling with horizontal scaling. Question: Would virtical scalingbe the same as horizontal if we rotate virtical scaling by 90°?

[@manjuender6286]

Super computer at super market Harvard vs Oxford 😂

[@VijayInani]

C >> C++ ... LOL

[@jitinkumar835]

1st