Your system design videos helped me get a job at Amazon! Thank you so much for all the great work you are doing! :)
@gkcs3 жыл бұрын
Congratulations 🍻😁
@Jag1443 жыл бұрын
Congrats Say!!!!
@zhizhou64432 жыл бұрын
@@gkcs Nice!
@zhizhou64432 жыл бұрын
And thanks!
@shubhamsingh69292 жыл бұрын
For new readers, I think it should be explicitly pointed out that Consistency in ACID is very different from Consistency in CAP. ACID consistency means that the values in DB should be valid i.e. if we are making two transactions both should complete before committing to disk. CAP consistency means read requests to different nodes of a distributed system return the exact same data. A system can exist without CAP consistency but a system should never exist without ACID consistency. ACID consistency is must-have for any RDBMS to be reliable/useful while CAP consistency can be a trade-off as per requirement. To be honest C in ACID doesn't add any proper value. If a transaction is Atomic, Isolated, and Durable, it automatically becomes consistent.
@firoufirou31612 жыл бұрын
" C in ACID doesn't add any proper value", I don't think so, we don't have the concept of keeping the db in a valid state in Atomic, Isolated, and Durable. None of these mean that we should have the constraints respected.
@madacsg2 жыл бұрын
"automatically becomes consistent" - if "C" already told you how to utilize "AID" properly...
@ildar5184 Жыл бұрын
" if we are making two transactions both should complete before committing to disk." What does that mean? How I understood consistency in ACID is that constraints like foreign keys, data types etc. must always be fulfilled, i.e. you can't commit a transaction if you're inserting a foreign key that doesn't exist in a referenced table.
@zod1ack_csgo_clips425 Жыл бұрын
@@ildar5184 That's referential integrity but in one way it ensures consistency
@rizwan050587 Жыл бұрын
Atomicity is you are talking about
@samarthtandale9121 Жыл бұрын
The way you teach concepts from scratch/ bottom-up way is the most favourite thing in this channel !
@gkcs Жыл бұрын
Thank you 😁
@benparker80002 жыл бұрын
Amazing video. So easy to understand complicated topics. I have very limited experience with this and the pace and the content was perfect.
@benparker80002 жыл бұрын
Did the eventual consistency video ever come out?
@cengizandak42413 жыл бұрын
Great explanation. However 2 phase commit is not usually recommended in the current industry due to a possible failure on the coordinator or the microservies will mix the things up by keeping locking the resources, timing out etc.. Therefore there are other options like 3 phase commit or SAGA which are widely used in fintech as well as in big companies.
@srikanthmaganty5242 жыл бұрын
I like ur energy and the selfless appetite to share with community...
@robotempire3 жыл бұрын
Thanks bro I have been doing front-end for the past 5 years. I’m just putting myself back on the market now and this video is a great refresher on the fundamentals consistency in distributed systems.
@sankalparora9374 Жыл бұрын
Amazing explanation! Thanks a lot! I feel more and more confident in system design daily... Thanks to you, sir!
@gkcs Жыл бұрын
Thanks Sankalp!
@hamsalekhavenkatesh34409 ай бұрын
Nice, its also worth knowing more than 2PC, we can also use TC/C (Try Cancel/commit ) which does not explicitly lock the resources; rather it works by issuing compensating tx in case of failures..
@yt-hq6cu3 жыл бұрын
You said facebook is down and it was down after a week. what a coincident!
@amit_dwivedi3 жыл бұрын
The Tshirt Gaurav! 😍 NITS ❣️
@nameunknown0073 жыл бұрын
One thing I learnt from this video today is you can’t get super computers in super market.
@jedaademola11 ай бұрын
😂😂😂
@stIncMale3 жыл бұрын
'C' in CAP is what 'I' in ACID. 'C' in ACID has no analogy in CAP and means that there are constraints and the DBMS makes sure they are not violated. More specifically, 'C' in CAP is linearizability, while 'I' in ACID is serializability (and not even strict serializability, which would be more similar to linearizability). While both are what is called "consistency model" or "correctness condition" (a similar concept in programming languages and in processor instruction set architectures is called "memory model"), they are not the same.
@gkcs3 жыл бұрын
That's a good point. The "Consistency" term lines up a lot more with Isolation in ACID. I think Martin Kleppmann has spoken about this a few times.
@aniketsingh9956 Жыл бұрын
That 'C++' expression was priceless 😁
@gkcs Жыл бұрын
Hahaha!
@lakeman41013 жыл бұрын
When i started listening to your videos ... i had no idea other e than you know your stuff, right now I can keep up quite alright...i now appreciate the level of your skill...and the chronological argument you give and use to explain. I love this man
@stalera3 жыл бұрын
Great work. Thanks for the knowledge. However, in the explanation there's 1 server with write which makes it again a single point failure and if we increase the write servers then we're back to square one. Can you address this?
@TheNayanava3 жыл бұрын
in one of my previous organizations we solved this by enabling multi-master, or multiple primary. We had three primary nodes but all of the write requests would only come to one primary node. there was automatic failover in this scenario, there a linux virtual server which takes care of routing the traffic, the LVS was updated with the public IP of the node which was the then primary. and every time the call would land on the LVS it would resolve to the IP which was the primary. there was automatic failover in this scenario. But as you see every write would be successfully ack-ed only when all three servers had the latest commit.
@cengizandak42413 жыл бұрын
2 phase is not recommended usually. There is a better option like 3 phase commit or SAGA. l would recommend you to check them out
@TheIndianGam3r3 жыл бұрын
We cant have more than 1 write servers in all types of DBs. For example, postgreSQL doesn't allow multi-master setup. This issue is resolved by implementing a High availability cluster management tool along with your database. For example , Galera Cluster for Mysql and Patroni for postgreSQL. Here, in case a master fails, one of the read replica servers is chosen as a new master , and a proxy on top handles the change in route. Multi-master setup can have many problems regarding duplicate entries if not handled well.
@TheNayanava3 жыл бұрын
@@TheIndianGam3r yeah we were using a gallera cluster on top of Maria db, but although it was a multi master, the writes are accepted only on a single master, the others were exact copies of the master accepting the writes. This helped in automatic failover instead of manual failovers. And true enabling writes on multiple masters actually increased the chatter, that's why we disabled writes on multiple masters.. this was more of a multiple replicas of the same master
@mukeshstorge73843 жыл бұрын
In this cluster each slave will have the knowledge of Master's health by checking the heart beat. Once the Master goes down leader election will takes place and one of the slave becomes master. There are some algorithms involved here like RAFT or Consensus for leader election.
@stephensmithwick75023 жыл бұрын
I loved the pause on the C++ joke >_< as if to say yes I’m going there
@eduardolacerda15572 жыл бұрын
Thanks a lot for sharing this content! this is very helpful. Although I was aware of a lot of concepts there, your samples help me to figure them out a lot better. :)
@bimakumkum2778 ай бұрын
Thank you for this great explanation
@gokuls99292 жыл бұрын
That 'Yeah! I made a joke!' look at 8:32 is priceless!
@aravindkumaresan27472 жыл бұрын
Hi Gaurav, thanks for making such great contents regarding consistency in distributed systems. But in case of 2 Phase commit, if one of the follower node machine goes down, then the leader who got commit response from other follwers, also can't commit it and may need to rollup the transactions. So until and unless the single follower comes back, the system doesn't accept any write operations, so it still puts in a single point of failure right ?
@harshkatkade42063 жыл бұрын
In the 2 phase protocol, consider a scenario. If server C receives a begin transaction order from A, and further fails to receive a commit order, what if we lock the read requests for some amount of time from C, until a "begin" transaction orders isn't committed yet, so that the leader can keep trying to commit the change in C. Latency is increased but I guess consistency can be maintained. Is this a correct? Please let me know
@newbie80512 ай бұрын
Woah, the tshirt is from NIT Silchar daaaamn great stuff as always, thanks budddy
@gkcs2 ай бұрын
Cheers!
@sahilzainuddin71343 жыл бұрын
Just finished watching Social Network, and right in the next video click, I see an example of the same.
@surenderthakran66223 жыл бұрын
Loved the video. Just to nitpick a little, Kafka belongs more to the pub/sub family rather than the message queue family.
@TheGenerationGapPodcast2 жыл бұрын
It does both of those thing well. Dont nikpik
@thisalma3 жыл бұрын
Very well explained ❤️
@tylerbenton44953 жыл бұрын
Would be great if the people who disliked the video commented on why they disliked it. Did you disagree with something he said? If so, write it in the comments to educate us on why you disagree.
@praveensg3 жыл бұрын
Insecurity
@kayodeadechinan59283 жыл бұрын
A very wise comment! Thanks
@jeelanyelidandla24772 жыл бұрын
Gaurav, do the UK, US, and other countries' governments agree to store their data across other regions? what do you think?
@imperfecto77343 жыл бұрын
08:30 he realises how c++ language got its name. its intuitive :D
@kalpakHere3 жыл бұрын
One of my favourite topics in this domain. Up next: Consensus ??
@darkstudio31702 жыл бұрын
He is wearing NITS Hacks T-shirt. I still remember when he came to NIT Silchar as a judge for NIT Hacks and we had a photo with him.
@gkcs2 жыл бұрын
I'm proud to wear it 🤠
@hasanulislam31123 жыл бұрын
Could you recommend any good system design concepts apart from your interview-focused course?
@Lima3578user3 жыл бұрын
KZbin recommend your video at 6am today ..ever since im hooked ... prepping for technical program manager role .. hope you would make some program centric videos too
@Thrillseeker4193 жыл бұрын
Here are my concerns: how do you track the prepare statements sent? How will server B differentiate two duplicate prepare statements? Also, as part of the prepare statement execution can we not update a status column of a row to indicate it is in process of being updated? Also how does the leader keep track of all of the data that needs to be sent out? What if there is just too much data to be sent out for one leader to handle?
@akhilguptagunturu30486 ай бұрын
Great explanation. Thanks for creating this video :)
@vickyanand58985 ай бұрын
Rollback sent by the leader can fail as well right ? Near to 23rd minute
@BarHemo323 жыл бұрын
Why not use kafka or rabbitmq and store in them all of the updates, and each database will subscribe and make the changes according to the queue. The data will be eventually consistence, is this a good idea?
@AnPham-uz3td2 жыл бұрын
But kafka / rabbitmq will need a way to send update to those follower dbs, and to guarantee consistency between those follower dbs it will just like the 2 phase commit
@Bibhaw Жыл бұрын
What is the use of kafka if all the communication been hapening directly from Leader to follower ?
@DejaimeNeto3 жыл бұрын
"Maybe C was updated here; maybe C++", shots fired
@praveenkurapati73003 жыл бұрын
While rollback sent by the master, does it send to all slaves ?
@marcpaguilar2 жыл бұрын
The very quick smile you gave the camera when you said "c++" 🤣🤣🤣🤣
@shivprakashy3 жыл бұрын
Don't know if you have a video on the books/blogs you follow. 👏👌#goodcontent for #systemdesign
@sophianachiba6633 жыл бұрын
very clear as always... what does the prepare statement phase offer that the commit doesn't in the 2 phase commit? since they can both fail and both may need to be rolledback.
@kajalpareek82913 жыл бұрын
Hi Gaurav ,Thanks for such informative content. Looking forward for a series on design pattern
@RAJU96223 жыл бұрын
Excellent Gaurav
@sagararora28303 жыл бұрын
Now I get to know When we went to banks and they are like servers are not responding, but actually in the backend they are keeping themselves consistent. :D
@basitahmad80673 жыл бұрын
Next Hotstar 🙏... High in demand
@gkcs3 жыл бұрын
Check out the free course preview at InterviewReady 😎
@zaleel3 жыл бұрын
Have a question. If all the commit/prepares fail after multiple retries. Should we roll back on master as well? And in in 2 PC, instead of prepare and commit. Why can't we just: First send the update in phase 1 i.e. commit and then in phase 2 just compare A /master to slaves. If if matches it's done. If it fails we mark the state as disputed or eventual roll back
@raj_kundalia Жыл бұрын
Thanks for the video
@sumitkumarmallick60403 жыл бұрын
Can you please share the link to the video where I can find the idempotency concept?
@saiprasad843 жыл бұрын
Isn't the leader a SPOF here? Just curious.. may be if the leader fails, one of the followers become the leader? Wonder how that might happen and if there are databases that make it happen automatically.
@ThePujjwal3 жыл бұрын
Isn't 2PC used for maintaining atomicity in case of a distributed transaction ? Got me a bit confused about we leveraging it to achieve consistency.
@TheGenerationGapPodcast2 жыл бұрын
Atomicity is all or nothing. Consistency is data copies should be the same on different on different bot in a distributed environment
@suchismitagoswami56092 жыл бұрын
Great explanation. I am having one questions as followed. Ideally, the followers or the replicas should reside in different regions or data centers to ensure availability. So, this introduces latency in every write operation in transforming the data synchronously from leader to the followers if we follow 2 phase locking. How to handle this ?
@sidderverse2 жыл бұрын
I believe you will have to go for eventual consistency.
@TheGenerationGapPodcast2 жыл бұрын
That is why there are tradeoffs between consistency, availability and performance. There are mutually exclusive. Eventual consistency is the middle ground.
@DebajyotiDev2 жыл бұрын
Is C in ACID properties o SQL same like CAP theorem? What u said is for CAP. But if u check the oracle Docs Consistency in ACID means Data should not violate any DB constraints FK, UK, Col types etc…correct me if am wrong.
@TheGenerationGapPodcast2 жыл бұрын
You are right. C in acid relate more to atomicity and idempotency. While the C in CAP refers to data copies in a distribuuted environment must be the same. Hash value of each copy must equal if there are no collision in the hash function.
@praveensg3 жыл бұрын
Wow, nice and bright. New camera?:)
@pallavgurha3 жыл бұрын
Hi Gaurav, just wondering whether high water mark index will help here and the client can be assured that the commit is acknowledged as anyways everything noways a async in nature and once all the nodes in the cluster are up to the speed we can start showing the change to the client.
@nodirbeksharipov4582 жыл бұрын
you're a genious man
@mukunthsenthilkumar88223 жыл бұрын
This video is pure gold! Great stuff, Gaurav!
@vinays24933 жыл бұрын
What does GKCS stand for
@ParthPatel-jn6io2 жыл бұрын
At 1:24 there's a mis-communication and writing on board for Get profile A
@Varunshrivastava0072 жыл бұрын
How did you solve the latency in a consistent system. Still servers have to send request from UK to EU for db updates (to remain consistent).
@sureshbabu87942 жыл бұрын
What about partitioning? You didn't explain or I might have missed..
@Amritanjali3 жыл бұрын
again learned a lot
@gkcs3 жыл бұрын
Thanks Amritanjali! Good morning 😁
@Amritanjali3 жыл бұрын
@@gkcs good afternoon ☺☺
@sagararora28303 жыл бұрын
Hi Gaurav Sen, when the video on Eventual consistency will come. Waiting for it
@gkcs2 жыл бұрын
Added it here: get.interviewready.io/learn/system-design-course/consistency-in-distributed-systems/Eventual-Consistency
@shakeelahmed80153 жыл бұрын
So we can roll back messages from Kafka 🤔
@n4naveeen3 жыл бұрын
Hi Gaurav, can you pls make a video on lifecycle of code? i e, what happens after the coding is done?
@gkcs3 жыл бұрын
I have spoken about deploying code and versioning client libraries at InterviewReady. You can check out the first design of "Design an email service" here: get.interviewready.io
@manveersingh58222 жыл бұрын
That C++ smile and expression at 8:30 .. haha
@sorandom94523 жыл бұрын
In case of single leader, there are high chance of single point failure.
@TheNayanava3 жыл бұрын
in one of my previous organizations we solved this by enabling multi-master, or multiple primary. We had three primary nodes but all of the write requests would only come to one primary node. there was automatic failover in this scenario, there a linux virtual server which takes care of routing the traffic, the LVS was updated with the public IP of the node which was the then primary. and every time the call would land on the LVS it would resolve to the IP which was the primary. there was automatic failover in this scenario. But as you see every write would be successfully ack-ed only when all three servers had the latest commit.
@eatcodegame49523 жыл бұрын
Incase for example you have 3 nodes in a cluster, incase of any write you can get it ack from more than 1 node so you have more than one node than master which does have the updated copy and it’s gonna take place of master incase of master goes down.
@TheGenerationGapPodcast2 жыл бұрын
Use a replica set of masters and use a kubernetes to orchestrate the environment.
@yashwanthd19982 жыл бұрын
We are trying to solve single point of failure and consistency with distributed databases but ending up with the same . What am i missing here
@TheGenerationGapPodcast2 жыл бұрын
If you are refering to the seemingly single point of failure of the master, note that the master would be a member of a set of master. A service kubernetes would orchestrate the master. If one down kuberntes would start a copy of the master. You would an avalaible problem but that is point of the video that perfect consistency and perfect availability is not possible
@theSDE23 жыл бұрын
Liked the new edit. Kudos to your techie video editor 🤓
@gkcs3 жыл бұрын
Yes, it's amazing 😁
@Raja_rngnj3 жыл бұрын
Hi Gaurav, Great video! But I have a doubt. Why does communication over a long distance lead to latency? Don't servers use electromagnetic waves(speed of 3 x 10^8 m/s) to communicate with each other. So, the distance of a few 1000KM's will only cost few milliseconds. Is that enough to cause latency? Or did I miss something?
@gkcs3 жыл бұрын
There are more routers to hop on.
@TheGenerationGapPodcast2 жыл бұрын
The last mile is the bottle neck
@obinator90654 ай бұрын
8:33 that smirk :D
@akashtadwai94713 жыл бұрын
Hi Guarav, In a two-phase commit, do the transactions of all the servers be rolled back if one of the servers fails to acknowledge?
@pste223 жыл бұрын
yes it should! 2 phase commit is all about establishing consistency between all the related services!
@TheGenerationGapPodcast2 жыл бұрын
Good question. There should be a thing called Almost Consistent. Use reference counting. There should also a thing called Regional of Consistency. Data sharding should necessitate this.
@vaibhavmehta363 жыл бұрын
2PC is used in which Industry?
@gkcs3 жыл бұрын
Textiles. And also some financial transaction systems.
@aj-loves-tech Жыл бұрын
8:32 that smile hits me every time 😂😂😂
@gkcs Жыл бұрын
C++ 😛
@zuowang98812 жыл бұрын
I don’t think he released the next eventual consistency video
What happens if transaction is to add 100 rs and leader retried 100 times, so 100 rs added 100 times?
@gkcs2 жыл бұрын
The transaction id will make the server ignore 99 of the later requests.
@harshshah27912 жыл бұрын
@@gkcs I would like to ask one more thing that I see your SD course on IR includes many videos which are already there on KZbin, so can you give an idea what I will get extra if I buy the course? And also what is the source of your SD knowledge? Any goto books, sites, pages?
@gkcs2 жыл бұрын
@@harshshah2791 All the lessons in the "additional free resources" have been picked from this KZbin playlist. The other three sections (Fundamentals, High level design and Low level design) are only available at InterviewReady. I use various books, blogs and my personal experience as a software engineer in the content. The course is designed to help you get better at software engineering (to get better at design discussions, coding, etc...).
@harshshah27912 жыл бұрын
@@gkcs I will surely buy it
@swastikjainsj3 жыл бұрын
one more question When we delete anything it store in our computer only in hard disk just remove the link so that we cannot find that file And now we can store another data on this data it will overlap. but how can we transfer the deleted storage files to cloud that means our disc will can work for more years
@TheGenerationGapPodcast2 жыл бұрын
You may able to find using an hex editor
@rajvadheraju35683 жыл бұрын
It seems like you mixed Consistency and Distributed Transactions when you talk 2PC
@Tarsemsingh-rd8lx3 жыл бұрын
Thanks Gaurav, I have question, which tool support 2 phase commits? Kafka can you please suggest open source tool
@swastikjainsj3 жыл бұрын
how Inertnet work ? GB MA how jio airtel control net speed ? Any new info ? apart from underground wires 😐
@rajveersingh20562 жыл бұрын
Tsunami in Harvard☺️
@arulantony21373 жыл бұрын
25:00 SAGA pattern ??
@adityagoel17383 жыл бұрын
Retrying every 5 seconds doesn't make sense if the system is down because it's not gonna be up and running soon and hitting it at every 5 seconds leads to wastage of resources.
@sankalparora9374 Жыл бұрын
That gaze 8:32.
@gkcs Жыл бұрын
😛
@mohammadhemel64123 жыл бұрын
While I am watching this videos, Facebook, Instagram and WhatsApp are down
@samarthagrawal34153 жыл бұрын
Sir i am very much inspired from your english conversation, will u make a video on how to do coding problems means how to build logic for that please sir it's my humble request, please make video on that
@Amritanjali3 жыл бұрын
he already made one video on this check-in his channel
@samarthagrawal34153 жыл бұрын
@@Amritanjali ok thanks...
@liatris693 жыл бұрын
Data sharing shyness is real.
@tze-ven2 жыл бұрын
Do you know you don't need supercomputer to scale? You can scale with cloud computing. The best thing is that you don't need physical shopping. You just click and pay. 😜
@TheGenerationGapPodcast2 жыл бұрын
He was contrasting virtical scaling with horizontal scaling. Question: Would virtical scalingbe the same as horizontal if we rotate virtical scaling by 90°?
@mrrishiraj883 жыл бұрын
🙏🙏🙏
@imperfecto77343 жыл бұрын
13:38 "I wont explain everything here, I'm lazy too "
@manjuender62863 жыл бұрын
Super computer at super market Harvard vs Oxford 😂