6. Consistent Hashing in Hindi with Example | System Design - Consistent Hashing | High Level design

[@uraveendra]

have refered many videos on consistent hashing but this video cleared all my doubts and crisp and clear content, well narrated 👏

[@ConceptandCoding]

thanks

[@uraveendra]

@@ConceptandCoding I have a rudimentary question here. Consistent hashing will be used for scaling up and scaling down systems right ? for any unexpected server fail over, we need to relay the redundant systems running at separate server/cluster right ?

[@tanazshaik678]

Hands down the BEST video on consistent hashing, am glad I found your channel.

[@rajanjain1156]

Bhai mne bhut vidoes dekhe muzhe ye best lagi, full detail and simple explanation , ab mn soch raha hun LLD and HLD apne jo playlist banye hai uske complete kar leta hun

[@suchitrasrinivas4330]

Thanks for making consistent hashing so easy to understand, especially where you explain about Virtual Nodes.👍👍

[@ConceptandCoding]

Glad it was helpful!

[@BrijeshYadav-xu5kj]

Recently i started watching your videos, they are very helpful, thanks for making these videos ,

[@ConceptandCoding]

thank you

[@SatyamKumar-bw4vi]

Hare Krishna..! Bhaiye Great work

[@ConceptandCoding]

Radhe radhe. Thank you

[@Rosh__138]

Time bacha liya yr ak video sa samj aa gaya wrna 10 12 dakh ka bi ni atti hoti Explained very nicely and briefly 👍👍

[@rohitpv7676]

Best video on Consistent hashing

[@KumarSahil78]

First time watching you, and it's really great experience, your way to teaching is just incredible 🔥.

[@ConceptandCoding]

thank you

[@sagarbiradar1104]

Very very Clear explanation🔥🔥

[@ConceptandCoding]

Glad it was helpful!

[@nabajyotimajumdar4511]

best video on consistent hashing

[@Engineer_With_A_Life]

Amazing content, crystal clear explanation.

[@ConceptandCoding]

Thanks

[@Engineer_With_A_Life]

@@ConceptandCoding One small question, I too have an iPad but looking for good tool to make notes and diagrams. Could you tell which software you used in this video to present these diagrams.

[@ConceptandCoding]

@@Engineer_With_A_Life one note

[@chillaxabhinav]

Easy to understand, Thanks

[@shubhamagarwal1434]

Very well explained...awsome!!!

[@anonymous34565]

Great explanation. Pls also make some mock interview on machine coding round.

[@ConceptandCoding]

sure

[@rishigaumat]

in the app server point of view, we can think of the internal cache(which each server will be maintaining for similar request, and the request will route to the new server when the mod changes). the throughput of the application server will be more when the mod changes and the cache will be wasted. this is the problem with the normal hashing.

[@___vandanagupta___]

Great explanation sir!! Keep up the good work!!!

[@ConceptandCoding]

Thanks a lot!

[@ankitsh6557]

In case of virtual objects, when a server is taken down. All the associated virtual objects will also be removed from the ring. In this case the rebalancing will be high.

[@AmitSharma-yb9vc]

Actually, not; if there are n keys and m servers, then on average n/m keys are associated with the server; and virtual server help to go close to this average in uneven distribution.

[@LegitGamer2345]

When the main server is taken down we can choose some virtual server as the main one based on some algorithm since the virtual server is a replica of the main server, what do u guys think?

[@girishanker3796]

The virtual servers are taken out from different locations in the ring. The concept of consistent hashing with virtual servers is to distribute even load to all the servers.

[@JavaWithAFlow]

Thanks, for this crisp explanation . I

[@vikramkumar-mi1yk]

Excellent explanation... I have one question like while making replicas for virtual objects are we randomly assigning it or are we using the same mod hash function?

[@RahulPatel-lq6qp]

would like to see live demo of this to understand it better

[@collab_with_tushar_raj]

@ConceptandCoding Thank you for touching upon the "servers being equidistant" and "servers being next to each other" parts in the ring. I have a very lame question. When we create virtual servers, to solve the issue with the servers being next to each other, is there any logic we follow, like a math function that generates random points in the circle? How do we know that we have achieved 1/n% of rebalancing in this case? Can you give one small use case where we can understand how to calculate the 1/n% of rebalancing after creating virtual points? Like I said, apologies for my lame question. I would really appreciate it if you cleared this doubt. Thank you in advance!

[@rahulsharma15]

Thanks for your efforts buddy.keep up the good up work like these.

[@ConceptandCoding]

thank you

[@nikiteshnandale8264]

instead of randomly assigning positions, we can implement an algorithm, which will assign positions to all the servers , such that all the servers will be equidistant from each other. If all servers will be equidistant from each other Like they were in the early examples of Consistent hashing, then I think our purpose of implementing consistent hashing will be fulfilled. feedbacks are welcomed

[@ConceptandCoding]

Right

[@KumarSaurabh-nu9cr]

If you plan to keep all the servers equidistant from each other, then an addition or a removal of a server will lead to movement of servers on the ring which would lead to more rebalancing. No?

[@Nerddog12344]

​@KumarSaurabh-nu9cr Correct

[@tanazshaik678]

@@KumarSaurabh-nu9cr right

[@saurabh3755]

nice video

[@yashsingh3819]

aapki videos bohot helpful rehti h. accenture innovation chalenge pr bhi ek video bana dijiye please. Hame thodi help ho jaegi

[@prakharjalan7546]

Will we be able to just rebalance 1/n keys even if a server which is replicated thrice goes down?(In virtual objects) and what if the number of servers exceeds the size of the virtual ring?

[@310gowthamsagar5]

great video.

[@bishwajeetbasudevdas3157]

Nice explanation

[@ConceptandCoding]

Thanks

[@nikhilkumar-jy6mz]

For HLD interview do they ask us to write the code or just to explain the concepts ? because I never saw any youtuber explaining using code

[@ConceptandCoding]

foe HLD no coding is required only concepts and for LLD you need coding

[@chessfolks3827]

Absolutely worth it. Can you also share the reference or book where you prepare the material?

[@ConceptandCoding]

Generally i read, "Design data intensive application" But this topic i read from DynamoDB research paper, as DynamoDB also uses Consistent Hashing, so they explained it in detailed.

[@Tanzim-ho7nf]

Owo❤. Thanks 👍

[@SanyamJain-l1k]

Video 7 is not available. Why?

[@pkmkb7138]

For this example you have taken a queue or ring of size 12 for 3 servers so how did you come to the no 12, is there any relation,

[@sohammahajan3483]

I think the ring size of 12 for 3 servers in the video is just a simplification to make it easier to visualise/teach how consistent hashing works

[@ankitgupta-ph4nk]

Great Explanation🎉

[@ConceptandCoding]

Thank you

[@mehulparmar9976]

If hashing function provided to partition key makes node assigned random. Then why it is advised to choose a partition key as random as possible

[@shrad6611]

just one thing sir, 1/n is not 1 raise to the power n , its 1 divided by n. It creates so much confusion in the video.We say raise to the power when 1^n

[@ConceptandCoding]

thanks for correcting and sorry foe the confusion buddy

[@shrad6611]

@@ConceptandCoding btw best videos brother

[@piyushmathpal4244]

We are designing insta and sharding(horizontally) based on USERID, definitely we will fall into the problem of hot users and ovio shard will grow big as ronaldo posts a lot. How do we handle such case ? Note: Sharding by POSTID is very oldschool. Querying db for getting ALL post of a user will be an expensive operation which seems one of a primary ask from our system, here.

[@ConceptandCoding]

Thats why Consistent Hashing technique is used, as sharding by postid is old-school :) Whatever problem you have mentioned it can be resolved using consistent Hashing itself.

[@piyushmathpal4244]

@@ConceptandCoding Can you please elaborate how? Sharding based on userID will put one of the hot users in a single shard. If that goes down, user's data goes down. How does consistent hashing solves our issue here? It will be used to allocate a userID to a specific server only. Correct me if I'm wrong. Even if we have virtual replicas, still user data will go to a single shard. How do we develop a robust solution here?

[@rupeshjha4717]

@@piyushmathpal4244 i think the hash key of diff post id wont go to the same server, it will be distributed over circle and can be handled accordingly by diff nodes

[@SaumyaPrakash-q2w]

Hey Shrayansh , I have a question .Virtual servers are created at the start or when the keys are overloaded to one server?

[@ConceptandCoding]

In consistent hashing, virtual nodes are created at the start to evenly distribute keys across physical servers and to prevent overloading a single server.

[@gurupreetsingh8347]

when we talk about to replicate the servers of virtual objects , are they actual physical replication of machine or just virtual replication ? and whatever it is but how we come up with the location on the Ring to replicate them ?(virtual object) because we already used mod hashing to place the actual servers on the ring, and what about hash colison while calcuating hash values for Servers, ? two or more servers can have same hash-code so how to handle it!!?

[@DeepakYadav-fh9tz]

Really good questions, But since I have seen consistent hashing at multiple places here are the answers: 1. It is just a virtual replication not the actual replication. We could do it by having multiple hash functions per server. The hashing functions should be in a way that satisfies the properties of keeping them in a ring. 2. hash collisions interesting point, but since we are just assigning the load, suppose k1 and kn colloide they simply move to the same server.

[@1997uday]

I have a question, we are figuring out the initial placement (lets say each DB server) using a hash function. After that, we place the replica of those db servers (virtual servers) to balance the storage of any keys (data). Now, if we get a read request, even after doing a hash (and mod) of it, we will have to know the locations where the virtual servers are placed as well to know which server that read request should be mapped to. Will we store the mapping of virtual db server and its location somewhere?

[@meetakbari9897]

I think the mapping of virtual db and its location can be stored in a centralized metadata store, such as a distributed key-value store or a metadata database. So when a read request comes in, you would first hash the key to determine the initial server where the data is located. Then, you would consult the mapping of virtual db to identify the actual db where the data replica resides. Finally, you route the read request to that server to fetch the data.

[@logicalrisi]

thanks....bhai notes bhi dediya kro

[@ishaankaustav727]

how are we deciding the size of queue in virtual ring ? suppose we have flood of request with key (k1) , in this case all request will go at server 1 only , so how does consistent hashing is solving the problem ?

[@ConceptandCoding]

Ack of the question

[@harshagarwal_net]

that means your hash function is not proper. or may be you not choose right colume for hashing. because even distribution is responsibility of hash function.

[@tarunrawat1281]

Can you share LLD for same ?

[@piyushpathak1186]

how these things are implemented in code level Now I want to see that

[@subhamacharya4472]

Why we are taking 12 sectors in the virtual ring ?

[@ConceptandCoding]

Hi Shubham, i just took an example, generally its a large number not 12.

[@subhamacharya4472]

@@ConceptandCoding how to decide, how many we need to consider for the virtual ring?

[@ConceptandCoding]

@@subhamacharya4472 Shubham it depend upon the company needs.

[@dipanshumewara6116]

And how can you decide ring sectors as requests can be in millions and ever increasing

[@Abcnews18]

In our example we have taken the virtual ring size of 12, but how do we decide that number? Is it the maximum number of nodes possible we assumed? Could you please tell us.

[@ConceptandCoding]

It's just a example i took may be in reality for big companies it might be in 100a or thousands

[@Abcnews18]

@@ConceptandCoding thank you so much 🙏

[@FazilKhan-vr6sw]

Hi, what would happen of the no if servers reached to the size of virtual ring .?

[@ConceptandCoding]

Hi Fazil, as per my understanding, the hash space(total no of nodes it can have) for Consistent hashing is Huge (generally in 2^256) so scale up issue do not come here.

[@FazilKhan-vr6sw]

@@ConceptandCoding Thanks a lot for clarifying.

[@himanshukaushal8375]

If we have 100 keys and 5 nodes then how many rebalancing nodes are needed in total? I didn't understood that point.

[@ConceptandCoding]

ACK of the question,will reply by tomm.

[@sushil1922]

Suppose S3 is getting request of 90 keys. Then if we remove s3. Then all the requests going to s3 needs rebalancing similar to normal hashing. Isn’t it?

[@sushil1922]

Only benefit of consistent hashing is even distribution at particular time using virtual locations of every servers? And not rebalancing ?

[@rv0_0]

I joined membership yet i am not able to access this content. Please help.

[@me57shreyashjadhav37]

Suppose there are three payment microservice. Each has its own DB. Now if user1 comes with request. His ID will be unique and he will be assigned to any particular microservice again and again everytime . Until new server of microservice is been introduced or existing microservice is removed (according to consistent hashing). Now my question is, if user1 wants to do payment . He did payment and his payment status will be stored in DB of that particular payment microservice. Now by chance if that DB fails or service fails after the transaction is done. Now here how is this issue handled.

[@vineet4991]

There are always db replicas. Master slave will be there. Also there’s something called Write ahead logging. All these can be applicable here.

[@amitbansal7754]

Bhai 1 raise to n nahi 1 by n number hoga vo

[@lakshyachhabra7837510764]

Hi i have one doubt so let’s suppose if two servers not replicas point to same hash key which means let’s suppose you have 3 servers and 2 of them points to same hash id which means 2 Then how and which server will take that request??

[@Engineer_With_A_Life]

You are saying two servers will have same hash key, in general this should not happen, because we always try to handle hash collision in these designs. But still there are some chances that servers can have same I'd, in this case you can consider other parameters to route the request like priority, session count, etc. (Depends on the use case, but this is very very rare)

[@harshitagarwal2682]

👍👍

[@biggbali]

How to know total no. of keys ?

[@priyadarshinir9730]

Great Explanation! i have few doubts... let's say a key is requested repeatedly and that key lies is one of the replica's, so request goes to the same replica everytime. how's consistent hashing helping here? aren't we putting heavy load on the same replica every time? how's the traffic equally distributed in this case?

[@amituchat4865]

(1) A consistent hashing helps even distribution of unique keys across all servers. (2) A single key request is designed to hit one unique server where the value resides, so if one request arrives repeately for million times then by design it should hit the same server million times. (3) Bigger question is what situation made this usecase possible? and (4) Do you think caching those request/respose could help in this usecase?

[@piyushtale0001]

@@amituchat4865 Or we can add a Rate Limiter

[@mayursamnani8964]

What does hashing the server signify at 18:48?

[@girishanker3796]

Passing the name or ip address of the sever through a hash function

[@gopichinthagumpula6481]

What if I don't know hindi.. And want to learn the content you are creating

[@amituchat4865]

Subtitles??

[@irbansin]

There are courses in other languages too. If there is none, create one. It will increase diversity of that language

[@parthsanghavi6019]

too many ads in the lecture, I got frustrated watching ads in finishing a single lecture. Sorry to say but this needs to be re thought

[@KumarSaurabh-nu9cr]

1 raise to power n keys nhi ho rha rebalance.. Please yaar, jo likh rhe ho woh toh bolo kam se kam

[@TsarVladolfPutler]

Comrade Shreyansh, you have been drafted. Report to your nearest conscription office where you will be tasked with building scalable military-grade systems which would be invulnerable to Ukrainian and NATO cyberattacks

[@Ashish-ty5rb]

Military mainly uses real time systems I don't think scaling them largely is super important untill unless you are trying to take over whole world at once 💀💀

[@ajaysabarish9645]

Please speak in english or atleast add english subtitles

[@ConceptandCoding]

Hi Ajay, i used mixed language, it would be easily understand

[@ajaysabarish9645]

@@ConceptandCodingAs a non hindi person, any Hindi word goes above my head,making it impossible to comphrehend the lecture. Can you please at least add english subtitles ?

[@amituchat4865]

@@ajaysabarish9645 I understand your concern, if you do not have option like subtitles etc then just listen attentively, most hindi words used are filler words (one time learning, by making a list) but the key words vocab are always english. In the quest of making hindi words list, you will end up listening attentively and hopefully get a crystal clear understanding of a topic as a biproduct. Hope it helps

[@shwetankgupta07]

Not able to join telegram group via given link. Please help @Concepts && Coding

[@ConceptandCoding]

t.me/+KwJhmit5yDczNzY1 Pls try this

[@mridul1075]

Bhaiya hld ka members only free krdo 🥲

[@shankysays]

Hi shrayansh, i needed some preparation guidance from you. Can i connect to you over LinkedIn or Facebook whichever is comfortable for you?

[@ConceptandCoding]

yes linkedin i am good.

[@shankysays]

I sent you connect request. Please accept it so that i can message and get some valuable guidance from you. My name is shashank Kumar

[@shankysays]

My name is shashank Kumar