System Design - Sharding | Data Partitioning

Sharding or Data Partitioning:
- Sharding is a technique to break up a big database into many smaller parts
- Horizontal scaling means adding more machines, which is cheaper & more feasible
- Vertical scaling means improving servers
- Partitioning Methods:
- Horizontal partitioning:
- In this we put different rows into different table(db), i.e. rows can be based on location with zip codes, this is also range based sharding
- Problem is if range is not choosen carefully, it'll lead to unbalanced servers.
- Vertical Partitioning:
- In this we divide data to store tables related to a specfic feature to their own servers.i.e. In Instagram, we can have 1 for user profile, 1 for photos, 1 for friend list.
- Problem is in case of additional growth, its necessary to further partition a feature specific DB across servers.
- Directory based Partitioning:
- In this, create a lookup service which knows your current partitioning scheme & get it from DB access code.
- To find out a data entry, we query directory server that holds the mapping between each tuple key to its DB server.
- This is good to perform tasks like adding servers to the DB pool or change our partitioning scheme without impacting application.
- Partitioning Criteria:
- Hash-based partitioning:
- In this we apply a hash function to some key attribute of the entity we're storing, that gives partition number.
- Make sure to ensure uniform allocation of data among servers
- Problem is it effectively fixes total number of DB servers, since adding a new server means changing the hash function, which would require redistribution of data & downtime, the workaround is Consistent Hashing.
- List Partitioning:
- In this each partition (DB server) is assigned a list of values, i.e. APAC, EMEA, US region has respective partition.
- Round-robin Partitioning:
- One by one assign which ensure uniform data distribution
- Composite partitioning:
- Combining any of above partitioning schemas to devise a new scheme. i.e First list partitioning with hash partitioning in each.
- Common problems with Sharding:
- Joins & Denormalisation:
- Performing joins on a database which is running on one server is straightforward, but if DB is partitioned & spread across multiple machines, it's not feasible to perform joins that span database shards.
- These joins won't be performance efficient.
- Workaround is to denormalise the DB so that queries that required joins can be performed on single table.
- Referential integrity:
- Trying to enforce data integrity constraint suh as foreign keys in a sharded DB can be extremely difficult. Most of RDMS do not support this.
- Rebalancing:
- When data distribution is not uniform.
- When there is lot of load on a particular Shard.
In this video, we're going to reveal Sharding used in System Design:
- What is Sharding/ Data Partitioning
- Sharding Methods
- Sharding Criteria
- Sharding Challenges
★☆★ SUBSCRIBE TO ME ON KZbin: ★☆★
kzbin.info
what is sharding in mongodb,
database sharding example
sharding vs partitioning,
sharding mysql,
sharding techniques,
consistent hashing sharding,
sharding crypto,
what are sharding criteria
CHECK OUT CODING SIMPLIFIED
kzbin.info
I started my KZbin channel, Coding Simplified, during Dec of 2015.
Since then, I've published over 200+ videos. My account is Partner Verified and I get my earnings direct deposited into my account every month.
★☆★ VIEW THE BLOG POST: ★☆★
thecodingsimplified.com
★☆★ SEND EMAIL At: ★☆★
Email: [e-mail]

[@lakeshed]

Great Video, lucid, well explained and structured... Thank you

[@CodingSimplified]

Thanks. Keep watching.

[@vinaykumar15]

Great Video Sir, Its very helpful to learn about the techniques and how we can implement in different scenarios. I have a question on Sharding drawback while applying joins on the tables in SQL db which resides in different regions- how can we tackle this problem and how should we approach this? Thanks.

[@CodingSimplified]

Hi, thanks for liking the video. Generally, if we want to apply sharding on SQL db, we do de-normalization, which means that we copy those tables on other servers which are involved in Join queries. Doing this will make sure that all join queries are happening on same server. But this will lead to data duplication on several servers, which is not good and as well on real world scenario not feasible to copy on every join query. Let me know if you still have doubts

[@NaturalPro100]

Good and clear explanation

[@CodingSimplified]

Thanks. Keep watching.

[@NaturalPro100]

@@CodingSimplified Can you provide a link which explains this practically ?

[@AbhishekKumar-hi8oj]

nice explanation.

[@CodingSimplified]

Thanks. Keep Watching.

[@AbhishekKumar-hi8oj]

Yes, nice collection, just saw your playlist, I will go through all.

[@AakashGoplani]

Nice Video. Suggestion - Try making videos more interactive like drawing on whiteboard or having animations instead of going through PPTs.

[@CodingSimplified]

Thanks for suggestions. Sure, will improve on this part.

[@CodingSimplified]

Thanks Aakash for suggesting. After your comment, I thought about it. To improve video quality, now I've purchased Drawing tablet, which will be helpful in explaining diagrams. In recent videos I used it & found extremely helpful in explaining. Thanks again for constructive feedback.

[@2dapoint424]

Is partitioning and sharding the same??

[@sherrylfernandes8393]

Can u make a video of hash partitioning

[@CodingSimplified]

Sure, I'll make about it. Thanks for suggestion. By the way, Hash partitioning is just another way of sharding, where we shard based on hashcode of key, where key is any attribute through which we want to distribute data.

[@vivekchaudhary1768]

so, sharding means to divide the database or the tables in it.

[@CodingSimplified]

We need sharding when our database is that big that it can't accommodate in 1 server; it requires more servers to fit in. So we shard the database, now how database will be shard, according to that tables of database will be distributed in servers.

[@rachna66644]

Hello Sir, Your videos are just what I needed to prepare for interviews. May I ask one help, Can you provide me any reference for the books where I can read more about these topics. I need basic introduction to designing and coding these. Please help. I have been searching but it hasn't been fruitful.

[@CodingSimplified]

Even I could not find any System Design related books. All info i gathered from different sources & captured all together in videos. Being said that, if you see any issue in understanding any topic, you can ask me by posting comments. Thanks for showing interest in channel.

[@sreenivasshenoy5208]

Check out Designing Data-Intensive Applications

[@MrDishajain]

You did not explain how ACID compliance is not met. You gave hardly a minute on it.

[@AbhishekBalani93]

He did that for the most part of the video. Just narrating the bullet points. And using half a sentence to explain those points. No a series of good videos.