What is Database Sharding?

Database sharding is a technique used to optimize database performance at scale. It relies on separating data into logical chunks so that they can be separated and queried independently. Learn more about sharding in this video.
Looking to get hands on experience building on AWS with a REAL project? Check out my course - The AWS Learning Accelerator! courses.beabetterdev.com/cour...
📚 MY RECOMMENDED READING LIST FOR SOFTWARE DEVELOPERS📚
Clean Code - amzn.to/37T7xdP
Clean Architecture - amzn.to/3sCEGCe
Head First Design Patterns - amzn.to/37WXAMy
Domain Driver Design - amzn.to/3aWSW2W
Code Complete - amzn.to/3ksQDrB
The Pragmatic Programmer - amzn.to/3uH4kaQ
Algorithms - amzn.to/3syvyP5
Working Effectively with Legacy Code - amzn.to/3kvMza7
Refactoring - amzn.to/3r6FQ8U
🎙 MY RECORDING EQUIPMENT 🎙
Shure SM58 Microphone - amzn.to/3r5Hrf9
Behringer UM2 Audio Interface - amzn.to/2MuEllM
XLR Cable - amzn.to/3uGyZFx
Acoustic Sound Absorbing Foam Panels - amzn.to/3ktIrY6
Desk Microphone Mount - amzn.to/3qXMVIO
Logitech C920s Webcam - amzn.to/303zGu9
Fujilm XS10 Camera - amzn.to/3uGa30E
Fujifilm XF 35mm F2 Lens - amzn.to/3rentPe
Neewer 2 Piece Studio Lights - amzn.to/3uyoa8p
💻 MY DESKTOP EQUIPMENT 💻
Dell 34 inch Ultrawide Monitor - amzn.to/2NJwph6
Autonomous ErgoChair 2 - bit.ly/2YzomEm
Autonomous SmartDesk 2 Standing Desk - bit.ly/2YzomEm
MX Master 3 Productivity Mouse - amzn.to/3aYwKVZ
Das Keyboard Prime 13 MX Brown Mechanical- amzn.to/3uH6VBF
Veikk A15 Drawing Tablet - amzn.to/3uBRWsN
🌎 Find me here:
Twitter - / beabetterdevv
Instagram - / beabetterdevv
Patreon - Donations help fund additional content - / beabetterdev
📚 References:
Great Sharding Article - aws.amazon.com/blogs/database...
#Databases
#Sharding
#SoftwareEngineer

[@RandomShowerThoughts]

honestly might be the most complete and thorough explanation of sharding.

[@BeABetterDev]

Thanks so much for your kind words!

[@hamadaparis3556]

You've simplified your explanation like google engineers do when they give lectures, I'm sorry if that sounds strange but I've realized that the people who simplify complex things they really know what they are doing awesome man Cheers.

[@BeABetterDev]

Thank you so much for the kind words!

[@mohamedshehata3979]

Nice words

[@Aidanhyland]

I am burning through all your videos. You are making me a better SAAS Test Engineer! Keep up this great work!

[@v.m.5850]

Watched countless videos and barely understood the concept. Your video on the other hand explained everything along with pros and cons super simply. Thanks a ton.

[@abhishekghosh5550]

This is seriously such a great video man. I spent the entire Sunday understanding Sharding. Not that I didn't get started with the concept, however, this video just made everything clear at the end of the day. Thank You.

[@yfzhangphonn]

Best lesson about database scalability I found, so easy to understand.

[@poloska9471]

Dude you make some really awesome content. Please please keep making videos! I love the clarity of your speech, voice, and presentation. I understand and can follow along in your videos a lot better than more other channels. Earned my subscription and likes! Keep killing it homie!

[@BeABetterDev]

Thank you so much for your kind words and welcome to the channel!

[@dannydatt]

Network guy trying to get an understanding in a different field. That's an outstanding walk-through and very much appreciated. Thank you for your work and quality presentation.

[@BeABetterDev]

Glad it was helpful!

[@bharat_arora]

Finally found some decent content over this topic. I already had an idea on this topic just wanted to revise it. Thanks a lot for making the insightful videos.

[@quangluu79]

Good stuff man. I love the clarity you bring to a subject. Subscribed.

[@wlcheng]

Great video! Such a clear explanation of how database sharding works.

[@Alexan6548]

Very clear. One of the best tutorial I have ever seen

[@ase713]

Dude, this was outstanding! Super helpful and covered everything I needed to know!

[@cd92606]

Great video, especially your description about the non-uniformity problem.

[@BeABetterDev]

Thanks Rotary Dialer! Yea the non-uniformity issue is one I've been personally bitten by in the past. Glad you enjoyed the video!

[@ihateorangecat]

Best of the best explantion I kinda start to understand the Sharding in just video.Thank You and I subbed!

[@mathisinav4267]

Hands down! the best explanation I've seen on database sharding, excellent!

[@BeABetterDev]

You're so welcome. Glad you enjoyed.

[@ChauDuong1982]

Thanks for the videos. Great explaination.

[@saiaussie]

Hey dude, you're a star! Very clear and upto the point! I cant thank you enough.

[@Bhaskarlnm]

Daniel, no words.. looking at your playlists content and videos …amazing. Great great effort to help people. Kudos to you 👏👏👌👌👌

[@BeABetterDev]

You're very welcome!

[@patrick1778]

you are so good at explaining concepts

[@harishbendale6818]

Very clear, and simple explanation.

[@BeABetterDev]

Glad it was helpful!

[@eternalnight9453]

New here. Loved your talk! Your presentation and teaching is elegant and simple. Really appreciate it, thank you!

[@BeABetterDev]

You are so welcome!

[@nodrift9503]

Perfect explanation. Thank you

[@codespace747]

Best video ever made on sharding

[@willemplug3366]

Super clear. Thank you!

[@filesopen6188]

this video entails very good explanation and this also entails complex understanding.

[@BeABetterDev]

Thank you!

[@antonrozhansky7254]

Great explanation, Daniel. Thank you

[@BeABetterDev]

You're very welcome Anton!

[@panggrayta]

woww...!! great videos, great presentation, great explanation. thank you, keep sharing..

[@OscarMartinez-nt6zn]

Great explanation!

[@lucasarbex926]

Great content man!! It helped me a lot!! Keep up with the good work!

[@BeABetterDev]

Thank you!

[@Lordnoashi]

Amazing explanation, loved it. Thank you, it will help for the future interviews I have.

[@BeABetterDev]

Glad I could help!

[@sharonleibel]

Great explanations! Thanks, Keep it coming!

[@BeABetterDev]

Thanks Sharon!

[@AnilKumar-lb3qf]

Excellent presentation, very good explanation 👍👍

[@user-zr4jl4ms3v]

best explanation of sharding i've heard!

[@BeABetterDev]

Thank you very much!

[@rjjlucy]

In most ~20min videos, I get tired soon and close them after 5min. I can’t believe your video is so good that I totally forgot time and finish watching all of it

[@BeABetterDev]

Thank you so much Jingyi! Its these kinds of comments that keep me motivated to make more content :) Stay safe Daniel

[@francoisbourdages2906]

très bien expliqué, merci

[@jackforcecity]

Great job. Very well explained!!!

[@BeABetterDev]

Thanks so much Jackson! Glad you enjoyed :)

[@taniaasim]

This is great and super clear. Thank you!

[@BeABetterDev]

You're very welcome

[@arikedada]

great video, I understand what idempotency operations entails, thank you

[@santoshlml]

Well explained. Thank you!!

[@BeABetterDev]

You're very welcome Santosh! Glad you enjoyed.

[@estebanquintana156]

Great explanation. Thank you

[@BeABetterDev]

You are welcome!

[@devdewboy]

Thanks for the straight forward easy to grasp concept of sharding. Give this to someone else and we would have gotten a bunch of technical wordy mumbo-jumbo.

[@shubhamchandra9258]

Awesome video. Very clearly explained.

[@BeABetterDev]

Thanks!

[@rajt1998]

Very well explained. Thank you

[@BeABetterDev]

You're very welcome!

[@jeevaengg]

good explanation!!

[@dorbenmoyal7639]

Great explanation..! Thanks 🙏🏼

[@BeABetterDev]

You're very welcome Dor!

[@duzgunkenan]

Great content, thank you

[@JamesQQuick]

This was awesome. Thanks!

[@RajuGupta-st1hj]

Thank you so much for the post. Good work. Keep it up.

[@BeABetterDev]

You're very welcome Raju!

[@JayPatel12928]

Watched some of your random videos on sys design, and now im hooked. Great content!

[@BeABetterDev]

Thanks so much J! Glad you enjoyed!

[@amrikmalhans]

@@BeABetterDev yes

[@rschmidtzalles]

clear and concise. subscribed

[@BeABetterDev]

Thanks Libert and welcome!

[@chandnisaini9176]

Well explained!!

[@BeABetterDev]

Thank you!

[@kishoretvk]

Great vidoe, keep up the good work

[@eugeniosp3]

Bro I'll watch anything you make. If you made a video teaching me how to watch paint dry I'd take notes. Keep up the damn good work my mans.

[@user-ws5xs1nv4p]

Thanks a lof for this video!

[@hpandeymail]

Very well formed content .. thanks 🙏

[@BeABetterDev]

You're very welcome!

[@kgcpk]

Superb explanation 😍

[@SofiaGoyal]

Really good work man... such a detailed video...

[@BeABetterDev]

Thanks Sofia! Glad you enjoyed :)

[@rayprusia4753]

Your videos are awesome! Thanks

[@BeABetterDev]

Thanks ray!

[@simonemariottini1011]

Really useful content! Keep it up!

[@BeABetterDev]

Thanks so much Simone!

[@itiscinnamoncafe]

Love longer videos ❤

[@milequinze]

Awesome! Thanks a lot!

[@BeABetterDev]

You're very welcome

[@tamaraamanda2483]

Prepping for Amazon TPM interview and this is so helpful!

[@BeABetterDev]

Thanks Tamara and good luck on your interview! Make sure you focus on those leadership principles !

[@user-tu1kf6xd8d]

great explanation thank u so much

[@BeABetterDev]

You are welcome!

[@sandrojorgeoliveira175]

Thank you, Dude!!!

[@mivel9763]

Had a hard time grasping on what database sharding actually meant but your video really helped me understand it, thanks! :)

[@BeABetterDev]

You're very welcome!

[@3042640426]

Thank you so much

[@trantrongty8065]

Thank you that really helpful great video

[@BeABetterDev]

You're very welcome Tran!

[@random-characters4162]

God bless you, sir ✌️

[@donaldkennedy7993]

superb explanation of DB scaling & sharding & W/R databases for a non DB person ;)

[@drummerboi4eva]

excellent video

[@markryan430]

OMG, I just Sharded myself!

[@andrewkicha1628]

Great job on this one, I came here to know more about sharding, but I learned lots of useful information before you even dived into the topic ;)

[@BeABetterDev]

Glad it was helpful!

[@samlinsell900]

Vids are awesome, really enjoy them. Interesting that you didn't touch on the lack of thought to database design, indexing and maintenance etc as a way to improve performance. Interested to know why? Especially given the cost of scaling in serverless environments.

[@rishiraj2548]

Thanks

[@gulsimsek7988]

thank you!

[@shivakumarranjithn5584]

Great content!

[@BeABetterDev]

Thanks so much Shivakumar!

[@vikasravindran1312]

Excellent !

[@BeABetterDev]

Thank you so much Vikas!

[@geekaffairs6475]

very well explained

[@BeABetterDev]

Thanks so much!

[@royal4210]

Thank you programmer charlie

[@skmahaboobbasha6059]

Great vedio please make vedio on opsmanager installation on production environment

[@fethicetin9186]

Thanks for the video

[@BeABetterDev]

You're very welcome Fethi!

[@drew4980]

Are there any database tools that make this easier? Couldn't someone write some software to create a wrapper around a sharded DBMS that could handle the routing and re-sharding with a given hashing key?

[@hualiang2182]

Nice tutorial. Wonder in real word scenairo, is the routing layer something sits in the application code or it's implemented on the database side?

[@swaroopas5207]

Great video! But how do we handle foreign keys in sharding?

[@peterroger249]

Much thank you for your great KZbin help. I am new to Excel and Chatbot. How can I migrate the Excel database, export it from Microsoft Azure WebApp, and import it into AWS Chabot? Keep having errors missing QID and others on the AWS Chabot console. Please help show me the fastest way to convert the Excel and make it compatible with AWS Chatbot?

[@ibrahimkoz1983]

You're great.

[@BeABetterDev]

Thank you!!!

[@subhasishhalder4817]

How come I didn't find your channel before?

[@alexeykorovko6704]

very good explanation, thank you one point is not clear - do we really have advantage of availability / fault tolerance, in case we have an intermediate layer that routes the requests? for me it is like the same, isn't it?

[@mohamedyoussef8835]

Awesome video +++++++++++ 🙂

[@HeavensMeat]

I know you have had other dynamodb videos here but would it be possible to have a more in depth video dealing with sharding in dynamodb and also utilizing this with python/boto3 vs the cli? I know it's not really the same type of sharding per se but this video reminded me that I am interested in seeing that kind of thing

[@BeABetterDev]

Hey HeavensMeat! You're suggestion is a great idea for a new video idea, thanks you! I'll work on incorporating this into my todo list. Cheers!

[@r-rtz]

A more interesting concept though is how you generate these unique id's that are used in the sharding / partitioning and ensure uniqueness

[@DuyTran-ss4lu]

Awesome

@BeABetterDev What if I were to opt for synchronous replication for my read replicas? Wouldn't that provide me with a high level of consistency (strong consistency) between the master node and the replica nodes? Besides, AWS RDS provides async replication for read replicas, does that mean it is eventual consistent? If so, if I am building an application that needs to opt in strong consistency, shouldn't I use AWS RDS read replicas then? What would be an alternative option to that?

[@loaizar95]

amazing video!! Understood almost everything and am not a it guy.. the only thing I did not get is the difference between partition mapping and routing :(

[@dhriajbhandari]

Great video. Thank you. I just have a question about routing for the determining the shards. Is it always necessary? I was thinking that you could just do modulus on the id to get the shard number instead (eg: customer_id: 12, num_of_shards = 4 so the shard would be 12 % 4 = 0). That way you don't have a single point of failure on the router. What are the downsides to this approach vs router end-point ?

[@BeABetterDev]

Hi there this is a great point thanks for sharing. The problem with using modulus is that it can get difficult to change the assignment of data to shard if you need to re-shuffle your data. With a single table acting as the authority, this can be done trivially.

[@poketopa1234]

What I always miss in these videos is, doesn’t introducing a routing layer just kick the can down the road? Now you have all traffic going to a singular routing node, which is not scalable and can fail. What happens when you need to scale the routing node?

[@cyclomiha]

Hmm..how about PITR? For analytics you could have replica with multi-master approach to each shard, right?

[@benhunt2023]

This feels like the khan academy of coding. Well done.

[@yna8588]

Can we scale up and scale down the storage of database as per daily requirement using sharding?