What is Database Sharding?

[@RandomShowerThoughts]

honestly might be the most complete and thorough explanation of sharding.

[@BeABetterDev]

Thanks so much for your kind words!

[@hamadaparis3556]

You've simplified your explanation like google engineers do when they give lectures, I'm sorry if that sounds strange but I've realized that the people who simplify complex things they really know what they are doing awesome man Cheers.

[@BeABetterDev]

Thank you so much for the kind words!

[@mohamedshehata3979]

Nice words

[@cameronmccoy2204]

😊😅😊😊pp

[@cameronmccoy2204]

@@BeABetterDevpppp

[@cameronmccoy2204]

@@BeABetterDevp Pop p p pppp

[@dannydatt]

Network guy trying to get an understanding in a different field. That's an outstanding walk-through and very much appreciated. Thank you for your work and quality presentation.

[@BeABetterDev]

Glad it was helpful!

[@Aidanhyland]

I am burning through all your videos. You are making me a better SAAS Test Engineer! Keep up this great work!

[@v.m.5850]

Watched countless videos and barely understood the concept. Your video on the other hand explained everything along with pros and cons super simply. Thanks a ton.

[@sn-wg9gp]

Been studying system design for interviews. All the videos handwave to sharding. We would shard the db across different regions. I had rough idea what it is that we split the db in to smaller pieces, but nothing concrete. Now it make perfect sense with this amazing video

[@abhishekghosh5550]

This is seriously such a great video man. I spent the entire Sunday understanding Sharding. Not that I didn't get started with the concept, however, this video just made everything clear at the end of the day. Thank You.

[@yfzhangphonn]

Best lesson about database scalability I found, so easy to understand.

[@rjjlucy]

In most ~20min videos, I get tired soon and close them after 5min. I can’t believe your video is so good that I totally forgot time and finish watching all of it

[@BeABetterDev]

Thank you so much Jingyi! Its these kinds of comments that keep me motivated to make more content :) Stay safe Daniel

[@ase713]

Dude, this was outstanding! Super helpful and covered everything I needed to know!

[@cd92606]

Great video, especially your description about the non-uniformity problem.

[@BeABetterDev]

Thanks Rotary Dialer! Yea the non-uniformity issue is one I've been personally bitten by in the past. Glad you enjoyed the video!

[@codespace747]

Best video ever made on sharding

[@filesopen6188]

this video entails very good explanation and this also entails complex understanding.

[@BeABetterDev]

Thank you!

[@bharat_arora]

Finally found some decent content over this topic. I already had an idea on this topic just wanted to revise it. Thanks a lot for making the insightful videos.

[@devdewboy]

Thanks for the straight forward easy to grasp concept of sharding. Give this to someone else and we would have gotten a bunch of technical wordy mumbo-jumbo.

[@Alexan6548]

Very clear. One of the best tutorial I have ever seen

[@mathisinav4267]

Hands down! the best explanation I've seen on database sharding, excellent!

[@BeABetterDev]

You're so welcome. Glad you enjoyed.

[@MohammedMubashshir-q8v]

Awesome explanation of sharding, one of the best videos out there. Thanks brother!

[@ВладимирЛапенков-г1э]

best explanation of sharding i've heard!

[@BeABetterDev]

Thank you very much!

[@eugeniosp3]

Bro I'll watch anything you make. If you made a video teaching me how to watch paint dry I'd take notes. Keep up the damn good work my mans.

[@Bhaskarlnm]

Daniel, no words.. looking at your playlists content and videos …amazing. Great great effort to help people. Kudos to you 👏👏👌👌👌

[@BeABetterDev]

You're very welcome!

[@saiaussie]

Hey dude, you're a star! Very clear and upto the point! I cant thank you enough.

[@patrick1778]

you are so good at explaining concepts

[@wlcheng]

Great video! Such a clear explanation of how database sharding works.

[@harishbendale6818]

Very clear, and simple explanation.

[@BeABetterDev]

Glad it was helpful!

[@Lordnoashi]

Amazing explanation, loved it. Thank you, it will help for the future interviews I have.

[@BeABetterDev]

Glad I could help!

[@poloska9471]

Dude you make some really awesome content. Please please keep making videos! I love the clarity of your speech, voice, and presentation. I understand and can follow along in your videos a lot better than more other channels. Earned my subscription and likes! Keep killing it homie!

[@BeABetterDev]

Thank you so much for your kind words and welcome to the channel!

[@tamaraamanda2483]

Prepping for Amazon TPM interview and this is so helpful!

[@BeABetterDev]

Thanks Tamara and good luck on your interview! Make sure you focus on those leadership principles !

[@ChauDuong1982]

Thanks for the videos. Great explaination.

[@mivel9763]

Had a hard time grasping on what database sharding actually meant but your video really helped me understand it, thanks! :)

[@BeABetterDev]

You're very welcome!

[@JayPatel12928]

Watched some of your random videos on sys design, and now im hooked. Great content!

[@BeABetterDev]

Thanks so much J! Glad you enjoyed!

[@amrikmalhans]

@@BeABetterDev yes

[@arikedada]

great video, I understand what idempotency operations entails, thank you

[@donaldkennedy7993]

superb explanation of DB scaling & sharding & W/R databases for a non DB person ;)

[@Anton_Rozhanskii]

Great explanation, Daniel. Thank you

[@BeABetterDev]

You're very welcome Anton!

[@IQUE928]

incredible explanation, thank you!

[@RajuGupta-st1hj]

Thank you so much for the post. Good work. Keep it up.

[@BeABetterDev]

You're very welcome Raju!

[@sharonleibel]

Great explanations! Thanks, Keep it coming!

[@BeABetterDev]

Thanks Sharon!

[@eternalnight9453]

New here. Loved your talk! Your presentation and teaching is elegant and simple. Really appreciate it, thank you!

[@BeABetterDev]

You are so welcome!

[@lucasarbex926]

Great content man!! It helped me a lot!! Keep up with the good work!

[@BeABetterDev]

Thank you!

[@rajt1998]

Very well explained. Thank you

[@BeABetterDev]

You're very welcome!

[@bambooyu5960]

Thank you so much for the great explanation

[@quang.luu.179]

Good stuff man. I love the clarity you bring to a subject. Subscribed.

[@subhasishhalder4817]

How come I didn't find your channel before?

[@JamesQQuick]

This was awesome. Thanks!

[@taniaasim]

This is great and super clear. Thank you!

[@BeABetterDev]

You're very welcome

[@jackforcecity]

Great job. Very well explained!!!

[@BeABetterDev]

Thanks so much Jackson! Glad you enjoyed :)

[@andrewkicha1628]

Great job on this one, I came here to know more about sharding, but I learned lots of useful information before you even dived into the topic ;)

[@BeABetterDev]

Glad it was helpful!

[@rschmidtzalles]

clear and concise. subscribed

[@BeABetterDev]

Thanks Libert and welcome!

[@saifmohamed1776]

which better to start with for database basics: - introduction to database systems c.j date . - database internals. // if there are any better or recommended books or materials pls mention. * Great explanation.

[@BeABetterDev]

Hi Saif, This is a tough question to answer. I would step back for a moment to ask why are you trying to learn about databases? I think the answer will guide how/what to tackle first. For example, if you're just planning on using dbs, the database internals may be a bit overkill (but good to know overall). Could you tell me more about why you're learning db's and maybe I can guide you more? Thanks, Daniel

[@saifmohamed1776]

@@BeABetterDev to be aware of the basics in general like concepts physical logical at first And in backend specific. I'm very grateful for your concern

[@BeABetterDev]

Hi Saif, I briefly looked at the two resources you mentioned, I think a better choice is to read Database Internals. I feel that it is much more modern and covers some of the important aspects of database challenges today such as distributed systems and availability. The other book is quite dated and although I'm sure would be beneficial, I think things have changed so rapidly recently that I'm concerned the content will be a bit stale. One thing to note is to not get too bogged down with the details. To be a great developer with database understanding you don't always need to understand the low level details. Knowing how things work at a high level with the ability to dive deep when you need to is much more valuable. Hope this insight helps and I wish you best of luck on your studies. Daniel

[@saifmohamed1776]

@@BeABetterDev thank you

[@santoshlml]

Well explained. Thank you!!

[@BeABetterDev]

You're very welcome Santosh! Glad you enjoyed.

[@nodrift9503]

Perfect explanation. Thank you

[@willemplug3366]

Super clear. Thank you!

[@drew4980]

Are there any database tools that make this easier? Couldn't someone write some software to create a wrapper around a sharded DBMS that could handle the routing and re-sharding with a given hashing key?

[@SofiaGoyal]

Really good work man... such a detailed video...

[@BeABetterDev]

Thanks Sofia! Glad you enjoyed :)

[@chandnisaini9176]

Well explained!!

[@BeABetterDev]

Thank you!

[@hpandeymail]

Very well formed content .. thanks 🙏

[@BeABetterDev]

You're very welcome!

[@peterroger249]

Much thank you for your great KZbin help. I am new to Excel and Chatbot. How can I migrate the Excel database, export it from Microsoft Azure WebApp, and import it into AWS Chabot? Keep having errors missing QID and others on the AWS Chabot console. Please help show me the fastest way to convert the Excel and make it compatible with AWS Chatbot?

[@estebanquintana156]

Great explanation. Thank you

[@BeABetterDev]

You are welcome!

[@panggrayta]

woww...!! great videos, great presentation, great explanation. thank you, keep sharing..

[@AnilKumar-lb3qf]

Excellent presentation, very good explanation 👍👍

[@MegganCurrell]

Great analysis, thank you! Could you help me with something unrelated: My OKX wallet holds some USDT, and I have the seed phrase. (alarm fetch churn bridge exercise tape speak race clerk couch crater letter). How should I go about transferring them to Binance?

[@r-rtz]

A more interesting concept though is how you generate these unique id's that are used in the sharding / partitioning and ensure uniqueness

[@benhunt2023]

This feels like the khan academy of coding. Well done.

[@bajtre]

Great explanation!

[@BeABetterDev]

Glad it was helpful!

[@samlinsell900]

Vids are awesome, really enjoy them. Interesting that you didn't touch on the lack of thought to database design, indexing and maintenance etc as a way to improve performance. Interested to know why? Especially given the cost of scaling in serverless environments.

[@markryan430]

OMG, I just Sharded myself!

[@simonemariottini1011]

Really useful content! Keep it up!

[@BeABetterDev]

Thanks so much Simone!

[@rayprusia4753]

Your videos are awesome! Thanks

[@BeABetterDev]

Thanks ray!

[@skmahaboobbasha6059]

Great vedio please make vedio on opsmanager installation on production environment

[@francoisbourdages2906]

très bien expliqué, merci

[@itiscinnamoncafe]

Love longer videos ❤

[@geekaffairs6475]

very well explained

[@BeABetterDev]

Thanks so much!

[@trantrongty8065]

Thank you that really helpful great video

[@BeABetterDev]

You're very welcome Tran!

[@kgcpk]

Superb explanation 😍

[@jeevaengg]

good explanation!!

[@milequinze]

Awesome! Thanks a lot!

[@BeABetterDev]

You're very welcome

[@alexeykorovko6704]

very good explanation, thank you one point is not clear - do we really have advantage of availability / fault tolerance, in case we have an intermediate layer that routes the requests? for me it is like the same, isn't it?

[@فيافيالتأملمهمةإصلاح]

great explanation thank u so much

[@BeABetterDev]

You are welcome!

[@loaizar95]

amazing video!! Understood almost everything and am not a it guy.. the only thing I did not get is the difference between partition mapping and routing :(

[@Tiparium_NMF]

I love this breakdown, but it does somewhat leave me wondering when Sharding would be a good vs a bad idea. The cons seem pretty hefting in comparison to the pros. It would have been nice to run through a few specific different use cases and when one strategy would be better than another.

[@duzgunkenan]

Great content, thank you

[@random-characters4162]

God bless you, sir ✌️

[@HemitPatel-s3f]

is the sharding process explained in this vid the same as in redis clusters?

[@kishoretvk]

Great vidoe, keep up the good work

[@gunnerandersen4634]

Hi there, a nerdy engineer here, I think you could bind some kind of hash or unique ID to your queries, and then track this as a state, then you can verify that a replica has gone through that particular state to make sure the values you are using are at a valid state for the new given query. Not saying this is any good at all, don't get me wrong, as a matter of fact, I think performance-wise might be actually very bad, but it kind of solves the event problem somehow.

[@royal4210]

Thank you programmer charlie

[@cyclomiha]

Hmm..how about PITR? For analytics you could have replica with multi-master approach to each shard, right?

[@vanchark]

This is one of the best explanations that I've seen on this topic, but it still isn't really clear to me how the problem of eventual consistency that you see with the single master-slave paradigm is fixed with sharding. It seems like the problem is still there, since even with shards this master-slave paradigm still exists, just on a smaller scale now

[@GrzegorzPiotrowski-g4s]

Thanks a lof for this video!

@BeABetterDev What if I were to opt for synchronous replication for my read replicas? Wouldn't that provide me with a high level of consistency (strong consistency) between the master node and the replica nodes? Besides, AWS RDS provides async replication for read replicas, does that mean it is eventual consistent? If so, if I am building an application that needs to opt in strong consistency, shouldn't I use AWS RDS read replicas then? What would be an alternative option to that?

[@studychitchat7535]

nice video.. Can u pls tell the software u r using for making this video

[@BeABetterDev]

Hi there, thanks for the kind words. I am using photoshop with a drawing tablet. You can learn more about my approach here: kzbin.info/www/bejne/bHfOaqt6pa-brM0

[@shivakumarranjithn5584]

Great content!

[@BeABetterDev]

Thanks so much Shivakumar!

[@dushyantchaudhry4654]

questions: 1. Database is a slightly misleading term.. when we say database don't we really mean the software (RDBMS / NoSQL) that logically organises the data stored in storage SSDs? 2. If yes are we not splitting the responsibility of the software? i,e. The data still is in the SSD library right? Just the database management software is loaded in different servers and each DBMS server given responsibility for only some of the queries.

[@yna8588]

Can we scale up and scale down the storage of database as per daily requirement using sharding?

[@atacoonthis]

This dude is the Cr1tikal of databases

[@BeABetterDev]

Haha thank you Daniel!

[@puryteevee]

Thanks Critikal!

[@sandrojorgeoliveira175]

Thank you, Dude!!!

[@Anonimus_13]

Cool video) What app do you use for drawing?

[@BeABetterDev]

Adobe Photoshop and a Veikk drawing tablet!

[@asian1599]

doesn't the routing layer introduce single point of failure as well though?

[@poketopa1234]

What I always miss in these videos is, doesn’t introducing a routing layer just kick the can down the road? Now you have all traffic going to a singular routing node, which is not scalable and can fail. What happens when you need to scale the routing node?