What is Database Sharding?

[@RandomShowerThoughts]

honestly might be the most complete and thorough explanation of sharding.

[@BeABetterDev]

Thanks so much for your kind words!

[@hamadaparis3556]

You've simplified your explanation like google engineers do when they give lectures, I'm sorry if that sounds strange but I've realized that the people who simplify complex things they really know what they are doing awesome man Cheers.

[@BeABetterDev]

Thank you so much for the kind words!

[@mohamedshehata3979]

Nice words

[@cameronmccoy2204]

😊😅😊😊pp

[@cameronmccoy2204]

@@BeABetterDevpppp

[@cameronmccoy2204]

@@BeABetterDevp Pop p p pppp

[@Aidanhyland]

I am burning through all your videos. You are making me a better SAAS Test Engineer! Keep up this great work!

[@abhishekghosh5550]

This is seriously such a great video man. I spent the entire Sunday understanding Sharding. Not that I didn't get started with the concept, however, this video just made everything clear at the end of the day. Thank You.

[@cd92606]

Great video, especially your description about the non-uniformity problem.

[@BeABetterDev]

Thanks Rotary Dialer! Yea the non-uniformity issue is one I've been personally bitten by in the past. Glad you enjoyed the video!

[@v.m.5850]

Watched countless videos and barely understood the concept. Your video on the other hand explained everything along with pros and cons super simply. Thanks a ton.

[@MohammedMubashshir-q8v]

Awesome explanation of sharding, one of the best videos out there. Thanks brother!

[@yfzhangphonn]

Best lesson about database scalability I found, so easy to understand.

[@rjjlucy]

In most ~20min videos, I get tired soon and close them after 5min. I can’t believe your video is so good that I totally forgot time and finish watching all of it

[@BeABetterDev]

Thank you so much Jingyi! Its these kinds of comments that keep me motivated to make more content :) Stay safe Daniel

[@sn-wg9gp]

Been studying system design for interviews. All the videos handwave to sharding. We would shard the db across different regions. I had rough idea what it is that we split the db in to smaller pieces, but nothing concrete. Now it make perfect sense with this amazing video

[@poloska9471]

Dude you make some really awesome content. Please please keep making videos! I love the clarity of your speech, voice, and presentation. I understand and can follow along in your videos a lot better than more other channels. Earned my subscription and likes! Keep killing it homie!

[@BeABetterDev]

Thank you so much for your kind words and welcome to the channel!

[@ase713]

Dude, this was outstanding! Super helpful and covered everything I needed to know!

[@Bhaskarlnm]

Daniel, no words.. looking at your playlists content and videos …amazing. Great great effort to help people. Kudos to you 👏👏👌👌👌

[@BeABetterDev]

You're very welcome!

[@filesopen6188]

this video entails very good explanation and this also entails complex understanding.

[@BeABetterDev]

Thank you!

[@saiaussie]

Hey dude, you're a star! Very clear and upto the point! I cant thank you enough.

[@mathisinav4267]

Hands down! the best explanation I've seen on database sharding, excellent!

[@BeABetterDev]

You're so welcome. Glad you enjoyed.

[@Alexan6548]

Very clear. One of the best tutorial I have ever seen

[@wlcheng]

Great video! Such a clear explanation of how database sharding works.

[@JayPatel12928]

Watched some of your random videos on sys design, and now im hooked. Great content!

[@BeABetterDev]

Thanks so much J! Glad you enjoyed!

[@amrikmalhans]

@@BeABetterDev yes

[@devdewboy]

Thanks for the straight forward easy to grasp concept of sharding. Give this to someone else and we would have gotten a bunch of technical wordy mumbo-jumbo.

[@ChauDuong1982]

Thanks for the videos. Great explaination.

[@JamesQQuick]

This was awesome. Thanks!

[@Lordnoashi]

Amazing explanation, loved it. Thank you, it will help for the future interviews I have.

[@BeABetterDev]

Glad I could help!

[@bahot9057]

Very well explained, great video!

[@ВладимирЛапенков-г1э]

best explanation of sharding i've heard!

[@BeABetterDev]

Thank you very much!

[@codespace747]

Best video ever made on sharding

[@eugeniosp3]

Bro I'll watch anything you make. If you made a video teaching me how to watch paint dry I'd take notes. Keep up the damn good work my mans.

[@IQUE928]

incredible explanation, thank you!

[@eternalnight9453]

New here. Loved your talk! Your presentation and teaching is elegant and simple. Really appreciate it, thank you!

[@BeABetterDev]

You are so welcome!

[@Anton_Rozhanskii]

Great explanation, Daniel. Thank you

[@BeABetterDev]

You're very welcome Anton!

[@lucasarbex926]

Great content man!! It helped me a lot!! Keep up with the good work!

[@BeABetterDev]

Thank you!

[@harishbendale6818]

Very clear, and simple explanation.

[@BeABetterDev]

Glad it was helpful!

[@mivel9763]

Had a hard time grasping on what database sharding actually meant but your video really helped me understand it, thanks! :)

[@BeABetterDev]

You're very welcome!

[@drew4980]

Are there any database tools that make this easier? Couldn't someone write some software to create a wrapper around a sharded DBMS that could handle the routing and re-sharding with a given hashing key?

[@patrick1778]

you are so good at explaining concepts

[@arikedada]

great video, I understand what idempotency operations entails, thank you

[@quang.luu.179]

Good stuff man. I love the clarity you bring to a subject. Subscribed.

[@RajuGupta-st1hj]

Thank you so much for the post. Good work. Keep it up.

[@BeABetterDev]

You're very welcome Raju!

[@sharonleibel]

Great explanations! Thanks, Keep it coming!

[@BeABetterDev]

Thanks Sharon!

[@rajt1998]

Very well explained. Thank you

[@BeABetterDev]

You're very welcome!

[@tamaraamanda2483]

Prepping for Amazon TPM interview and this is so helpful!

[@BeABetterDev]

Thanks Tamara and good luck on your interview! Make sure you focus on those leadership principles !

[@santoshlml]

Well explained. Thank you!!

[@BeABetterDev]

You're very welcome Santosh! Glad you enjoyed.

[@taniaasim]

This is great and super clear. Thank you!

[@BeABetterDev]

You're very welcome

[@OscarMartinez-nt6zn]

Great explanation!

[@shubhamchandra9258]

Awesome video. Very clearly explained.

[@BeABetterDev]

Thanks!

[@donaldkennedy7993]

superb explanation of DB scaling & sharding & W/R databases for a non DB person ;)

[@panggrayta]

woww...!! great videos, great presentation, great explanation. thank you, keep sharing..

[@andrewkicha1628]

Great job on this one, I came here to know more about sharding, but I learned lots of useful information before you even dived into the topic ;)

[@BeABetterDev]

Glad it was helpful!

[@chandnisaini9176]

Well explained!!

[@BeABetterDev]

Thank you!

[@estebanquintana156]

Great explanation. Thank you

[@BeABetterDev]

You are welcome!

[@rschmidtzalles]

clear and concise. subscribed

[@BeABetterDev]

Thanks Libert and welcome!

[@saifmohamed1776]

which better to start with for database basics: - introduction to database systems c.j date . - database internals. // if there are any better or recommended books or materials pls mention. * Great explanation.

[@BeABetterDev]

Hi Saif, This is a tough question to answer. I would step back for a moment to ask why are you trying to learn about databases? I think the answer will guide how/what to tackle first. For example, if you're just planning on using dbs, the database internals may be a bit overkill (but good to know overall). Could you tell me more about why you're learning db's and maybe I can guide you more? Thanks, Daniel

[@saifmohamed1776]

@@BeABetterDev to be aware of the basics in general like concepts physical logical at first And in backend specific. I'm very grateful for your concern

[@BeABetterDev]

Hi Saif, I briefly looked at the two resources you mentioned, I think a better choice is to read Database Internals. I feel that it is much more modern and covers some of the important aspects of database challenges today such as distributed systems and availability. The other book is quite dated and although I'm sure would be beneficial, I think things have changed so rapidly recently that I'm concerned the content will be a bit stale. One thing to note is to not get too bogged down with the details. To be a great developer with database understanding you don't always need to understand the low level details. Knowing how things work at a high level with the ability to dive deep when you need to is much more valuable. Hope this insight helps and I wish you best of luck on your studies. Daniel

[@saifmohamed1776]

@@BeABetterDev thank you

[@SofiaGoyal]

Really good work man... such a detailed video...

[@BeABetterDev]

Thanks Sofia! Glad you enjoyed :)

[@nodrift9503]

Perfect explanation. Thank you

[@hpandeymail]

Very well formed content .. thanks 🙏

[@BeABetterDev]

You're very welcome!

[@samlinsell900]

Vids are awesome, really enjoy them. Interesting that you didn't touch on the lack of thought to database design, indexing and maintenance etc as a way to improve performance. Interested to know why? Especially given the cost of scaling in serverless environments.

[@AnilKumar-lb3qf]

Excellent presentation, very good explanation 👍👍

[@willemplug3366]

Super clear. Thank you!

[@milequinze]

Awesome! Thanks a lot!

[@BeABetterDev]

You're very welcome

[@rayprusia4753]

Your videos are awesome! Thanks

[@BeABetterDev]

Thanks ray!

[@Tiparium_NMF]

I love this breakdown, but it does somewhat leave me wondering when Sharding would be a good vs a bad idea. The cons seem pretty hefting in comparison to the pros. It would have been nice to run through a few specific different use cases and when one strategy would be better than another.

[@subhasishhalder4817]

How come I didn't find your channel before?

[@duzgunkenan]

Great content, thank you

[@jeevaengg]

good explanation!!

[@trantrongty8065]

Thank you that really helpful great video

[@BeABetterDev]

You're very welcome Tran!

[@francoisbourdages2906]

très bien expliqué, merci

[@random-characters4162]

God bless you, sir ✌️

[@فيافيالتأملمهمةإصلاح]

great explanation thank u so much

[@BeABetterDev]

You are welcome!

[@simonemariottini1011]

Really useful content! Keep it up!

[@BeABetterDev]

Thanks so much Simone!

[@ihateorangecat]

Best of the best explantion I kinda start to understand the Sharding in just video.Thank You and I subbed!

[@markryan430]

OMG, I just Sharded myself!

[@itiscinnamoncafe]

Love longer videos ❤

[@poketopa1234]

What I always miss in these videos is, doesn’t introducing a routing layer just kick the can down the road? Now you have all traffic going to a singular routing node, which is not scalable and can fail. What happens when you need to scale the routing node?

[@GrzegorzPiotrowski-g4s]

Thanks a lof for this video!

[@peterroger249]

Much thank you for your great KZbin help. I am new to Excel and Chatbot. How can I migrate the Excel database, export it from Microsoft Azure WebApp, and import it into AWS Chabot? Keep having errors missing QID and others on the AWS Chabot console. Please help show me the fastest way to convert the Excel and make it compatible with AWS Chatbot?

[@dushyantchaudhry4654]

questions: 1. Database is a slightly misleading term.. when we say database don't we really mean the software (RDBMS / NoSQL) that logically organises the data stored in storage SSDs? 2. If yes are we not splitting the responsibility of the software? i,e. The data still is in the SSD library right? Just the database management software is loaded in different servers and each DBMS server given responsibility for only some of the queries.

[@alexeykorovko6704]

very good explanation, thank you one point is not clear - do we really have advantage of availability / fault tolerance, in case we have an intermediate layer that routes the requests? for me it is like the same, isn't it?

[@kgcpk]

Superb explanation 😍

[@r-rtz]

A more interesting concept though is how you generate these unique id's that are used in the sharding / partitioning and ensure uniqueness

[@channuangadi7504]

and there is another complex thing is the id generation (here Customer ID) when we shard we have to make sure duplicate ID should not be generate, can we have video on ID generation in distributed computing

[@geekaffairs6475]

very well explained

[@BeABetterDev]

Thanks so much!

[@3042640426]

Thank you so much

[@royal4210]

Thank you programmer charlie

[@loaizar95]

amazing video!! Understood almost everything and am not a it guy.. the only thing I did not get is the difference between partition mapping and routing :(

[@asian1599]

doesn't the routing layer introduce single point of failure as well though?

[@skmahaboobbasha6059]

Great vedio please make vedio on opsmanager installation on production environment

[@cyclomiha]

Hmm..how about PITR? For analytics you could have replica with multi-master approach to each shard, right?

[@drummerboi4eva]

excellent video

[@socialawareness1643]

i HAVE A Question: What if the Shards returns the incomplete information . Means If customer queries the DB and shard returns the incomplete info ?? Then whats the use . Why NOT the backup is a good option ?

[@HeavensMeat]

I know you have had other dynamodb videos here but would it be possible to have a more in depth video dealing with sharding in dynamodb and also utilizing this with python/boto3 vs the cli? I know it's not really the same type of sharding per se but this video reminded me that I am interested in seeing that kind of thing

[@BeABetterDev]

Hey HeavensMeat! You're suggestion is a great idea for a new video idea, thanks you! I'll work on incorporating this into my todo list. Cheers!

[@kishoretvk]

Great vidoe, keep up the good work

[@SuperWhatusername]

Great content

[@HemitPatel-s3f]

is the sharding process explained in this vid the same as in redis clusters?

[@얀고양이-f9h]

What if one of the shard node is down. For HA, we still replica for each shard node .

[@shivakumarranjithn5584]

Great content!

[@BeABetterDev]

Thanks so much Shivakumar!

[@rishiraj2548]

Thanks

[@benhunt2023]

This feels like the khan academy of coding. Well done.

[@gunnerandersen4634]

Hi there, a nerdy engineer here, I think you could bind some kind of hash or unique ID to your queries, and then track this as a state, then you can verify that a replica has gone through that particular state to make sure the values you are using are at a valid state for the new given query. Not saying this is any good at all, don't get me wrong, as a matter of fact, I think performance-wise might be actually very bad, but it kind of solves the event problem somehow.

[@atacoonthis]

This dude is the Cr1tikal of databases

[@BeABetterDev]

Haha thank you Daniel!

[@Anonimus_13]

Cool video) What app do you use for drawing?

[@BeABetterDev]

Adobe Photoshop and a Veikk drawing tablet!