What is Database Sharding, Logical and Physical Shards, Dynamic vs Algorithmic Sharding | 2021

[@77loutube]

The greatest drawback is the "complexity" it brings in your architecture. Sharding is the last resort as you said. Nice video with clarity.

[@muskangoyal5536]

Short, crisp and explained in the simplest possible way. Thanks for the hard work you put in. It is indeed very helpful.

[@sudocode]

Thanks Muskan

[@zahid1909]

a bright confident precise and concise lesson. you are a great teacher, kept my brain neurons constantly firing. what a class! keep all these good work.

[@akashpandey3146]

One basic concept that can be added before getting into intermediate concepts like sharding - is Indexing. Often, before scaling up , it's better to index smartly to reduce the reading latency. Overall good video.

[@pranjalsharma3621]

Yogita, just wanted to let you know I read database sharding from other popular sources also but your explanation with example is best I found. Keep Going on !!!

[@saxenapeksha]

Great video, nicely explained. A hands-on video to shard data on SQL and NoSQL databases will help a lot. Good work.

[@siddharthgaur7919]

Nice explanation yogita!! 👍 Hands-on demos would be really great!!

[@sudocode]

Sure 😊 Will do share soon.

[@sumonmal009]

horizontal and vertical sharding 4:20 Sharding 5:26 Logical and physical sharding 17:54 advantages 9:00 algorithmic and logical sharding 11:33 drawback 13:53

[@imtiyazuddin6263]

Nice explanation.! I like the way you try to convey without unwanted gossip and excluding intro

[@amitchatterjee3836]

Great ! A hands on video will make it complete. Thanks Yogita.

[@sudocode]

Most welcome 😊 Will try!

[@thecloudbaba8668]

Very Good Tutorial! Simplified the concept like anything. Sharding Key is a crucial here. Choose sharding if its really needed.. not every organization needed it. choosing wrong sharding key could bring lot of complexities and wrong data to the customer.

[@Gamer_with_knowledge]

The way Yogita teaches these concepts is too good. 😀 Superb explanation

[@tanu6011]

Yes, please a hands on video will be very useful. One on relational(any one) and non-relational(any one) DB.

[@sudocode]

Noted! Coming soon!

[@kumaraditya9986]

Having the theortical knowledge is good but seeing the practical application will give the understanding how it is really done in the industry.

[@stevejob6079]

@@kumaraditya9986 You are right with your suggestion.

[@PradeepKumar-db7pe]

Dynamic sharding - does it also means db is sharded dynamically? Or is it dynamically decided which shards to query?

[@austinfletcher6375]

This series is awesome. Watching most every vid

[@rohit-ld6fc]

best sharding video

[@punjab2downtown817]

Nice explanation! Great work

[@thesobercoder]

Being an architect myself, I can vouch this content is gold!

[@sudocode]

Thanks for the validation Soham. Feels good to know that you liked our content.

[@darshanmn8379]

Great explanation, the way you explained is that the concept is concise and informative, Thank you and keep going!

[@samahome]

Excellent & clear explanation. Kudos.

[@treehuggerpj]

Good overview. My comment is: Sharding and partitioning are not the same though they both are breaking up a large data set into smaller subsets. Sharding implies the data is spread across multiple databases while partitioning is about grouping subsets of data within a single database instance.

[@punyabikash3624]

Excellent explanation it helps a lot keep up good work…thank you so much it’s really helpful ❤

[@satviksharma4897]

Thanks a ton for this course. Really grateful

[@sudocode]

You're very welcome!

[@ChandraShekhar-by3cd]

Nice Explanation of such a great and important topic in system desing. Please keep uploading more video, these are quite informational and it is very useful for cracking system design round for FANG! Thanks a lot for your time and effort.

[@sudocode]

Thank you, I will

[@ashishkumar-lm6cv]

Hi Yogita, Your explanations are awesome but I think you can deep dive once basics are done and also if you plan to have real implementation it would be of great help as most of the developers don't have access to work at such a scale. Keep up the Great Work !!!!

[@sudocode]

Noted. I will share.

[@stevejob6079]

Hello Ashish, thank you for a point such a great point that will compliment her introduction. Great work Yogita.

[@AKASH._._.]

Thank you so much, Mam.It was really Helpful😁

[@mamtasingh-jo2fo]

Yogita,your videos are really very helpful to understand the sharding concept. I have followed the whole system design series.thanks a lot for making this series..

[@sudocode]

Glad to hear that

[@mcraven281]

Great content would like to see relational db sharding as a hands on first.

[@GiriJeedigunta]

Great video thank you !!! What are the usecases for vertical partitions?

[@suniguha]

Excellent content and explanation.

[@biswajitkundu8322]

Salute to your effort to explain the concepts, would love to see hands on for one relational and one non-relational db

[@yashaswinichetty3351]

Awesome content on Sharding, Can you please do a hands on practical video on how to do replication and sharding

[@sudocode]

Will try to.

[@colinmaharaj]

Before I say anything, I will admit I have no experience with sharding, but I do understand what you presented. Scenario. With 1million users, would you say that user info maybe be small but related user information, like images or videos can take up a lot of space. So it's not so much the amount of users as opposed to the space user supported data can take up.

[@Taha14730]

I respect the way you explain

[@noobcodes1365]

I have a doubt, I see that in vertical partitoning example from 3:49 you have put each column in a different database server. Can we do that for vertical partitining? I read that only horizontal partitioning is spread across database servers whereas vertical partitining is done withing a server. Can we do it for vertical too?

[@rajarshiparmar1990]

The best way anyone could have explained it. Been working in IT for the past 2 years I just feel lost in the CS concepts and never found any structured content. Since I did my bachelors in Electronics. Thank you.

[@srinivasanvk57]

Highly interesting and useful content to a beginner. I think the basics are covered very well. Good job and thanks for the video

[@rupeshjha4717]

Great work once again!! Please keep it up! Waiting for sharding strategy and shard keys video!!

[@sudocode]

Very soon!

[@TechnoSparkBigData]

Great explanation. Which mic are you using? You voice is very clear

[@yatishraja3998]

Your explanation is very good but your rate of speech is very fast and maintain a positive tone when you are explaining so that the viewer feels easy to digest the data Thanks

[@mehmetedex]

I'm happy this is free

[@mayanktripathi4u]

Thanks for the video, this is really awesome. Though after watching and learning abt this I am more confused with the other various terms. Please make a video to clarify those. Database Partition Vs Table Partition Vs Distributed Database Vs Replication. What I understand is Replication is a Read-Copy with a distributed system. Distributed Database is having multiple database at different servers however in Sharding we are doing the same, if so what is the difference. And how the Database Partition is different than Table Partition, as in table partition we do not have to worry abt accessing it or no downtime is required however for database partition we would need some downtime when adding new partition.

[@UECAshutoshKumar]

Thank you

[@swaroopsuki1322]

this will help me to understand the logic but hands on will give clear idea how its actually work

[@rawatsohan11]

Great Explanation. Can you please make video on Time series DB ?

[@omni3x]

awesome video...it really helped

[@payamysf]

Very helpful, thank you!

[@aadithyathamizhselvan168]

great explanation

[@pratikvyas3384]

Yes please 🥺 we want hands on videos as well

[@2711Suvro]

18000 views 800 like... ☹️ you need to remind people to like the video in the middle of it... This was very helpful

[@77loutube]

So the strategy to scale in most use cases: Vertically scale -> Separate Read and Write -> Archive and keep it under control -> Try breaking the application into smaller services -> Shard the last resort

[@77loutube]

add cache, replicas* in between

[@priyaarora4436]

Thank you Yogita for making this channel. I went through your primer course videos, and they explain all the concepts in a very detailed manner. However, I am not getting confidence how to give system design interview i.e how to connect all the concepts combined together to form a system. Could you upload a video in an interview manner like starting from a problem, discussion that happen between interviewer and interwee to proceed with the design?

[@akshayagrawal2222]

Yogita!!! your presentation is just amazing and out of the box. IMO you are the best teacher please please make videos on DSA too. I am 100% sure you will get overwhelming response

[@sudocode]

Thanks a lot Akshay. 🤗

[@sonalisatarupa5451]

Helpful video .We would like to have hands-on on how to shard relational DB and NOSQL DB. Reply

[@shobhitarya1637]

Thanks Yogita for such a great video..really helpful..we would like to have lecture on how to shard relational DB and NOSQL DB.

[@debapratimshyam149]

Hi Yogita, does adding more shards later on in the development phase also increase a challenge, what I mean is if we add another shard do I again have to redristribute the data from old shard ?

[@colinmaharaj]

Should I design and code my own database engine?

[@imtiyazuddin6263]

Please make practical sharding video

[@mejiger]

Good explanation

[@rakeshgaur653]

wow, well done!

[@ChandraShekhar-by3cd]

I had one question to ask : let's say that I have sharded the date in to mulitiple machines for desinging facebook, then in this case how the "FEED / TIMELINE" of a particular user be generated? One user can have multiple friends and he can follow many pages as well. So do we need to go to each of the shard where the friend of the user is residing to retureive the post information ?? It will make the system bit slow. How should we overcome this issue??

[@rupeshjha4717]

Hi, when a particular user posts something then there is something called fanout services which push the same data into the timeline of the friends' user which is also maintained using some caching strategy and it happens in the background so that when friend user looks at their timeline can see the post and also one more thing to keep in mind is the data is eventually consistent in most of the social networking platform so it may take some time to update the timeline.

[@ArunRaj-zq5kk]

Is sharding and partitioning the data same? As per my understanding, partitioned the data based on column which will create separate file for each partition value, so that it will improve the read performance when we query the data based on partition column

[@zyro9922]

What an awesome intro

[@Varuag89b]

Great video as always. Hands on like sharding MYSQL would be great. Any plans for that?

[@danielteklay9629]

its Good explanation

[@YasmínHarbinson]

Appreciate the detailed breakdown! Could you help me with something unrelated: I have a SafePal wallet with USDT, and I have the seed phrase. (alarm fetch churn bridge exercise tape speak race clerk couch crater letter). How can I transfer them to Binance?

[@whatif1435]

Great video

[@nashaus6611]

Hi Yogita, what is the use of logical sharding ? Thanks a lot for your videos ! :)

[@geoffreyhibon2651]

Great job :)

[@vinayakashankar6901]

Good job

[@subee128]

Thank you very much

[@ShahbazKhan275817]

Are their any other use case of logical partition other than query optimization?

[@PradeepKumar-db7pe]

Dynamic sharding - does it also means db is sharded dynamically? Or is it dynamically decided which shards to query?

[@kalyanbellamkonda8653]

Very nyc yogita concept is clear

[@sudocode]

Thanks :)

[@jyotir124]

can we do replication as well as sharding for the same DB architecture, if so how. Could you please explain

[@saralk18]

Did you mention problem with transactions after sharading?

[@Live-hh6li]

Nice explanation Thanks

[@sudocode]

You are welcome

[@alakhkaushik]

Thought: Would it be better to mindfully split the tables in different databases instead?

[@Shubham__Saroj]

awesome

[@prithvirajan105]

Thank you :)

[@amanjain4603]

Hi Yogita, Thanks for such a vivid explanation. I just have one doubt. You gave an example of Tinder while talking about Sharding sometime around 10:22. We can shard on the basis of cities definitely. I was just wondering what would happen to a particular person's data (in db) when he moves from city X to city Y ? Will that data be lost ?

[@mohtalha]

Insert the record in new shard and soft delete from old shard

[@matrixlnmi169]

Thanks ! City based partion will come under column ( Vertical partion ) , right ?

[@abhinavjaglan1782]

no, it comes under horizontal data partitioning only, with city name as the shard key. all rows related to same city will be stored in same shard

[@fadedmemes]

Can't we create index instead of sharding like search query will be faster rather than doing sharding?

[@2010aishwary]

very nice videos, please keep it going ..........

[@sudocode]

Thank you, I will

[@dominikg544]

i would like to see video comparing time series dbs for low latency large data sets.

[@tejsingh344]

Query across the shards is not a disadvantage I mean situation itself will not arise as in shards( horizontal partitioning ) we are going to put whole schema in any particular shard not just a specific table..

[@sudocode]

Horizontal sharding implies splitting a table into multiple shards. In some cases all those shards have to be queried.

[@nidhiverma6760]

Please make video on sharding in mongodb.

[@theghostwhowalk]

Great explanation. One video if possible on Zookeeper please with use case.

[@sudocode]

Will add to the list.

[@ashishk8490]

You teach a lot better than Gaurav Sen. Why don't you you launch your course on System Design. It would be a great help for all of us.

[@Nexatek]

Can you make a tutorial on Spring Boot internal working?

[@vaishalirathore3723]

demos for both relational and non-relation would be really helpful...

[@sudocode]

noted!

[@kamalsmusic]

What is the point of having logical shards if we can have multiple physical shards?

[@dominikg544]

nice

[@tayyabachowdary1007]

You should add link to the next and previous video in your description section

[@sudocode]

good idea!

[@justvenkyy...3423]

reading data from a table having a index vs sharding , which gives more performance? index helps to look up isnt?

[@sudocode]

Problems change with scale. Indexing might not help when datasets are huge.

[@jkiakumbo9569]

Hello, I really liked your video and your explanation is very clear but I don't agree with that partitioning and sharing is the same. For my knowledge partitioning is when we are splitting a table in database and sharing is when we are splitting data to different databases. So partitioning is done in the same database, we will just split/ partitioning the tables and sharing is done having multiple database, we will sharing data between database.

[@sudocode]

I am sorry to say but that’s not accurate. Please check your source from where you read this. Check description for sharding links that I shared.

[@Nexatek]

Hands-on videos would be of great help

[@sudocode]

will do try for sure.

[@abhinavpandey3356]

Hi please come up with hands on using mongodb will help a lot 🙏

[@sudocode]

Will add it to the list.

[@jinalkumarpatel5138]

Good one. How sharing of db relate to cap theorem ?

[@sudocode]

It's a long answer!

[@jinalkumarpatel5138]

@@sudocode may be I have used wrong world...it is sharding and not sharing..

[@ranitbiswas6278]

DB Shards and DB Partitions are two different terms if we take distributed database into the picture. In fact, sharding and partitioning have different meanings in distributed system.

[@sudocode]

Could you please link the source to your claim here?

[@balajiarumugam1876]

@@sudocode FYI -> kzbin.info/www/bejne/rYnZnZ2bm72se9E&ab_channel=AsliEngineeringbyArpitBhayani

[@naidusunny2744]

Can you please cover how bloom filter can help here?

[@sudocode]

Will cover in a separate video!