Designing INSTAGRAM: System Design of News Feed

  Рет қаралды 641,408

Gaurav Sen

Gaurav Sen

Күн бұрын

Пікірлер: 598
@elliemay1748
@elliemay1748 4 жыл бұрын
I interviewed for PayPal a few weeks ago, and this was the exact systems design question they asked. They said to me, “Do a system design for Instagram”. I smiled, because I had just watched this video a few days prior, and so I knew exactly how to answer. Thank you for this video, you helped me get a job, for real :)
@gkcs
@gkcs 4 жыл бұрын
Congratulations!
@sonushete4431
@sonushete4431 3 жыл бұрын
wow awesome Elli May got job in Paypal 😁
@brandonzheng1092
@brandonzheng1092 3 жыл бұрын
What's there to be proud of when you've seen the answer to an interview question beforehand?
@ashishsharma-nz5pq
@ashishsharma-nz5pq 3 жыл бұрын
@@brandonzheng1092 so one should not feel proud anyways, cuz he/she had studied that in books b4... lame perception
@pratik3106
@pratik3106 3 жыл бұрын
@@brandonzheng1092 I sense happiness rather than pride, none of that this person said implies proudness, and even if so, why not? I'd be proud.
@samanrajaei8129
@samanrajaei8129 4 жыл бұрын
Damn this kid is good. Better than most of the "veteran" system architects I've worked with.
@dijoxx
@dijoxx 3 жыл бұрын
You must have worked with some really crappy architects if that is indeed the case.
@xReisk
@xReisk 4 жыл бұрын
As a backend developer who strugles to do projects due to thinking to much and taking a lot of time to make dumb things... I think I have found one awesome channel for me. Thanks for your videos man!
@gkcs
@gkcs 4 жыл бұрын
😁
@Tuanvu-tb1mh
@Tuanvu-tb1mh 5 жыл бұрын
Hi Sen, for the database design I think you should go from Logical ERD first then derive Physical Tables from there, it is more natural approach
@amlanroy5274
@amlanroy5274 2 жыл бұрын
Well, I wasn't even looking for this. Just got a random recommendation and now I'm watching this with full focus at 3AM. Never thought this day would come😂 Awesome video.
@ChitranshuVashishth
@ChitranshuVashishth 5 жыл бұрын
I'm not even a CSE student. But watching your videos, Gaurav, actually intrigues a lot and motivates me to actually learn more about programming and design some my own scalable system one day. Thanks Gaurav.
@blasttrash
@blasttrash 5 жыл бұрын
Try coding train channel. You can watch most of his videos and have fun. Its like watching a movie and he does real coding. For example watch this video, even if you dont know programming, you will still understand it and its so much fun. Coding the snake game kzbin.info/www/bejne/d5KqfGCcn5J1d68
@ci7alex1
@ci7alex1 5 жыл бұрын
Explains a lot why there are so many well paid people behind each successful online service, so complex, wow
@HarkiratSaluja
@HarkiratSaluja 5 жыл бұрын
Probably the first video of the system design I have seen. Being a front end developer, I had a fair idea about the things but the way you explained just wow
@bmw1553
@bmw1553 4 жыл бұрын
Gaurav, first of all, thank you for a fantastic, simple and clear explanation. Second of all, I can imagine the work went in to put this video, it must be humongous task in preparing the right content, taping, editing, etc. Great work!
@gkcs
@gkcs 4 жыл бұрын
Thank you!
@apurvsawant5703
@apurvsawant5703 4 жыл бұрын
All the important concepts are explained very simply and this is what makes this video amazing.
@nayankhanna2367
@nayankhanna2367 4 жыл бұрын
Been addicted to this channel recently and binge watching it even though I have my exams ongoing xD. Man you are the best. (saying this from my experience of having watched more than 100 "Real" Coding KZbinrs) My Systems Design knowledge is growing leaps and bounds by watching these and I plan to implement these good designs just after my tests are over. I have worked on several Applications as a backend developer, and always stressed heavily on scalability, flexibility, ACID properties. But, your channel has taught me a lot more good techniques and design concepts.
@gkcs
@gkcs 4 жыл бұрын
Thank you 😁
@MichaelAMomo
@MichaelAMomo 3 жыл бұрын
Seriously man, It is a great help
@kennethcarvalho3684
@kennethcarvalho3684 2 жыл бұрын
Life is incomplete without a Gkcs design video
@gkcs
@gkcs 2 жыл бұрын
Hahaha!
@Yan-rv8mi
@Yan-rv8mi 4 жыл бұрын
7:16 I think we do need a "type" on the Activity table. For example, suppose there's a postID being "123" and a commentID also being "123". Since both postID and commentID can be interpreted as activityID on Activity table, if there a row on activity table with activityID being "123" we don't know it's for the post or for the comment, unless we have a column "type" to distinguish between them.
@jellyrabbits375
@jellyrabbits375 2 жыл бұрын
True in this case, it depends on your system though. In case where the id is a UUID, then there won't be a case where postId and commetId would be equal.
@vishalarora6068
@vishalarora6068 4 жыл бұрын
what you said is load balancer is work of service discovery or metadata service like zookeeper/consul/etcd and what you described as gateway is the work of load balancer or reverse proxy. Looks like you haven't used these systems practically(I am not blaming) and trying to inform others based on(your interpretation of) what you read online.
@zshanz
@zshanz 4 жыл бұрын
good point.
@adamberry7536
@adamberry7536 4 жыл бұрын
Can I ask a question about your comment. I totally agree what he's describing as a load balancer is actually more like Zookeeper. But I'm confused about the gateway comment. My understanding, which I may be wrong about, is that a gateway will handle authentication, authorization, and then route the incoming HTTP request to one or more services to accomplish the task at hand depending on the configuration. So yes, it sort of acts as a reverse proxy with the addition of authentication logic and possibility of making synchronous service calls. But I don't see how the gateway is a load balancer. It doesn't distribute API calls based on load. It distributed then based on function. If you wanted load balancer between a service and the gateway or a service and another service you would still need a load balancer. Is this correct?
@ZeroInfinityCoder
@ZeroInfinityCoder 5 жыл бұрын
@Gaurav i think it should be "postId" in place of "activityId" in "comment table" when you were explaining feature no. 2 as let say we want to find all the comments for a particular post , then we will look into comment table for column postId. correct me if i am wrong
@augmentos
@augmentos 4 жыл бұрын
Would love to see a system design of notifications (activity feed) in twitter/IG etc. Aggregate etc.
@yog2915
@yog2915 4 жыл бұрын
Fantastic work it is because of people like you skills of general masses are also rising
@theakatsuki2113
@theakatsuki2113 3 жыл бұрын
Wow his content is really at next level I love this guy and respect his efforts and the amount of hard work he puts in each and every video
@AtomicAkshay
@AtomicAkshay 3 жыл бұрын
Great video man, really appreciate the fact that you've been posting such indetail conceptual content for free.
@sahajarora2162
@sahajarora2162 4 жыл бұрын
Hi Gaurav! Very thankful to you for sharing your knowledge with the rest of the world! I have 3 questions about the GATEWAY: 1) Is it a Micro-Service? If not, what exactly is it (i.e what does it contain)? 2) It seems like a single point of failure, looking at the diagram. 3) If we have multiple instances of a Gateway, then would the Load balancers be needed in between Client and the Gateway Service ?
@pman-codes
@pman-codes 4 жыл бұрын
Hi Gaurav, Greetings. I love your work, I am a subscriber and a frequent liker. However, I find an implicit assumption in your system which considers the Instagram mobile app as thin clients. The process of storing the posts in the cache in the server would result in an unscalable system. I believe the posts are cached in the user's app memory(cache and physical storage), considering that these apps have a considerable chunk of internal storage used. An added proof for this would be if you try to open up Instagram in offline mode you can still see past posts and a toast message which says "couldn't refresh feed". I would like to have the cache on the user's system and then an identifier that is stored in a place where you are storing the cache of the post on the server. (considering the news feed functionality. This can be applied for other uses too. ) Thanks.
@gkcs
@gkcs 4 жыл бұрын
This is a very good point. Thanks for posting 😁
@vikask7246
@vikask7246 4 жыл бұрын
@@gkcs can you explain system design for telegram
@dustindiaz
@dustindiaz 4 жыл бұрын
@@gkcs the posts ARE stored in application cache... but it doesn't invalidate the fact that mobile apps aren't still thin clients. a user can delete an app, or visit from a third party integration (not built by instagram) -- in which case these timeline feeds are still stored in horizontal caches. you wouldn't believe the amount of money instagram/twitter/etc spend on memcache to make this happen.
@adminwadidaw576
@adminwadidaw576 4 жыл бұрын
@@dustindiaz so which way do i have to follow? Do i need to cache posts in client side? I confuse in like sectiob , whenever user click like, should the client side make a request?
@dustindiaz
@dustindiaz 4 жыл бұрын
admin wadidaw caching on the client is helpful when revisiting an application. This way a user can be presented with information immediately. Caching on the server, on the other hand, is necessary for large scale services to deliver things like timelines since raw sql queries based on this system design would cause the system to fall over with just a decent amount of traffic
@sankalparora9374
@sankalparora9374 Жыл бұрын
Amazing stuff - not only informative - but interesting! Thanks!
@JaswinderSingh-uw2hf
@JaswinderSingh-uw2hf 3 жыл бұрын
Regarding Hybrid approach : practically User1 follows the ordinary user and celebrity as well. now when post done by ordinary user it will push to user1 but when post by celebrity, system/client has to pull. now how client know when it has to pull ? @gaurav sen sir, can you please explain. or correct me if I misunderstood something...
@devarajt6965
@devarajt6965 5 жыл бұрын
hi Bro.. Actually the way you explained the stuff is very simple and clear.. Thanks for your time for making such videos..
@gkcs
@gkcs 5 жыл бұрын
Thank you!
@jayanthmanklu8642
@jayanthmanklu8642 5 жыл бұрын
Hi Gaurav, Your energy is just unmatched! Audience Request: Please consider doing a video on how would one architect IRCTC Tatkal Booking scenario - with hundreds of thousands of tickets sold in 2 to 3 minutes time duration. Thanks
@gkcs
@gkcs 5 жыл бұрын
I'll try to work on this 😁
@krutikpatel906
@krutikpatel906 2 жыл бұрын
Hi Gaurav, thank you for amazing content. Can you please share your thoughts on why you chose SQL database for all these data instead of NoSQL? Since the volume is high and eventual consistency seems to be ok, can we use NoSQL database for this kind of data? Thanks
@harisridhar1668
@harisridhar1668 3 жыл бұрын
13:30 Hi Gaurav - thanks for pointing out the need for a load balancer with the snapshot technique stored onto Gateway for network routing when we horizontally scale the server-side. But why is communicating with the load balancer inefficient? Is this to avoid constant network calls ( which are slow ) and to utilize the SS, which can be stored into memory-side on the Gateway application?
@A.n.a.n.d.k.r.
@A.n.a.n.d.k.r. 4 жыл бұрын
I didn't knew the dbms subject was so much exciting....
@HemantSharma-fw2gx
@HemantSharma-fw2gx 5 жыл бұрын
Thanks for explaining the practical use of all we study in our syllabus..Your videos are superb!
@gkcs
@gkcs 5 жыл бұрын
Glad to hear that!
@aakashjapi4394
@aakashjapi4394 4 жыл бұрын
Great video! That said, in your descriptions of the database schema, you should mention hotspotting as a justification for certain decisions as well. Namely, a very good reason to not add a "likes" column to posts is that it creates a lot of contention on rows in a single table, especially because single posts can get hundreds of thousands of likes. You arrived at the same conclusion - building tables that allows for writes to avoid contention and thus reads to be aggregations (which can then utilize caching) - but I think focusing on the larger problem of hotspotting motivates your design decisions better.
@rujotheone
@rujotheone 4 жыл бұрын
Noob question, please what is hotspotting
@osmanbaskaya7400
@osmanbaskaya7400 2 жыл бұрын
@@rujotheone some records are getting queried more than others. the specific instance that contains the record will be much busier than the rest of the system. you're not balancing the load ideally uniformly.
@amitagnihotri30
@amitagnihotri30 5 жыл бұрын
From Designing Tinder to Instagram, in a very short time :D
@gkcs
@gkcs 5 жыл бұрын
Hahaha, just 6 months 😉
@KShi-vq4mg
@KShi-vq4mg Жыл бұрын
fantastic explanation. but could you clarify one thing? Once the posts are stored into DB by a user, you notify userfeed service. user feed service gets followers for given userid and update post feed to "Each user's queue" which could be max 20. could you explain how queues are established? are these dynamic? say there are 20 followers. so 20 queues?
@dataman4503
@dataman4503 2 жыл бұрын
Actually a lot of considerations and thinking in multiple angles is required while doing a System Design. Sometimes, it's just like 'hey, where would that service get the data from? would it need any authentication? etc/. etc./,' Prepare well!!!
@AseshShrestha
@AseshShrestha 3 жыл бұрын
Love you system design videos. Love from Nepal 👍
@rajanlagah4529
@rajanlagah4529 2 жыл бұрын
at 20:20 Why storing feed in cache as LRU ? I mean if feed is used it should be deleted or replaced by new one right ?
@nishantt6419
@nishantt6419 4 жыл бұрын
I don't see any discussion on the type of databases used SQL vs NoSQL and also how you are scaling the DB, what type of queries you are using and how you are partitioning the DB? Also, it would be more helpful if you can explain with some concrete examples on the capacity estimates like the total users, Daily Active users, daily file uploads, total DB/Object storage etc
@harkirat1
@harkirat1 5 жыл бұрын
Great Explanation. Clear and concise.
@gkcs
@gkcs 5 жыл бұрын
😁
@JitendraSinghPal
@JitendraSinghPal 3 жыл бұрын
Nice explanation. Excellent work
@bianbian621
@bianbian621 5 жыл бұрын
great video! the animation part is awesome. I like all your system design videos.
@gkcs
@gkcs 5 жыл бұрын
Thank you 😁
@chathurabuddi
@chathurabuddi 3 жыл бұрын
What a wonderful channel!!! just subscribed
@satpar9062
@satpar9062 5 жыл бұрын
From the mobile system design perspective, pull model is not suited for reasons like battery consumption, drop in network connectivity but a nice explanation of various possibilities.
@emmanuelevbuomwan2665
@emmanuelevbuomwan2665 4 жыл бұрын
Concise, at the same time; broad and easy to understand.
@gkcs
@gkcs 4 жыл бұрын
Thanks 😁
@NikPnchl7
@NikPnchl7 3 жыл бұрын
Will we store the post meta-data in a relational db ? Would that quickly fall apart given the scale of Insta ? Or would a nosql like Cassandra be the way to go with the tables you've described ? What do you think ?
@Eduardo-fk7ft
@Eduardo-fk7ft 4 жыл бұрын
I'm building a reddit clone, and your way of designing the news feed gave me a lot of ideas, thank you!!
@blinkkeebs
@blinkkeebs 4 жыл бұрын
me too! reddit clone from ben awad's tutorial? :D
@Eduardo-fk7ft
@Eduardo-fk7ft 4 жыл бұрын
@@blinkkeebs No, I'm just going over the reddit site and reverse engineering the features, but I've also changed some things. BTW I'm building it in MEAN stack.
@blinkkeebs
@blinkkeebs 4 жыл бұрын
@@Eduardo-fk7ft Oh! I'm also building reddit clone but I use React and Next.js for the server-side rendering. The db I use is postgresql. I think I should add some extra features for the project and I found this video :)
@Eduardo-fk7ft
@Eduardo-fk7ft 4 жыл бұрын
@@blinkkeebsVery nice tech stack!!. Yes, the best way to learn anything is to make it your way or change it, that is something that works best for me. Good luck, and may google be you best friend!! :P
@PramodShetty
@PramodShetty 5 жыл бұрын
Since the following table(user Id, follower id, timestamp) can grow very big, sharding or indexing would be required. On which column should that be done. If I index one of the this id columns, one of the service, getFollowing or getFollower will be affected badly.
@stepup8108
@stepup8108 3 жыл бұрын
your system design implementation is goign out off my head , i think need to study the basic then only i can get u what you wants to say
@dbtechprojects2392
@dbtechprojects2392 4 жыл бұрын
Thanks for posting this, the part on how to handle the news feed helped me out a lot, originally I could only think of the first method which the administrative tasks are way too high, precomputing the news feed is an option I didn't even think about. thanks :)!
@gkcs
@gkcs 4 жыл бұрын
Glad it helped 😁
@johnfrades
@johnfrades 4 жыл бұрын
Awesome man, im glad to see your channel, subscribed immediately! Very helpful!
@praveen3123
@praveen3123 5 жыл бұрын
Hey Gaurav great videos bro.. Every software engineer should know system designs to build scalable, robust applications.. keep rocking!
@gkcs
@gkcs 5 жыл бұрын
Thank you!
@praveen3123
@praveen3123 5 жыл бұрын
@@gkcs by the way now you're a software engineer with at least 3- 4 years of experience by now. Do you still practice algorithms? I'm in this dilemma whether to practice or take it light
@gauthamhonnavara
@gauthamhonnavara 5 жыл бұрын
@@praveen3123 Never stop learning !
@rohan1456
@rohan1456 4 жыл бұрын
05:05 select count (*) from likes where post/parent_id = 'xyxyx' AND type = post. mentioning this incase anybody gets confused
@raveenamewani7963
@raveenamewani7963 4 жыл бұрын
Hi Gaurav, I really like your videos they are very clear and to the point. I would like you to share in one of your videos how the Amazon Market Place Design will look and work like
@tvnathreviews
@tvnathreviews 3 жыл бұрын
NoSQL db would be better for this right?
@rizthetechie
@rizthetechie 2 жыл бұрын
Hi, Could you share more technical details on how this load balancer can have snapshot of each service. how are sharing of these snapshots between load balancer and api gateway actually implemented
@rg-fi4ef
@rg-fi4ef 3 жыл бұрын
hi Gaurav, few queries which were asked to me during one of the interview 1) Is the follower - followee ER design efficient when you have millions of people using the system ?
@rahul_bali
@rahul_bali 3 жыл бұрын
I don't understand this push model and pull model. How does this work ? Any detailed explanation, how can this be implemented ?
@AlphaWatt
@AlphaWatt 2 жыл бұрын
Thank you very much for this. Excellent explanation through such a complicated topic. Really helped me think through a follower service I have been struggling to commit to.
@swapnilgt
@swapnilgt 2 жыл бұрын
For pushing notifications, WebSockets won't work when the application is closed. Also, WebSockets is a costly way to manage connection just for updates related pushes. It will drain a lot of battery unnecessary. WebSockets are more useful in more real-time use cases when the application is actually open eg. Chat applications, Multi-player gaming applications. Notification for updates for posts need not be real-time. Firebase Cloud Messaging is the best way to do that.
@yuxiongzhu4249
@yuxiongzhu4249 5 жыл бұрын
Hi Gaurav, thanks for this great post. You look so young, how could you be so knowledgeable?
@vikaspizza
@vikaspizza 4 жыл бұрын
He is actually 45 years old. He designed a system that removes aging signs from his youtube uploads...
@reshmaasha4775
@reshmaasha4775 Ай бұрын
@@vikaspizza 🤣😂🤣😂
@code_report
@code_report 5 жыл бұрын
Great video!
@gkcs
@gkcs 5 жыл бұрын
When code_report says something, you better believe it 😎
@eleandro
@eleandro Жыл бұрын
About the sending of notifications to millions of users, if we are talking about pub/sub notifications, just send the notification to a topic where all users are subscribed to.
@azeeztaiwo2802
@azeeztaiwo2802 4 жыл бұрын
Good video, my question though would be why you chose an sql datastore and not nosql, considering that the app is read heavy and need to scale
@alexkuzmichev9051
@alexkuzmichev9051 4 жыл бұрын
I'm so glad I found your channel. Keep up the good work! Nice videos:)
@gkcs
@gkcs 4 жыл бұрын
Thanks 😁
@AbhishekGupta-kv9ne
@AbhishekGupta-kv9ne 3 жыл бұрын
@Gaurav : For sending notifications to user why should we use Long polling or Web sockets instead of SSEs (Server side events).
@abhishekgupta4570
@abhishekgupta4570 5 жыл бұрын
lot of learning with video and bro one request can u make video on your uber interview about question asked roundwise and that HR round which was pretty tough as you mentioned in video (Got job in uber).
@gkcs
@gkcs 5 жыл бұрын
Thanks Abhishek! I won't be mentioning the questions asked, because we aren't allowed to. "Got hired" will turn to "Got fired". 😝 You can go through the content on the channel, it's more extensive than an interview set 😁
@saurabh945644
@saurabh945644 5 жыл бұрын
I believe we can store the user feed on the client as well, since we can recompute in case of app is reinstalled. Other thing is cache will be updated even when user is not even using the app but if we store at client we will only be updating (Hybrid Model will be better) when user is up and running.
@gkcs
@gkcs 5 жыл бұрын
You can cache it on the client. But it's better to keep such responsibilities on the server.
@saurabh945644
@saurabh945644 5 жыл бұрын
@@gkcs I would agree, Some users like me use Instagram on browser only. :) As per as app is concerned we can keep on mobile but I see challenges as well lets say I did not open the app for like 4 hours and whole 20 post feed needs to be updated then we need to do big recalculation step. For fast response times as long as user is on the app we can cache feed on the app push will directly go to the mobile with out one extra hop in between. But I am still not sure what will be better may somewhere in between server + client would be ideal. I may be completely wrong. Thoughts?
@boteng1837
@boteng1837 3 жыл бұрын
9:45 why does Post table not have activityId. also, should comment table have a parentId to document how which post/comment the comment belongs to?
@pablo_aldana
@pablo_aldana Жыл бұрын
Awesome video! thanks a lot Gaurav for this very insightful. I have a question tho regarding of likes and feeds. When retrieving the feed, what would you say is the best way of knowing if the user has liked the post that is seeing at this moment? As calling the DB every time seems a bit overkilling. Keep cracking on!
@dennism5090
@dennism5090 4 жыл бұрын
I like the presentation, but I'm finding a common pattern of "go look at my other video" whenever a concept is brought up. This is fine if its a complete 30 minute video about CDN's, but for 30 seconds... PS: This video got 100x better at 12minutes in.
@sankarasubramanian568
@sankarasubramanian568 2 жыл бұрын
Amazing Video, Thanks Gaurav :)
@PrashantMarshal
@PrashantMarshal 5 жыл бұрын
I think we could have Likes in Posts/Comments table. The reason being they don't violate NF rules + it is going to be more efficient in terms of space. And if we think whether it's a characteristic of a post, then I can't think of why not? Also, since RDs support indexing, you could also include any suggestions on which all keys to index or anything that saves the world.
@gkcs
@gkcs 5 жыл бұрын
You mean the "count of likes for this post", or "who has liked which content"?
@PrashantMarshal
@PrashantMarshal 5 жыл бұрын
@@gkcs The count of likes.
@gkcs
@gkcs 5 жыл бұрын
@@PrashantMarshal Will you update this record on every like?
@PrashantMarshal
@PrashantMarshal 5 жыл бұрын
Gaurav Sen wouldn’t the strategy for updating the Activity table (be it batched queries or point queries) be valid for the Comments table too?
@ritasharma8994
@ritasharma8994 4 жыл бұрын
Hi Gaurav, Your explanations are very clear and relevant. I love your accent. Most of the guys use fake accents just for videos which irritates me a lot. One request I wanna put here for the system design of Metro system, stack overflow system as mostly I saw them in the tread.
@gkcs
@gkcs 4 жыл бұрын
StackOverflow is interesting, I'll try working it's design video soon :)
@jalandharchinthakunta3239
@jalandharchinthakunta3239 4 жыл бұрын
Great video Gaurav. Explained it in simple, crystal clear thoughts. However i have the below questions on this design: 1. As you have mentioned, when a celebrity like Justin Bieber publishes a post on Instagram having millions of followers, the system has to pre-compute the user feeds for all those millions of followers and it might end up crashing. Can we skip the pre-compute for these cases which involve millions of users and insert those feeds at runtime Or do we have any better way to handle this. 2. We are computing the feeds say 20 per each user, how do we handle when the user access more feeds than 20. I assume we hit DB for getting those feeds but won't it create heavy load on system when more users access feeds (>20). Is there a better way to handle this. Thank you so much for all your videos
@suchismitagoswami5609
@suchismitagoswami5609 2 жыл бұрын
Great explanation as always. However, I have a little doubt. Instead of sending notifications from Post Service to the user feed service directly, what iff we introduce an event bridge in between and different services will communicate with each other based on the event produced by other services?
@user-mt6kv1dw7m
@user-mt6kv1dw7m 3 жыл бұрын
Nice video! I have one question, though: If we precompute the feed, what happens when a user scrolls past the 20 posts we have cached? Then we would have to compute the rest of the feed at the request of the user, which would be inefficient, right? Sorry if I didn't understand that correctly.
@Happymejoyus
@Happymejoyus 4 жыл бұрын
question is about the DB design for followerID and followeeID : I assume none of the columns here have constraints on them. If my assumption is right, then how the TAT is in the limit without indexing on the table? Basically, I am not getting (i)how this table is connected to other parts of the whole schema and (ii)retrieving results in time rather not slowing down the total response time? Besides, Thank you very much Gaurav like always ;) :)
@tanveerdayan1363
@tanveerdayan1363 3 жыл бұрын
You have spoken about caching the first 20 posts. What about rendering the next 20 posts?
@ayushvrma93
@ayushvrma93 5 жыл бұрын
Hi Gaurav, First of all, a big thanks for all the videos that you are making. I am preparing for my interviews and these videos are helping me a lot. I would request you to please make a video explaining designing of a game, maybe a little complex game such as FIFA, which can give us an idea as to how to implement real-time occurrences. Since all the moves in a game need to be processed in real time it would be different from a system like Instagram as they can afford to have a delay of a few seconds but these games can not. Thanks :)
@gkcs
@gkcs 5 жыл бұрын
Thanks Ayush! I am working on tic tac toe currently. It'll be progressing towards more complicated games soon. 😁
@dhruvshah1549
@dhruvshah1549 5 жыл бұрын
Great content!!! Any tips or advise on how to determine the type of DB to use? I believe you opted to use some form of SQL database but what was the motivation behind it would be something useful for the audience(or at least me!!).
@gkcs
@gkcs 5 жыл бұрын
I went for the SQL DB because I hadn't spoken much about NoSQL databases then, and because I am more comfortable taking about SQL databases since I have professional experience working with them. Now, however, I would choose NoSQL for stuff that doesn't have strict consistency requirements. kzbin.info/www/bejne/roLReoFvl7xjpck
@dhruvshah1549
@dhruvshah1549 5 жыл бұрын
@@gkcs Makes sense. Thanks Gaurav..
@swapnilgt
@swapnilgt 2 жыл бұрын
In the use-case celebrity, the decision has also be made on the basis if the polling is affecting the user's device battery or not. WorkManager on Android is a great way to achieve that. It optimises the resources and takes decision when to give a CPU chunk to a particular application only when the device has decent resources to spare. This could could surely save a lot of CPU clocks on our server end for sending PUSH.
@rittwikarudra626
@rittwikarudra626 4 жыл бұрын
Very good explanantion Gaurav. The way instagram generates feed today has changed drastically with their new graph api which focuses more on relationships. It would be great to see a video on that.
@gkcs
@gkcs 4 жыл бұрын
I'll look into it :)
@vivek5562
@vivek5562 5 жыл бұрын
What should be the indexing for database tables, so that queries, sharding, etc. are optimized? Taking example of comment table It contains activityId as well as commentId. Should we have activityIdCommentId table separately and remove activityId from comment table?
@divyanshujuneja1291
@divyanshujuneja1291 3 жыл бұрын
Where can we study more about this "Gateway"?
@danniel881205
@danniel881205 3 жыл бұрын
Starting with DB tables might confuse people on the DB selection, it seems implicitly indicating it has to be SQL ?
@kavyajain8889
@kavyajain8889 3 жыл бұрын
Is it better to use firebase RTD to push real-time updates instead of using web sockets?
@devinsills1281
@devinsills1281 3 жыл бұрын
What about the next page of the news feed? As I understand it, the user can scroll down as many times as they like and they'll see older and older content. Your design optimizes page 1, but what about the next page(s) from the inevitable scroll down? I can see an extension of your pre-computation for page 2, 3, etc. When page 1 is full, the last (oldest) item is popped and prepended to page 2, and same for page 2, 3, etc. Eventually pages will be evicted from cache and you could use the oldest timestamp of your current page to manually fetch the next page, add it to cache, and return to the user on cache miss.
@indrajitbanerjee5131
@indrajitbanerjee5131 4 жыл бұрын
Nice Work Gaurav.
@mayankkashyap7165
@mayankkashyap7165 Жыл бұрын
@gaurav. First of all, your videos are great. Big thanks to you. I have a question. - if I add functionality of showing user’s own profile. - all his posts will also be cached? If yes, how and where?
@TuanAnh-qh2sx
@TuanAnh-qh2sx 3 жыл бұрын
What do you think if I design each entity to include only its attributes, no id of other entities? I have a "Edge" table (id1, edgeType, id2, data, timestamp), which save relations between ids. if I need, for example get list of posts of a user, I will only query from this table. Hope to hear from you soon
@gkcs
@gkcs 3 жыл бұрын
Try a graph database.
@TuanAnh-qh2sx
@TuanAnh-qh2sx 3 жыл бұрын
@@gkcs I tried graph database, but I see the processing of saving and executing edge is a bit complicated. So, I came up with this solution. Is this way impossible?
@anuragagnihotri5238
@anuragagnihotri5238 3 жыл бұрын
@Gaurav at 20:55 You said regular polling/http requests (I assume would be using some connection pool, not establishing tcp connection for each poll request) from clients/cellPhones asking for updates from server will have bandwidth/battery issues. But even when we go with LongPolling/Websockets, its actually keeping either connection always(Wbsockets) open/connected or maintain connection for longer period(Long Polling) between client and server for realtime data transfer but this will also have battery and bandwidth issues right? So mainly how do we measure Pros/Cons for this specific Instagram case between Long Polling vs WebSockets?
@shivammishra6630
@shivammishra6630 5 жыл бұрын
Hi Gaurav, I have a question. How can you have a foreign key reference to primary keys of multiple tables? You mentioned this while discussing "likes" in which parentId has reference to both post and comment.
@ravicrazyster
@ravicrazyster 2 жыл бұрын
late response!. I had the same Q but what I found is that its not possible. So you better break to comment_likes, post_likes tables
@chitwanawasthy393
@chitwanawasthy393 2 жыл бұрын
Hi Gaurav Great video it is. Thanks for this. Had a query. How efficient it would be when a celebrity having 50 million followers(or may be more) posts something and we need to add the post in cache for all of the followers?
@deepakzworld
@deepakzworld 4 жыл бұрын
What did I just watch?? Here are the system components I've explained in past videos, here are the components I'll explain in future videos. Here is your Instagram :P :P
@siddharthbhola4231
@siddharthbhola4231 5 жыл бұрын
@GauravSen: Excellent video! Please help me with a question: Can you please explain the reasoning/thought-process behind choosing a Relational database for Users and Feed Schema?? What factors do you consider when taking such a decision? Thanks and Regards
@YashArora721
@YashArora721 5 жыл бұрын
Dude who follows Stephen Hawking on Instagram and why would he even be on Instagram..😂😂😂 JK P.S. Great Video btw 👍
@cengizandak4241
@cengizandak4241 5 жыл бұрын
Hello, do you think that the pre computation should be done when a celebrity posts something? With respect to your architecture, over a million newsfeed will be updated with a one go.
@sunnyshekhar862
@sunnyshekhar862 5 жыл бұрын
Take a break from daily routine work. And watch #GauravSen's design videos... You'll get both the idea and chull (read Motivation ) to work on your own projects. Kudos !! Great work Gaurav 🙌👏😊
@anastasianaumko923
@anastasianaumko923 Жыл бұрын
Very elaborate, thank you! 🤩
@prachiprakash
@prachiprakash 2 жыл бұрын
@Gaurav Sen I think the timeline/newsfeed service and the notification service is a bit mixed up here
@ayushiajit3088
@ayushiajit3088 3 жыл бұрын
Hi Gaurav, First of all excellent work on the videos :) I have a doubt on the DB selection, so basically what i am understanding is when we need to store information about user we may use Mysql cause of strong relationships etc but since the content [activity] of the user is kind of unstructured wouldn't it be better to use NoSql? by unstructured i mean, we may or may not have caption, may or may not have images, instead can have videos, or comments in that case can be recursively long..Please correct me if im not going in the right direction! Once again awesome work :)
@apoorvbedmutha457
@apoorvbedmutha457 Жыл бұрын
imo it'll be better to have a combination of noSQL and RDBMS for example tables which need to be regularly updated such as no. of likes must be kept in a noSQL DB whereas things like content of a post which are not changed so frequently are better to be stored in RDBMS
WHATSAPP System Design: Chat Messaging Systems for Interviews
25:15
Gaurav Sen
Рет қаралды 1,9 МЛН
System Design: TINDER as a microservice architecture
36:41
Gaurav Sen
Рет қаралды 1,2 МЛН
Noodles Eating Challenge, So Magical! So Much Fun#Funnyfamily #Partygames #Funny
00:33
Data Consistency and Tradeoffs in Distributed Systems
25:42
Gaurav Sen
Рет қаралды 191 М.
Design Twitter - System Design Interview
26:16
NeetCode
Рет қаралды 530 М.
Instagram System Design | Meta | Facebook
16:38
ByteMonk
Рет қаралды 43 М.
System Design Interview: TikTok architecture with @sudocode
45:35
20 System Design Concepts Explained in 10 Minutes
11:41
NeetCode
Рет қаралды 1,1 МЛН