6:08 Search index - partitioning issues 8:22 Adding items to carts: Real time updates, Version vectors, CRDTs 13:42 Placing an order: Writeback Cache 16:35 Streaming/batch stock updates 17:55 Architecture
@KishoreShashank8 ай бұрын
Jordan, your content is super authentic and your style of presenting is very practical without any bullshit. Keep up the great work!!!
@jordanhasnolife51638 ай бұрын
Thanks Kishore!
@edmondlebeau2 жыл бұрын
Great content! It's been a productive summer for you Jordan Love the jokes too
@jordanhasnolife51632 жыл бұрын
Appreciate it Edmond, thanks for watching! Always helps that no one gets anything done at Google, who knows what would be happening if I were at Amazon haha
@dearozero3631 Жыл бұрын
I liked just for the intro.
@ninad1108199111 ай бұрын
Would a cache be a decent option for the management of carts? A) it’s transient data B) updates and fetching would be quick 3) Cost is not going to be a huge problem considering that we’re storing pretty less data on the cache
@jordanhasnolife516311 ай бұрын
Seems very reasonable to me!
@olabanji2 жыл бұрын
I dig your vibe throughout this video bro! Looking forward to more of this.
@jordanhasnolife51632 жыл бұрын
Thanks man! Stay tuned for 5pm EST today in that case! Glad to have you as part of the community.
@curtissitruc68942 жыл бұрын
After this intro I neeeeed the AWS system Design Video! Time to sub and wait Edit: Also am a big fan of the the references to other videos
@jordanhasnolife51632 жыл бұрын
Haha if I do that it'll have about 50 parts
@sribharathi3 ай бұрын
😊
@sribharathi3 ай бұрын
😊😊
@samanwayamaurya2838 Жыл бұрын
Hi jordan , in the product table (mongoDB) you talked about , are you sharding the product table itself or you are sharding the search cluster built over this product table? Also is this doable to shard the cluster indexes in mongodb.
@jordanhasnolife5163 Жыл бұрын
I think that both would be required, and yes.
@yiweizhang286 Жыл бұрын
Hey Jordan, I'm reviewing this design and have a question about the usage of message broker, specifically in the product update/search part. The product service updates the mongoDB table which then sends change-data to a Kafka queue to update the search index. Why do we use a message queue there between MongoDB and Search Index, but we don't use it between the product service and MongoDB? I'm trying to understand when to use a message queue and how to justify the usage. Again thanks for the great content, this video is specially clear and I learned a lot.
@jordanhasnolife5163 Жыл бұрын
Hey Yiwei! Thanks! I think the message queue was used here for between the db and search index because afaik change data in mongo has to be sent to a queue :) Could be wrong though, you're right in that it's not inherently necessary in this case
@yiweizhang286 Жыл бұрын
@@jordanhasnolife5163 thanks Jordan, my question is more about why we choose NOT to use a MQ between the product service and the MongoDB?
@jordanhasnolife5163 Жыл бұрын
@@yiweizhang286 I'd say when you can avoid having more components you probably should, it just adds complexity
@c_hlee6 ай бұрын
Hey Jordan, love your channel! This is my first time ever doing system design and your vids help so much. One question -- The way that the diagram has the LB is a little confusing, wouldn't an API gateway direct traffic to the correct service, and then LB placed in front of each service to handle distributing traffic between the replicated server instances? Or can a load balancer alone accomplish traffic directing too?
@jordanhasnolife51636 ай бұрын
Yeah I tend to lump them together, but you're correct!
@c_hlee6 ай бұрын
@@jordanhasnolife5163 thanks for the response and helping me to become the best giga chad I can be 😂 lol
@anupamdey4893 Жыл бұрын
When is the AWS System Design video coming ? We could have a series on AWS where you do an in-depth coverage of various AWS services.
@jordanhasnolife5163 Жыл бұрын
Once I start smoking crack (shouldn't be too long at this rate)
@mann8737 Жыл бұрын
Great video Jordan! Keep going! I have some questions tho. Why did you use MYSQL to store the orders? How about something like Mongo? Also, if SQL is the choice, how would the schema look like since we are partitioning the Kafka queue by product id? Thank you 🙂
@jordanhasnolife5163 Жыл бұрын
Yeah I think that you could use another DB than MySQL since we don't really need transactions for that part. As for your second question, I think maybe something just like orderId productId quantity would work for schema.
@rajrsa Жыл бұрын
Hey Jordan, Thanks for all these videos! I had a question regarding the choice of technologies in the architecture. At my job, I have used things like MongoDB, Dynamo, MySQL etc. but the applications that I have worked on usually had less than 1k users per day so we didn't really run into many concurrency or scaling issues. Now that I know about stuff like Riak, Flink etc. it makes sense to use them but in practice I have no idea how to implement them. In this case, do you think it would be a good idea to even talk about them in the interview? What if the interviewer asks me to double click on one of these technologies? I probably wouldn't know the answer... Wondering if it will be a good idea to stick with what I know and mention stuff like "btw, I heard some technologies that support X, I would love to explore them." etc. during the interview.
@rajrsa Жыл бұрын
Also, are you really this funny in real life or do you think about this content before making these vids? lol I bet you pull like nobody's business because you're pretty funny!
@jordanhasnolife5163 Жыл бұрын
1) I'd probably say: even though I haven't used these as much in my professional career, I think that theoretically for this type of problem we want want to use x 2) Still no bitches 😭
@metalalive2006 Жыл бұрын
Great content. I am learning the design and have a question : Whenever the order service in the diagram (since 17:54) receives a order request, does it require another (relational) database for saving the order first, and then push the same order to Kafka queue for inventory function (which includes another set of databases) ? so in the diagram there would be another link from `order service block` to `SQL DB icon` with `Kafkaqueue icon` side by side, does that make sense ? thanks
@jordanhasnolife5163 Жыл бұрын
I don't think it requires it, but you certainly could do that. In the end you can give the order an ID before placing it in the Kafka queue so that you can reassemble the order again later from the individual items.
@metalalive2006 Жыл бұрын
yea, it may be good to assemble the same order again for reasons like status update or invoice (maybe integration with other accounting software if required) ... etc
@metalalive2006 Жыл бұрын
Hi, Jordan, at 13:48 , you described that atomic operations and locks in database might slow down the application when a lot of customers are placing new orders concurrently, one of feasible option for the improvement is write-back cache. Can you provide links to relevant online articles or discussions describing similar write-back-cache idea ? (I watched the ticket-master design video from this channel, just wonder any other discussion which describes similar idea) , thanks again.
@soumik769 ай бұрын
Hi Jordan, when would you prefer CRDT vs version vectors for merging data between nodes?
@jordanhasnolife51639 ай бұрын
Well version vectors don't allow us to do any merging at the database level, they do allow us to detect conflicts though so that we can store siblings and do merging on the user level. I'd use a CRDT if they can handle merging the data, but if it's too complicated, version vectors and user merging may be better.
@soumik769 ай бұрын
@@jordanhasnolife5163 thank you
@blackcatcaptain2022 Жыл бұрын
version technique update is called optimistic locking
@АртемМежеловский Жыл бұрын
If someone made order then I think we should reduce amount of orders in product DB but how to do that?
@jordanhasnolife5163 Жыл бұрын
I was proposing we do it in microbatches - have a stream processing framework collect them and every 10 or so minutes update the count
@franklinyao38332 жыл бұрын
Can I mention DB locking to deal with concurrent editing on the shopping carts? Although lower throughput.
@jordanhasnolife51632 жыл бұрын
Definitely a solution, but let me remind you that it's still the case that you'd need something like a websocket to propagate the changes, unless a special database set structure was used. Agreed on the lower throughput part.
@kendochen62262 жыл бұрын
like your attitude to those asking to design AWS. 😄
@jordanhasnolife51632 жыл бұрын
Their wives are mine
@PhantasyAI07 ай бұрын
Hey jordan can you do a system design for something like shopify?
@jordanhasnolife51637 ай бұрын
Can you try and elaborate on which part of Shopify you think is tough to design?
@PhantasyAI07 ай бұрын
@@jordanhasnolife5163the database and the multi tenant aspect of it. Im trying to improve my database design skills. Would you say nosql would work better for something like shopify?
@xavierzhao34602 жыл бұрын
New potential topic to cover: eBay (Auction System)?
@jordanhasnolife51632 жыл бұрын
Will look into this as well!
@jordanhasnolife51632 жыл бұрын
Debating whether it's fundamentally any different than something like a chat application - as far as I can think of for now it seems somewhat similar, with the exception of not allowing illegal bids, meaning that you'd probably have to use atomics on a database if you want synchronous responses, or if stream processing is okay you could use that as well.
@tionx1266 сағат бұрын
Slid into the interviewer's wife's DMs. She hit me with a LC hard
@jordanhasnolife51634 сағат бұрын
Is that why you hit her with a hard of your own?
@Jashan771142 жыл бұрын
Can you recommend other resources for system design?
@jordanhasnolife51632 жыл бұрын
Well let me answer your question with a question? Is there anything else you want me to cover? If not and you just want something more formal, designing data intensive applications
@Jashan771142 жыл бұрын
@@jordanhasnolife5163 maybe videos on low level and high level designs both
@barzakovich2 жыл бұрын
I understand shit about system design, here only to enjoy jerk jokes
@jordanhasnolife51632 жыл бұрын
I'm not joking 😈
@grabarzowaty2 жыл бұрын
Keep the jokes. Don'e be a chicken.
@jordanhasnolife51632 жыл бұрын
Alright but if I get fired you're gonna have to pimp me out
@grabarzowaty2 жыл бұрын
@@jordanhasnolife5163 Best fire is HR violation fire. In the worst case you will monetize om yt your personal life dramas.
@jordanhasnolife51632 жыл бұрын
@@grabarzowaty Haha I think you may be slightly overestimating how lucrative youtube is compared to software engineering, but perhaps one day I can live the sigma male dream lol
@franklinyao38332 жыл бұрын
I am the first viewer. So what are the to-make videos? Options: - Payment like digit wallet or Paypal as an example of using transactions. - Real-time gaming leaderboard ...
@jordanhasnolife51632 жыл бұрын
Thanks for the suggestions, I'll think about those and if I feel the solutions are unique enough I'll probably go ahead and add them! Thanks Franklin!
@2tce2 жыл бұрын
@@jordanhasnolife5163 Yes. Leaderboard would be nice to explore. Then I'll like to know about an online chess game (or any online board games). I believe CRDTs will come in handy here.
@jordanhasnolife51632 жыл бұрын
@@2tce Will try to make a gaming video at some point, but do note that there is a leaderboard video already up (#19)