Distributed Sorting - Google Interview Question - Algorithm & System Design - Full 2 Hour Interview

[@QuanticDev]

I will publish my free courses on more advanced topics on quanticdev.com soon. Check them out when I publish them. I greatly appreciate any feedback. Edit: I realized that I failed to mention inter-node communication. A message queue like RabbitMQ or even better, streaming message passing software like Apache Kafka would fit into our use case very well. They both have options for high-availability and message persistence. Hence we can sacrifice 3-4 nodes to be our dedicated message queue handlers.

[@RAHULSINGH-zg9hp]

Kafka would be a better option since it gives default storage of 7 days which can be tuned as well

[@QuanticDev]

@@RAHULSINGH-zg9hp I agree. It's more work but is a better long-term solution.

[@svenschmidt1735]

Thanks for the video, definitely dense with lots of important information. I didn't get the distributed partitioning with 1000 value ranges at all, so a follow-up on d.p. would be greatly appreciated. Thumbs up

[@QuanticDev]

Yes I skimmed through it to keep the vid short. I will prepare an animated vid for it as it is best explained visually.

[@ShubhamSingh-ku2ow]

Amazing content!! Thanks for bringing such high quality material. Looking forward to many more like this!!

[@sangrammallick3304]

Awesome videos..will be waiting for more. Wish I had option to add more likes

[@lidajones6983]

Awesome content and explanation

[@SergeyZarin]

great lesson!

[@hannnah689]

Thanks for the vid, super useful!

[@TopToBottom25]

Good one

[@ducatidiavel4021]

Do the remaining 997 nodes read data packets from the database in parallel ? How do you ensure that each of those 997 nodes read their segment of the data correctly? Or do the control group issue these triggers sequentially ? I'm asking this cos assume the control group issued parallel triggers with exactly some way mentioning which segment of the data that specific node must read, what happens when it results in a read failure or the Node goes down at the time of the read. Do you receive some form of acknowlegement from the node ? Very nice video btw. Your videos are amazing. Its a wealth of info.

[@QuanticDev]

Good point. They would read the data in parallel to keep things simple. In this case, just checking for heartbeat is not enough. The nodes must acknowledge the successful read. Otherwise controller group can handle that node as a failed one, and redistribute its range to all other nodes as usual. I didn't think of read failures. There are probably many other smaller things that I failed to think about. I'll add them to the article version of this video when I finish it. Thanks for bringing it up!

[@ducatidiavel4021]

@@QuanticDev Thanks for explaining this :)

[@akashagarwal6390]

why not thinking of external merge sort in the first part?

[@lquixada]

great content indeed! it's hard to find video explaining how to scale algorithms with multiple machines. Do you know any other good resources to find content like this one?

[@QuanticDev]

I'm yet to come across even a single video like this one that is complete with the code, implementations, and animated descriptions. Mostly what I find is generic whiteboard systems design stuff, which is probably sufficient if you never actually do design but only acquire partial information as interview prep.

[@lquixada]

@@QuanticDev thanks for the quick response. Is there any good sites, repos or books that you would recommend?

[@QuanticDev]

@@lquixada "Designing Data-Intensive Applications" is the holy grail of all systems design books. The systems design section on the "Elements of Programming Interviews" is also pretty good, if you just want to learn stuff as an interview prep.

[@SiddharthKulkarniN]

This video is very interesting. Have a like and a Sub.

[@QuanticDev]

My pleasure.

[@大盗江南]

Thanks so much, buddy.......senior level interview, what does this mean , how many years of experience is considered as senior ? i am gonna do a sde 2 interview. i really hope that for system design question won't be so hard.. could u please give me any suggestion plz ? Thank u in advance.

[@QuanticDev]

I can't define seniority by years but if you are going for sde 2, solving the ever generic systems design questions should suffice (design a tinyurl clone, etc.). I don't actually know a great YTer with great resources on systems design actually. If you find any, let me know so I can recommend in the future.

[@大盗江南]

@@QuanticDev there is an Indian guy making great video called Gaurav sen 😊

[@大盗江南]

@@QuanticDev thank you buddy 🎊

[@QuanticDev]

@@大盗江南 my pleasure

[@QuanticDev]

@@大盗江南 I'll check his channel out. Thanks.

[@franzweitkamp]

Additional question I would ask: When can I start

[@QuanticDev]

If you can get 3 quarters of this interview right, I would directly say "as it suits you".