Google system design interview: Design Spotify (with ex-Google EM)

Would you like to be the HOST/INTERVIEWER on these videos? Get noticed, meet top engineers and earn some $$.
If interested, email [e-mail] or book a quick chat with me here: calendly.com/tom-is-here
Today's mock interview: "Design Spotify" with ex Engineering Manager at Google, Mark (he was at Google for 13 years!)
Book a coaching session with Mark: igotanoffer.com/en/coaching/t...
Chapters:
00:00 Intro
01:16 Question
01:51 Clarification questions
04:24 High level metrics
10:05 High level components
13:18 Drill down - database
19:30 Drill down - use cases
25:00 Drill down - bottleneck
37:30 Drill down - cache
35:02 Drill down - load balancing
38:00 Conclusion
40:55 Final thoughts
About us:
IGotAnOffer is the leading career coaching marketplace ambitious professionals turn to for help at high-stakes moments in their career. Get a job, negotiate your salary, get a promotion, plan your next career steps - we've got you covered whenever you need us.
Come and find us: igotanoffer.com/

[@IGotAnOffer-Engineering]

Get 1-1 system design interview coaching with FAANG ex-interviewers: app.igotanoffer.com/coaching/tech/

[@EstheticallyPathetic]

This video is a perfect example of how things should be explained. The way Mark has explained entire design is commendable. Kudos to the guy interviewing for being so patient and polite.

[@yaronnir]

it's always best to start the interview answer and define the "Functional requirements"(FR) and the "Non-Functional Requirements" (NFR) that are needed for the design. NFR could look like this for this design: 1. low latency 2. high availability 3. secured connection etc.... this helps to flush out point of failures, and bottlenecks early in the design.

[@bradfordsuby8064]

Pretty sure most interviewers asking you to design a system are going to all expect the same NFRs because that's just the way of the world. If it's not low latency and high availability, then it's just not going to be a good product.

[@shpluk]

Just the fact that the interviewer could shut up and listen to the answer makes this interview great. There is nothing otherworldly about design interviews, not much has changed or invented in the recent decades the only issue in my experience is that people can't just sit and listen, they'll be constantly asking questions, breaking up the train of thought, I'd say its a tutorial for the interviewer and not the other way around)

[@CommentGeneric]

Are you talking about the interviewer asking questions, or the interviewee? It's great for the interviewee to continually ask clarifying questions - it's more annoying if the interviewer is constantly asking questions, but there still needs to be a dialogue.

[@shpluk]

@@CommentGeneric awesome username you have there 👍 Yep dialogue is the key

[@adennis200]

Im still a junior but I remember some classes that featured system design and watching this interview brings up a lot of memories. What I also love about that is the "doing things from scratch" part. When you're dealing with system design, it usually means you're creating something new, a new app, a new service etc and that's always an exciting endeavour

[@IGotAnOffer-Engineering]

@adennis200 congrats on our most liked comment! We're actually looking for a new Host, it would be about 15hrs work per month, would you be interested?

[@abanerjee3704]

The elegance with which Mark explained it 🤌🤌. Exquisite!!!

[@sevilayozt]

great video to watch before an interview for any position in computer science fields...

[@sachinsingh1956]

Mark does a great job of explaining the different aspects of design in a clear and concise way.I really enjoyed this video,keep going on man 🤟

[@IGotAnOffer-Engineering]

Glad you found it useful :)

[@nintendu64]

Mark is only 24

[@conphident4]

This was the most realistic System Design interview video I've watched.

[@SeviersKain]

This is pure gold, explains almost everything when you need to learn what a system is and how it functions...very very useful !!! Thanks man !

[@IGotAnOffer-Engineering]

Great to hear!

[@pablopablod1240]

What a pleasure it's to listen to this kind of people and the way they design solutions, they make it look easy but it takes years of experience to abstract like that

[@freezefrancis]

That was a nice one. I like how Mark evolved the design.

[@chocobitties2595]

Like many others I was designing my own version alongside Mark's and I think the area he was a little weaker in (which he himself admits) was load balancing. My background is systems administration so I may have a different perspective on this. I think going back a few steps, chunking the data also serves an important role in the load balancing process. I would have songs chunked, from every retrieval source, so that as soon as the user presses Play, the song begins playing, and playback should always be an instantaneous process unless the servers are over capacity, which can occur because some song or album has gone viral. I would structure the web server load balancing so that client apps attempt to contact the server geographically closest to them first and utilize GSLB (global site load balancing) which combines layer 4 and layer 7 load balancing, as I/O capacity or concurrent connections (the two metrics I would prioritize) reach a threshold. Again, when talking about load balancing, it's important to determine what happens when maximum capacity on all servers is exceeded. When this happens in my design, the system will issue "tickets" for data chunks, served in the order they arrive in. This is where song chunking comes into play. Because we are chunking the MP3 data, we can still serve the first chunk of the song from the nearest available server as soon as that I/O is available, further ensuring near-instantaneous playback upon pressing the Play button. The rest of the song then has some time to download and cache to the client device, reducing the number of interruptions and pauses in playback due to bandwidth and concurrent connection overages.

[@kento8453]

Can you explain more about these “tickets” in LB

[@chocobitties2595]

@@kento8453 Yeah, think of surge queues. Surge queues are essentially placeholders for a pending connection that occur when load-balanced services are overloaded. Amazon's elastic load balancers (ELBs) for example have a spillover configuration that allows excessive requests to be dropped. With a combination of chunking and surge queues with spillover protection, you can continue servicing requests and the impact is only mildly noticeable from a client perspective.

[@RenanOliveira-cl2pr]

Yeah, I didn’t understand what are tickets.

[@simvo7802]

Why do you need to chunk the data instead of just streaming it? Streaming already sends data over time in minuscule chunks, so you can play the song immediately and don’t need to “find and assemble” the other chunks. Especially since each chunk takes time to download, but stream bits are instant one after next, how would chunks be a better solution here?

[@chocobitties2595]

@@simvo7802 Uhh streaming isn't a zero I/O operation that spends no time finding and assembling. Quite a bit of resources are involved in streaming, unless someone has created a perpetual motion machine already that I wasn't aware of. In this house we obey the laws of thermodynamics! Please don't delete your comment btw. You can reference it later sheepishly. We all are at different stages in the learning process, and a bit of retrospection can be refreshing.

[@SafetyLast-_-]

Great content, thank you! This channel should have more subs.

[@KShi-vq4mg]

This is a great session. This format works when interviewer is a good listener and allows Mark to finish what he has to say, put a logical end before transitioning to next stage of SD or asking questions. and that is great. Can you do a session where the interviewer is constantly interrupting? you neatly define stages of the SD interview and its a flow we (interviewees) would like to get into. but more often than not, interviewer doesn't wait for you to finish a topic.(usually non FAANG companies) they just want to get into details of a component. or more often than not ask "why". I personally find it hard to transition from "answering their question"(which could take easily 2-5mins) to getting back into original format I had planned for the session. and because I'm unable to logically complete i fail the interview.

[@jai_ver_rb17]

A few things I did along side while understanding Mark's POV: 1. I would usually introduce DNS geo routing earlier in the stage to route to the nearest LB 2. Also worth to have a Metrics collector that can always keep track of HITs of top 100 (or emerging hits depending on BI) per region basis in some form of a max heap and then have a scheduler to periodically walk through them to ensure that nearest CDNs are hot loaded / prewarmed with them. Reading from S3 is very slow and I would usually find other alternatives instead of chunk reading in an instance memory. Packet roundtrips can be costly especially in use case of streaming. 3. I also split the durable storage into two - user data storage (less frequently used in comparison) and songs metadata storage - this way DBs can be fine tuned for workloads. 4. If I told S3, I would also mention cross region replication just to touch it a bit and indicate that I was thinking of a DC going down entirely.

[@BLACKVOIDLYRICS]

You just said all what I thought about during this video 😅 Good point

[@startup_cult]

I knew every single measure and strategy which Mark presented here. But I dont think I would have been able to present it the way he did with a gradual continuous increase of complexity. Awesome answer Mark. I wish I could get interviews to be able to deliver these answers, Im good at that.

[@Leo-056]

Probable scope of improvement - I really appreciate Mark's design. He's really amazing and for what he has done in the past I just respect him so much. I just think that (like again please don't think I am trying to prove something I am literally nobody ), he went a little easy on playing the song ! Users can experience down time due to network issues and all where the request might not be processed. (If I am wrong please let me know) We can use SQS (a queuing service) where if a song is requested and the server is busy also the user is having a weak internet connection or the connection just timed out. When he clicks to play - We store the id of the song and the user in SQS and a separate server reads the Queue simply and gets the song streamed to the user via the bunch of web servers(as per the video)

[@shashwatswain6103]

Recently had an interview with the same, hadn't come across this video then. I wish my design was as neat as it is here. The simplicity does help explain the data flow a lot better.

[@user-mz9gf8ux8u]

Really like how Mark communicates so effectively, and designs iteratively.

[@carlosluque4753]

Love the video! Before playing the whole video I played around with a design of my own and ended up with pretty much the same design with some variations that I'll add bellow. I think Mark and the Interviewer missed on digging a bit deeper into one of the main requirements Finding Music. Mark talks about performing the search operation directly from RDS. Taking into consideration the scale of the system, that would have been a terrible decision. With millions of users, the search function would hit the DB constantly and generating read queries in the RDS instance that stores its data on disk. Resulting in overuse of the DB and high latency. In my design, I went for a dedicated search service that is powered by a Search Engine such as ElasticSearch. This service is populated in the background asynchronously by a Consolidator service. Essentially, each time a data is added to the RDS (new songs, etc) an event or message is sent to a queue, the Consolidator Service would get the new data and push it to ES. Then the users can search very fast for songs using a highly optimised Search Engine.

[@user-uf3no9wg3x]

Yeah, like the "finding music" part pretty much implied an efficient search system. In general, I don't think this is a good video to prepare for a system design interview because the interviewer didn't challenge the interviewee about anything. The hard part is being able to justify your choices, explain tradeoffs, admit limitations and make major optimizations on the spot.

[@IGotAnOffer-Engineering]

@carlosluque4753 great input, makes a lot of sense.We're actually looking for someone to help me Host, it would be about 15hrs work per month, would you be interested?

[@mrsbootsworkouts]

Very informative, thank you! Start with simpler design and get buy in in order to avoid going on a tangent into details.

[@apoorvaranjan787]

Amazingly explained. Thank you Mark.

[@shailymittal6682]

Thanks mark! Very helpful to basically see how to communicate effectively calmly and enhance the design step by step. I would've added couple of more things here though 1. Separate the application servers for Querying the songs vs playing the songs (As you mentioned the load can be very different and the servers which are playing the songs will have high network bandwidth usage) 2. Add cache to the metadata server also (Songs metadata to maybe cache the songs which are recently, from some famous genre etc)

[@JohnKoepi]

15:00 it’s important to mention that the problem with the storage separation is not about the data im/mutability (actually you can update data even in the blob storage). Primarily it is about how inefficient it would be to store 5MB blobs in any general kind of OLTP database that will cut each blob into pieces of 2KB sizes, build a separate table (toast) for it with index over each piece. And only then you would want have more efficient streaming and completely different types of local and global caching. So separation makes lots of sense just because one data is in small pieces and another is in big.

[@lagneslagnes]

Most RDBMes have special blob support where they do not store the blob in the typical buffer pool with those small 2KB-16KB sized pages.Bu But your point is valid in general. So is the interviewee's. Under normal circumstances, immutability means it would not take part in any transformation functions of transactions/queries in the RDBM (even if it was stored in it). It would just be dangling as a reference to an opaque entity that never gets transformed. So if we move the opaque/large/immutable item to external blob store, you really do not lose anything (you still have refernces to it that take part in the RDBM queries/transactions).

[@alb12345672]

@@lagneslagnes Why not just install them in the filesystem and have links to the blobs? The filesystem is extremely optimized for that. Maybe put 1000 songs in each directory. Any database has a cost. You could also memcache the top 1000 most popular songs.

[@kSergio471]

Also, blob storage uploads different parts of a file into multiple machines in parallel. With RDBMS, to achieve the same you could split a file manually and do some kind of sharding - but too much manual stuff. Though it’s not that relevant for Spotify with low load of writes, but in general it’s a good reason why rdbms are not good for blobs.

[@IGotAnOffer-Engineering]

@johnkoepi thanks for your contribution, you make a strong point. We're actually looking for someone to help me Host, it would be about 15hrs work per month, would you be interested?

[@o-super2744]

That's interesting to watch. The design looks very similar to the one I produced during an Amazon Interview with the Load Balancing, Cache and Server Geo Localization. I was feeling good as well about my interview however I have failed it. The most dramatic part about failing the interview is that we do not have any feedback on our mistake to improve on. The only mistake I see is that at the last test, I did not write down one of the requirements and when I finished coding the interviewer told me he said the opposite about this particular requirement and I had nothing to back up / verify who was right. So if you are about to go through an interview, lay down on paper all the requirements, validate them and then proceed to the coding part. Good luck out there.

[@bradfordsuby8064]

That's the pain of today's society with development interviews - no freaking feedback. Just "we moved forward with someone else".

[@yiannig7347]

It was nice that the interviewer just listened, and the interviewee presented a simple design. However, in real FANNG interviews, especially for Senior roles, you're expected to go into more detail, and the interviewer usually challenges your decisions.

[@LCaaroe]

Yeah all of this teaches me nothing

[@ShashwatVerma]

Try InterviewJARVIS

[@TheEWMaynard]

To be fair would you trust your EM to clear a system design interview?

[@noobgam6331]

yeah right. No. FAANG interviews are pretty easy. for their standards this would more than suffice. But If I was the interviewer, I'd nohire this response just based on the diagram alone. "Designing spotify" without considering DRM at all would be a major red flag for me. Maybe for junior/middle role, but the average field standards seem to be very low now

[@victorz7748]

@@noobgam6331 what is DRM?

[@DeepWorksStudios]

Great value please create more content like this

[@olegnikitindev]

Great job! Enjoyed watching really much

[@TomasMisura]

thank you very much for the video! I was looking for something like this. I am not the best solution architect ever but I would design Spotify by very similar way but I am grateful for design ideas

[@rajeswaril3931]

He is just genius!! The way he is explaining is REMARKABLE!

[@victorgarcia3526]

This was super interesting! Thanks for bringing this content to us!

[@IGotAnOffer-Engineering]

Thanks for watching Victor!

[@TheMillw0rm]

this was great . maybe it didn't add a lot to me in terms of technical aspects but the way mark was connecting the dots was really interesting that's exactly what you expect from an top notch engineering manager

[@bigpoppa6658]

To be able to watch this for free is just amazing. Thanks so much

[@IGotAnOffer-Engineering]

our pleasure, Big Poppa. Hope you enjoy the rest of the videos on the channel (plus more coming in a few weeks)

[@bosuaandcarot]

this interview is much more useful than my 3 months university course 😜

[@loveUbleach4ever]

usually I get bored by tech teaching videos but this is the first one that I am still watching.

[@owenmurphy2022]

wow, love this mock interview ! I think a lot of this is covered in AWS Cloud Certifications !!!

[@trungcaothanh2725]

Thank you for making the video! it's so useful for me to know something that needs to prepare once I want to look for new opportunities 😀

[@IGotAnOffer-Engineering]

You're very welcome!

[@paulinemorare5509]

A very intuitive tutorial interview ❤❤❤

[@sahilchanglani8886]

It was great. Thanks for doing this. I would like to see more of system design interviews.

[@niufei8888]

Really nice video. Another point is to dress up during the interview. Mark looks like a CTO-level person. That first impression is really important when leveling.

[@gilbertsenyonjo963]

Thanks both of you!

[@kevon217]

Super helpful. Thanks for the walkthrough!

[@IGotAnOffer-Engineering]

Glad it was helpful!

[@AnonyMooseUK]

As an IT veteran, I wasn't that impressed that. Thought it was a bit rough. No API for example, no multiple levels of load balancing (geo dns load balancing in first tier, at least second tier across databases. No multiple database, for example, per country, or per letter, like users a* database, the b* database for stability. Web app would be the website and the Web player. Api would a tier below that, serving apps and the Web tier. Would also need a app tier, for data crunching for playlist generation, integration to 3rd parties services like lyrics, user auth and subscription billing and what about ato scaling, monitoring, alerting, emailing and maintenance. CDN setup for geo awareness for localised streaming. No mention on third party auth or app integration. Was okay, but not great. Sorry.

[@passofar]

I agree with you, in general but: 1. I don't see how you can go through all those topics in 1h. 2. Those should have been mentioned and let the interviewer pin point the interesting ones to persuit. 3. Things like API design is probably too much at start, but which types of API would be supported, their purpose and location in the layout, etc, should have been mentioned. 4. Things like multiple LB layers, multiple DBs to segregate the data, CDN setup and how to make the system elastic and observable, at least for me, are things that you approach after you have the initial layour of the system. Meaning, after the basic design is done, its easier to breakdown it down further and it feels more difficult to do it from the get go, at least for my level of skill :)

[@kennethcarvalho3684]

Very helpful. Thanks for sharing.

[@bradfordsuby8064]

For load balancing, you'd also want to think about having certain webservers marked for specific tasks. Though I suppose that would be more like having 2 services - your lookup service and your streaming service. That way you don't have to worry about the weight/priority of IO based balancing vs CPU. Then your lookup services are CPU based and your streaming services are IO based.

[@4alvis]

honest interview...just missed some chunking idea for songs at my opinion...btw great interviewer....always acknowledging with positive body expression 😊

[@andrewwwlife]

Great format. Thank you very much :)

[@atanasmarkov9016]

A great video to explain how solution architects work and what knowledge they need. Actually if you need to stream it would be difficult through cdn that will send the whole file. If you have own servers close to users I would just make some large cache and a small streaming server from local file system. As RAM is not that expensive now I would even suggest RAM disk for songs. So when a user needs a song it is read from some cdn(just to minimize hops for geo regions of own local servers), then file reader marks access and starts streaming just using file read and write to socket or the file is passed to the end user. Such a simple streamer/reader will be able to handle tens of thousands of connections on a single server. At end of day or some percent of disk full a job should just delete files ordered by last access. Some small local rdb can help for the marking as you will not have 1 billion songs on the local disk. This may even be better than commercial cdn as it is your own one and price is lower.

[@klapaucius515]

Woah, did you just build your own CDN? Great comment!

[@MarkKlenk]

I love seeing some of these creative ideas to balance scale and cost. Putting on my manager hat, I could see this being an optimization added to the system after getting it up and running and stable using an off-the-shelf CDN. Time-to-market is often more important at the beginning, and cost becomes more of an issue at scale, at which point adding complexity may be worth it.

[@michaelsheinman9852]

Interesting but I'm skeptical here. How would you make lots of servers that are close to the end user? And wouldn't your caches fill up very quick, and you would be replicating storing lots of data within each server's large cache. If we assume that songs are accessed randomly the caching wouldn't useless and we would fetch from CDN every time

[@marvinalone]

24:00 the clinet app should be able to get the chunk of the mp3 directly with some sort of token (expires in, for example,15 mins, per Id or user, etc.), the web server should not fetch the mp3 chunk data for client app but generates access token only

[@abhijitmaji4377]

Thank you Pastor Rob

[@danieljohnmorris]

This is gold

[@kdakan]

The audio streaming would better work with 30 sec. chunks of audio, instead of loading the full track, which can vary in length, from am minute long track to 20 minutes long. Also, ordering the artists and songs based on both relevance to the search terms and popularity and user's personal listening habits and preferences should make sense. Artist, song, and user metadata are all connected with relations on multiple vectors, like genre, mood, country of origin, and lots of unknown relations (aspects) that come up from machine learning, etc.

[@fisnik8965]

Great suggestions, addition to first suggestion -> I would split the audio into chunks ONLY in cases when the length of the song is above a threshold, example if a song it's 2 minutes (say ~ 2.5mb), it would make more sense to download it all with single query rather than hitting the Audio DB four times.

[@user-uu5xf5xc2b]

how would it work when the user wants to seek to a part of the song ? i am not familiar with networking so i'm curious how the connection stays for example during a 1 hour song. if it makes a new connection it'd be slow i guess but if the connection isn't severed then the server might get too occupied ? how do we balance these ?

[@PaulPendor]

Yeah, if you think about how Netflix works, the media is encoded to multiple screen sizes and resolutions to deal with varying network conditions, and then chunked. The client then retrieves the next chunk of the stream from the nearest edge server. So just encoding the media to the multitude of client conditions and then disseminating the chunked content to edge servers is a hugely interesting engineering case and solution.

[@MarkKlenk]

@@user-uu5xf5xc2b Very interesting question. I like the idea of loading less than the full song to start playing and then continue to "read ahead" while playing. This is a common practice also for videos and increases the chances that the full song is loaded by the time you start seeking around. Still, it's not perfect, and you can imagine a scenario where the user seeks to the end of a 5-minute song right away, resulting in a delay.

[@iFireender]

I'm not a system designer (yet), but from my work in my bachelor's and master's - while it's a good idea and most probably how it is implemented, this is 'getting lost in details'. This is the specifics as to how the streaming gets optimized; and if you have time to talk about that after the system is fully designed, sure, that's good. But with ideas like this, it's easy to go 'so there's an app, and it talks to a server, which talks to a database that stores.. and by the way, the database does this, and this, and this' - and then one hour is up and the rest of your system is underdeveloped.

[@binaryboy80]

Redis for heatmap. Timescale db for stats and playback history. Asynnc Via a message queue

[@codingfun]

To calculate the space required for 100 million songs, we can use the following steps: Convert 5 MB to Bytes: Since file sizes are typically measured in bytes, we need to convert 5 MB to bytes. 1 MB is equal to 1024 * 1024 bytes. 5 MB = 5 * 1024 * 1024 bytes = 5,242,880 bytes Calculate Space for 100 Million Songs: Multiply the space required for one song by the total number of songs. Space for 100 million songs = 5,242,880 bytes * 100,000,000 Convert Bytes to Gigabytes (GB): Finally, we can convert the total bytes to gigabytes for easier interpretation. 1 GB = 1024 * 1024 * 1024 bytes Space for 100 million songs in GB = (5,242,880 bytes * 100,000,000) / (1024 * 1024 * 1024) Let's do the calculations: Space for 100 million songs in bytes = 5,242,880 bytes * 100,000,000 = 524288000000000 bytes Space for 100 million songs in GB = (524288000000000 bytes) / (1024 * 1024 * 1024) ≈ 488.28 GB So, approximately 100 million songs would require around 488.28 GB of storage space.

[@nickpeterson193]

Great clip, thanks!

[@MrArihaan]

Pleasantly surprised he came up w the example of european punkrock, as I’ve been playing in european punkrock bands for a while 😊 nice choice!! (And it really is a bit of a niche)

[@mykalimba]

This was very interesting to watch. I am currently a Senior Software Engineer, and will probably end my career at this level as I'm quickly approaching retirement age. I've always loved getting my hands dirty writing code, and have never had any aspirations to advance to the level of an Engineering Manager (or Development Director, etc.). But while watching this interview, I found that my thinking was in lock-step with Mark's, and I found myself answering the interview questions with essentially the same responses. I even blurted out several of the same responses _before_ Mark answered in the same way.

[@PankajKumar6493]

How old are you? I'm also have similar thought process. Don't wanna go beyond Senior Software Engineer as I think it's too much stress. But that would mean I'll have to retire late.

[@leosilva0411]

Great video! Thank you!

[@yoniziv]

This was so amazing. Thanks for sharing

[@IGotAnOffer-Engineering]

Thanks for watching!

[@marjot87]

Great and interesting interview! AWS Cloudfront with S3 backend automatically pulls a file from S3 if it is not cached already so the webserver could return the mp3_link at the Cloudfront distribution endpoint and Cloudfront would take care of everything else.

[@cphoover11]

Yea I think he overcomplicated this part

[@kSergio471]

Won’t it be an issue that CDN does not authenticate the downloader?

[@rosmelylawliet3919]

I would have gone deeper on API specs (some endpoints, how would they work?), the searching algorithm (roughly, db indexes? some middle caches?), and audio service (streaming, shared cached besides CDN, loading all in mem takes time where the user hears nothing, and is costly in RAM, discuss alternatives). A way to deal w/ metrics (data pipeline, no need for too many details). Also, mention CAP, what would u choose and why. Normally, you will forget to mention things, and the interviewer will ask accordingly; but as mentioned, it is usually better to have your key points exposed w/o the interviewer needing to question you.

[@genegade]

Wild that he didn't think about bandwidth. This is the make or break number for any streaming service

[@arnavhazra8806]

This was so damn cool, as a rookie CTO this is a great transfer application of SD concepts to learn from. Definitely coming back for more!

[@jialx]

What is a rookie CTO

[@hamzaf19]

@@jialx Chief Technology Officer

[@jialx]

@@hamzaf19 'rookie'

[@revenez]

Very useful! Thank you.

[@pranjalsrivastava1191]

I feel that with a humungous list of 100 million songs, we can implement a separate search server for the search functionality like a Solr search. It will reduce the searching time by a huge margin.

[@SusilVignesh]

This session is really good. Thanks for the video :)

[@IGotAnOffer-Engineering]

No worries!

[@fariobross7831]

ex google cause he used AWS instead of Google Cloud

[@arpitvijayvargiya1371]

😂 nice catch

[@pixelsbyme]

😂😂😂😂😂😂😂

[@gmanonDominicana]

This is very good! Also the separate meta database could make possible to search for multiple language content in the user language. I mean; the songs are available in every language to every user where the user doesn't need to worry about knowing foreign languages. Only the developer needs to make the data available to the local language, including the meta data while the streaming media could be gobal regardless language.

[@IGotAnOffer-Engineering]

@gmanonDominicana good point! We're actually looking for a new Host, it would be about 15hrs work per month, would you be interested?

[@ocamlmail]

Thank you, useful.

[@roydonk2878]

It's funny that the interviewer is trying so hard to nitpick everything that the more experienced guy is saying. "He should have said up front why he was splitting the databases into two." There's so much going on and you're working through a problem you were just given 10 minutes ago, no interviewer is going to care about if he addresses it up front or if they have to ask for clarification. It's all part of the process

[@OneStopMusic.]

This is such an amazing and informative video. Keep it up guys

[@IGotAnOffer-Engineering]

Glad you enjoyed it!

[@OneStopMusic.]

@@IGotAnOffer-Engineering yeah. Keep it up guys

[@thebluriam]

The fact that he's bringing up specific kpop groups makes my day.

[@deathbombs]

23:40 I think downloading the songs to play will need TCP, so not sure what he means by reading directly from DB into the server as an optimization. 33:10 he flows from one idea to another very smoothly, but the cache talk feels like he's just listing whatever is top of mind as well and a bit rambling so structure would be good

[@huylearning2543]

this video is mind blowing, it teach me a lot, thanks you.

[@IGotAnOffer-Engineering]

great to hear, thank you

[@LuisRuizHalo]

Google engineer and still uses as reference AWS lol, poor GCP. Nice vid BTW.

[@just_A_doctor]

So what ???

[@zikomo8913]

Emotional damage

[@Kitulous]

​@@just_A_doctormakes you think whether gcp is inferior

[@mainagmuriithi2772]

@LuisRuizHalo it's most likely is because he knew Spotify runs off AWS so it was the most relevant cloud for the context

[@Shitopia539]

GCP is dead

[@adrianmh]

It is interesting to see that even a 13 year Google Engineering lead (guy's a BIG shot) has to think about an approach. Makes my own work so much more relatable. I like the fact that he was not given the question beforehand

[@miettoisdev]

also, definetly wouldnt go with streaming audio from the webservers - for both scalability and separation of concerns. a finelly tuned CDN (having price constraints in mind) would do the job.

[@ventice11o]

How on earth is Mark's design related to Spotify? It could be stated "since the data limits are not skyrocketing, we'll take standard web app with scalable databases, caches and a load balancer and go on". He didn't ask if we are allowed to deliver the MP3s (so that any client can download it right away) or we have to add some security, deliver the file by pieces and reassemble them on the client side. In fact, he didn't address the security at all. He didn't address the side of the artist that should be paid for the listening of his song and how this FR is compatible with CDN that offloads your web servers. As an interviewer I would conclude that the applicant is aware of standard scalable web app practices but doesn't look into important aspects of the app that may strongly influence the standard decisions.

[@CertificationTerminal]

Awesome!

[@tscung]

Pretty good one, I would add: - No SQL for songs meta and keep users data in the relational database - mention encryption at rest (object storage and DB) and on transit (SSL on API call) - As the system is read-only so read-replica

[@Netmould]

Hmm. From my point of view there are some things missing that I would expect you to mention during system design interview. First and most impactful on system design are service metrics, like reliability, responsiveness, availability, and so on. I do understand you kind of included 'apparent expectations' in form of initial question - we all have an idea about 'what you're expecting from 'Spotify' service, but at the same time you have to quantify those, because an 'idea' is just an idea, different people (stakeholders, clients) can have different expectations for the same idea. Few basic examples: - median response time (for every use case) - uptime/availability requirements - RTO/RPO Exact numbers for those will make a tremendous impact on any high-level system design. Second - constrains (you guys went through few of them, but kind of missed usual ones). I can't stress enough how valuable to understand your project (system) constrains at the beginning - it could be money, time, some government requirements, technology requirements/constrains, anything. Main thing is - you just have to understand you can't design ideal system (in it's 'final' form) and try to get there from the start. There would be iterations, growth, compromises, technical debt - and as a system architect you have to plan things around all of that. This is going in hand with my first point, examples: - do we have enough time/money to provide 99.9% service availability? - how and in what time/cost we can add additional features? There are more theoretical ones, but it would be nitpicking at this point. Some practical issues in final design I would point out: - pretty sure there's authorization service missing (you absolutely don't want to handle them with your main app). - you don't want to handle both search and playback on same service, not with those numbers. - you have to use LB at least for metadata load - there will be a lot more metadata, I would split it in two (at least) - user-defining and content-defining. - you have to add metadata to your CDN, its a part of core user stories P.S. storing/accessing/updating your data fast on this scale is quite a loaded question by itself

[@nukeu666]

Uptime/RTO questions would add design regarding the HA/DR architecture which would be an extension of this design right? What impact would response time requirement have? Maybe add more caches? Understanding time/money restriction is important when scaling but would not have impacted design in video right?

[@Netmould]

@@nukeu666 You're right. I tried to say that in this exact case you can kind of assume those answers, and his design wasn't wrong. My point was - in this kind of real-life interview I would love to hear about constrains and service metrics in a bit more detailed way (or at least skim over) - how they impact system design at all. About response time - there is big difference in architecture between 500, 50, 5, 0.5 and 0.05 seconds in response time (I would guess you know this as well). Until you propose/discuss exact values/ranges, you can't really make a decision on "how to make my system".

[@andremilton5835]

Absolutely. There are a lot of really important pieces missing in the design I would expect an architect to include.

[@MarkKlenk]

Wow! Thank you for so much careful feedback. This is really good. Your comment "I can't stress enough how valuable to understand your project (system) constrains at the beginning" really resonated with me. That is absolutely true. One of the things I look for when I'm doing mock interviews if the topic of response time comes up is percentiles (50th aka median, 90th, 95th, 99th), but I clearly missed following my own advice there. :) Also good call on splitting the metadata - I think they have different purposes, so it would make sense to keep them separate.

[@Netmould]

@@MarkKlenk Thanks for responding, I'm happy there were some useful bits I could provide!

[@johnhenry9876]

What I noticed missing was TTL or time to live or file expiration. That should be part of the API call as we dont want to indefinitely store songs in our CDN or in Cache. And really any reference to APIs or tracking of session state to be able to continhe where a user left off.

[@Shivnandak]

Elastic search might work really well for the metadata db. It should cover the storage as well as the search functionality.

[@michaszewczak7392]

10gb-100gb of data in DB is not that much. Indexes will do a trick there

[@durgeshchoudhary]

@@michaszewczak7392 there will lot of dynamic tagging involved for the songs, simple text search would not suffice here. Some sort of lucene index Elasticsearch/Solr etc would really help here for full text search.

[@developer4Droid]

I think you don't need a websocket connection for chunk loading. Both HLS and MPEG-DASH are working though HTTP protocol for this purpose

[@rembautimes8808]

I think concurrency and fault tolerance is a big design consideration. If a web server goes down will it take down N users . I’d probably look at adopting something with Erlang. Great content and appreciate the input . Joined as a sub

[@liubovdudnichenko8506]

thanks, was really usefull

[@bombrman1994]

this is so easy, I don't know why some make it sound like a big deal. I am not even a native english speaker and I can understand this fully and can do same thing with any other system design requested in an interview. The only problem is getting the interview lmao

[@nagitoyup6929]

This is probably for some junior engineers. It has very basic concepts. The questions were not too technical to kinda push the interview towards high decision making skills. This is just list all technologies in any saas , and connect them,

[@MarkKlenk]

Indeed, my solution was about assembling SaaS "Lego blocks" to solve a problem. I think that judgment calls on which solutions to assemble carry some weight. I definitely value that in interview candidates when I'm doing mocks, but I may also ask them to go a bit deeper in certain areas if we have time.

[@IsUserADuck]

It is a system-design interview, that's kind of its purpose.

[@jairajsahgal7101]

thank you

[@nsm3824]

Great interview style! Q. Which tool is Mark using for drawing diagrams/texts?

[@TieDyeFriday]

Google Drawings.

[@jt4351]

My first thought was "the level of confidence to question a senior ex-Googler". Then, I remembered that Google has put out some less than stellar solutions. All in all, Mark explained it beautifully and it must have been a joy to work with him.

[@JacobAsmuth-jw8uc]

Can you give an example of some "less than stellar solutions" that Google has put out? What specific Google products do you think suffered from poor infrastructure design choices?

[@namnguyen-kc4kp]

For worldwide scaling, there's no need to save favorite songs in a local replica. We have Content Delivery Network (CDN) already serving that function.

[@disen135]

In real world you will be bombarded with a lot of questions.

[@carlesg0n]

great video mate

[@IGotAnOffer-Engineering]

Thanks for the comment, glad you enjoyed it

[@jayceazua723]

You could queries and can use O(1) operations partitioning in S3. Just wanted to clarify that.

[@YuriPolchenko]

oh, thats useful. Thanks!

[@shishirsonekar5661]

Great job done. I only have one question to understand. Did you miss talking about the security (authorization & authentication ) of data (music)? Or it is out of scope for this interview?

[@deathbombs]

Some observations: The beauty of system design is there's no right or wrong answer. I would've tackled these very differently(not mentioning replication until we are optimizing the design,...) but both methods would work as long as it makes sense I love his sense of detail, describing blob storage as linearly scaling, the songs being immutable are read only, the storage needed for various encodings... These make total sense but he really spells things out clearly.

[@SnehilBhushan]

I would have loved to see how albums, playlist, recommendations and song radio world be stored and served.