Design Youtube - System Design Interview

🚀 neetcode.io/ - Get lifetime access to every course I ever create!
Lets design the high-level architecture of youtube - similar to how we'd tackle this in a system design interview.
🥷 Discord: / discord
🐦 Twitter: / neetcode1
📷 Instagram: / neetcodeio
🎵 TikTok: / neetcode.io
🐮 Support the channel: / neetcode
⭐ BLIND-75 PLAYLIST: • Two Sum - Leetcode 1 -...
💡 DYNAMIC PROGRAMMING PLAYLIST: • House Robber - Leetco...
0:00 - Intro
0:15 - Background
1:30 - Functional Requirements
1:55 - Non-functional Requirements
5:35 - High-level Design
14:08 - Design Details
17:00 - KZbin video playback example
20:30 - Protocols
design youtube
design netflix
design hulu
design video streaming
system design interview
distributed systems
object storage
vitess
database sharding
#system #design #interview

[@NeetCode]

Let me know if you guys want more system design content - I've been wanting to do more and have a lot of ideas, so hopefully you all are interested! Let me know, and feel free to drop a like 👍 =) 🚀 neetcode.io/ - Get lifetime access to every course I ever create!

[@criostasis]

Design a system that pushes notifications with image and/or video to a mobile device using local storage only. No AWS or other external servers allowed. Once you have that done, let Eufy know, bc they f’d up big time 😂

[@jondogerman4195]

Thanks for the video! Could you recommend any good resources for learning System Design at a level like this? Thanks again

[@nikhil_a01]

More systems design content sounds great and if you want to do it, then you should!

[@wondays654]

Commented this on an old video but the api for neetcode is not working. I am unable to sign in with both Google and github.

[@AurelianoShowsTheWorld]

GIVE ME MORE

[@ankitsharma_yt26]

Thank you Mr beast for system design 😊

[@aelam02]

Neetcode! I've been applying to jobs for 7 months now since my graduation and struggled to get interviews as my resume was fairly unimpressive. I kept telling myself that I just needed to make sure that I killed any interview I DID get and your channel is what helped me to do just that. Got an offer today that blew my expectations out of the water. Thanks for all you do for all of us and keep it up!

[@NeetCode]

Congratulations, I'm really happy for you! 🎉

[@mayangjsoa]

are u fired

[@VY-zt3ph]

Hello Alex, what company you got the offer from??

[@aelam02]

@@VY-zt3ph Infineon Technologies, a German semiconductor company.

[@VY-zt3ph]

@@aelam02 u live in Germany brother??

[@alexanderschwaighofer1550]

Awesome Video! Thank you for going into detail and explaining what every part of an example systems architecture is accomplishing and why it's useful. One little nitpick -> (22:00 ) you say KZbin is using TCP which is (for quite some time now) not true anymore. If you look closely at the "client-protocol" used to send the video/audio data (you show that at 18:29 ) you will see that KZbin is using QUIC (HTTP3) which is built on top of UDP instead of TCP. It includes most of the functionality of TCP based HTTP protocol but is technically not using TCP. Just wanted to point that out in case anyone was wondering... Great video though, keep up the neet work!

[@j8ahmed]

Bro! Genuinely such eye opening content. Thank you! I am no where near solving problems like this yet but it's really helpful getting exposed to all the realistic considerations involved in designing large scale systems.

[@jasonswift7468]

Yes, we need it. Hope to have other content appear in the course, such as machine learning, data analysis.

[@jessicakoch2331]

seriously your explanations of complex topics are amazing, i’m so glad i found you! I was going cross eyed looking at graphs on Leetcode and their explanations were not helpful at all. Officially subscribed, thank you so so so much!

[@victorz7748]

First video recently that I haven't skipped for one second. Great work!

[@varshasingh1299]

We want more system design video...I love all your videos ❤️

[@muzamilahmed3127]

I think you've missed most points in the watching part. User can change the resolution for that we need multiple videos. Also, different OS will have different file formats like iOS, and android. so if there are 4 resolutions with 3 formats, 4*3=12. Each video should be stored as 12 different videos. Also, we need to store videos in chunks in the cloud storage. We will use main Dash File which contains all the video's information like chunk number, url to that chunk. So, if user skips to the 4th chunk, we'll check the dash file to get the url for the 4th chunk and will make the request for that particular chunk having specific resolution and format.

[@bumq5514]

Getting into system design and one of the best vids I've seen so far, great in depth detail

[@shuysnoopy]

These types of videos are such a blessings since it goes deep into a particular usecase or several design descisions which would be rather difficult to gain experience. I can try to learn as many frameworks as possible, do as many projects as possible, but if I dont have such real-world (or close to real-world) lessons like these I might be making mistakes that aren't so visible to the naked eyes.

[@themayhemera3046]

💯

[@imrajsingh8983]

such beauty looking at all the different high level trade offs and considerations going into designing an application that I have been using daily for the past 15 years

[@eswarprasad9773]

The fact that YT isn’t streaming a data source in complete but making HTTP requests to load chunks of video/audio and taping them at the client end BLEW MY MIND !!! my whole life seems like a lie , lol Great video, Thanks mate

[@dmytro-kazarinov]

You definitely killing it! Love this content!

[@akultyagi2818]

First i thought why is MrBeast is teaching system design lmaooo

[@atNguyen-jg5cp]

Awesome video ! Your explaination is easy to understand and it's detailed enough for a video like this. Would be great if you do more videos about system design

[@biro2200]

On the profile picture updating async, you could always solve that by referencing a latest.png and then only updating the object store to change latest.png to reference to the newest image and hence the changes would propagate much faster than updating 1000x video documents

[@Driver0077]

amazing explanations! thank you so much for your time

[@Souljacker7]

You are a great teacher. Thanks!

[@auroshisray9140]

Loved the conciseness of the video

[@sameer_sah]

2:14- Reliablity is defined as ability of a system to work as per specification inspite of extenal (high loads) or internal (faults) factors. If our non-functional requirement is videos to not disappear after upload , we are essentially looking for the system to be durable (not reliable).

[@MaxFung]

good point

[@wa_ka_su]

Please more system design content.🙇

[@Ogrizkov]

Thanks for amazing design throughout

[@sina7601]

You can easily have a user service and store all of the users information including the uploader there, hence you don't need to have it stored alongside the video but just have a userId as reference. This way you have decoupled your systems and follow a more domain driven design and if the user data changes you don't need to go through 1000 of videos just to update a profile picture. At the beginning of every system design interview/document you'd better write down your domains in order to see what are the different services you need before jumping into the design.

[@pramalvi]

Yes, it can be cached as well.

[@MaxFung]

i would prefer a user system as well

[@sina7601]

Putting huge video files in a queue is too much and very costly and slow but instead you should have it in S3 and then just populate an event to Kafka/RabbitMQ/etc which is consumed by the encoding service that in turn knows it has to fetch video with id: XYZ and start encoding it.

[@vyshnavramesh9305]

What if the file is fed into a pipeline and split into chunks in the first step and so parallely processed?

[@sina7601]

@@vyshnavramesh9305 You can utilize multiple threads and splitting them into chunks, however putting video files into a messaging queue is still a bad practice and using a technology for wrong reasons where it doesn't fit. Message brokers are not there to transfer BLOBs.

[@VY-zt3ph]

Hello Neet. I have a question. in the object oriented design interview course of yours do you have any roadmap planned like what future case studies you will be covering??

[@TheDipCompany]

At split second I thought that mr Beast is actually a software engineer

[@redrickTMN]

Great video, thank you!

[@shanecoleman2937]

For things like the user profile photo, I wouldn't think you'd store that in each video document. The document should contain a reference to the profile photo. Each user can only have one profile photo, so this is fine.

[@saralightbourne]

hasn't the author said that? profile picture is stored in S3, the DB just references it🤔

[@harryc5097]

Great video. I like the high level stuff.

[@antonchov736]

Thx bro, have only few hours before SD interview, vary usefull)

[@brianwang8546]

Love it! Design Leetcode next please :P

[@saranyavivekanandan9044]

Hi, thank you so much for sharing amazing contents!

[@saurabhverma1381]

The only video on youtube which actually assumes that "Design KZbin" is a fresh question that was ever thrown at you in an interview and you followed a top-down approach to present a possible design.

[@mahendranath2504]

nice thank you so much, thanks for sharing the knowledge

[@user-xl7wb9pr1g]

@NeetCode Thank you so much for such great contents! Could you please create a system design video on how to design a feature flag/toggle system similar to CloudBees or GitLab ones.

[@pierinazaramella967]

Hello! Thanks for the videos, they are very helpful and very good explanations and questions. 11:04, you talk about a load balancer, this was confusing for me, because the diagram is more for architectural design than application design (microservices, bff, kubernetes), is the LB mentioned as a BFF that manages authentication and requests, or is Kubernetes LB more part of microservices? Thank you!

[@primetimetran]

Hi Neetcode. Thank you for your work. Would you mind sharing/telling us what whiteboarding app you use please? Thank you!

[@nazramirez]

i was abt to ask same thing! Great vid!

[@CookieCurls]

There’s a lot of issues in this video. Durability instead of reliability, no explanation for how videos get from the object store to the CDN, NoSQL not being the best option for metadata, caching videos at the app server being a bad idea in general when you have a CDN, apparently saving profile picture URLs in the video metadata instead of just making a separate request for user data, etc.

[@69k_gold]

Although normalisation isn't expected in NoSQL, it's expected for you to normalize at a collection level in document databases like Mongo. It just makes it easy to deal with points you mentioned, like profile pictures and what not

[@tomyao7884]

Very informative!

[@NullNull-qn2ts]

you can scale num of encoding workers based on queue backlog length

[@mark601]

For the NoSQL user profile picture topic. Instead updating the picture in every document could it not make sense to store profile picture as a userid.jpg and overwriting this userid.jpg with changes hence the reference always pointing to the same destination and thus never requiring updating of references in multiple documents in case of picture updating?

[@S4ND1X]

Hey there! Thanks for the awesome video, just one question, wouldn't be better to send to the queue, encode and then store, so we only store the encoded video instead of the raw footage? At least in this case where we do not need the raw video anytime after

[@zachyang1041]

this is awesome!! btw, what app/website are you using to write this notes?

[@suman_achary_vlogs]

First comment, lots of love and respect. I solved lots of questions just from your explanations.

[@LeetDroid1]

Hi NC, please what tools do you use for making your videos? Hardware and Software tools

[@yasin_bhojani]

Great Video!

[@amirsasson10]

hi, Thanks, amazing video. seems like the they writing into the DB should be done only (or also) after the video is ready for watch - that would be after the encoder finished. no?

[@juliancodes]

Awesome content ❤

[@johnyeukhonwong830]

Even in sport live streaming, to the end users, they are mostly over TCP. TCP gets your congestion control, error code checking, and guarantee ordering delivery. With live stream, sure it sucks when you are 10 seconds behind or when you see the "10x speed up" or "fuzzy screen" when there is a slow network, but ultimately as a viewer we still want to see most of the live video to be played. The client side can throw error when there is a consistent connection error, or skip ahead when it finds itself way behind. With UDP you really don't get those benefits. TCP reusing connection also helps. In practice, almost no one will use udp. There are other proprietary protocols out there, but most of them depend on tcp.

[@SamyarBorder]

Great video 👌

[@moestaxx286]

your very knowledgeable man.. you just inspire me to keep studying tbh . gonna buy the course soon thanks for the video bro 👌🏽

[@grcdeepak1]

What tool/software do you use for writing/drawing in your videos?

[@kirillzlobin7135]

Very helpful. Interesting details about KZbin using MySQL with Vitess

[@praezy200]

Once a video is encoded and stored in the object storage (encoded), does the original video need to remain on the raw object storage? If it can be removed, then what is the use case of the raw object storage? Can raw object storage be removed from the design such that uploaded videos go directly to the message queue? Also, I thought sharding can only be implemented for NoSQL databases. If SQL databases can implement sharding, then they can also horizontally scale and I don't see a downside to using them over NoSQL databses?

[@danielgordos3803]

Message queues aren't generally optimised for holding large blobs of binary data like that - a raw video could easily be 1GB+. Also it would probably have to be placed in the queue in a single operation, while object stores usually offer multipart upload. The queue would probably just hold metadata like videoId, timestamp, videoType etc.

[@cpthermes3703]

Very nice!

[@awesomefj]

Why dont we skip the first object storage? Like the app server directly adds the video to the queue?

[@lucky-bj9de]

More sys design! Thanks!

[@alextech4881]

ce faci frt

[@lucky-bj9de]

Stau tu

[@SnehaSingh-ts7oi]

Since we have to favor availability over consistency, should we use some other database like Cassandra or DynamoDB than MongoDB?

[@sgwcollinss]

Should you in a design interview cover how many app servers you might need based on the DAU

[@iqbalibrahim4713]

so queue is a place for raw videos waiting their turn to be encoded by the encoding service?

[@Hjksivgx]

Can we ask for more of a system design stuff from you? Please!

[@tannerlin6849]

Finally an english tutorial

[@Sebastian-zs8cp]

what books do I need to read for this any suggestions? what exactly is the name of the topic i need to research?

[@dab]

The one thing that stumps me that gets asked all the time are things about updating the table of videos needed to be processed without losing the queue of videos that are to be processed. Any resources about improving the reliability of the encoding processor pipeline?

[@pramalvi]

You can use distributed message queue with persistence feature like Apache Kafka.

[@Meghss]

KZbin just suggested me this video!

[@sarvajitsankar3276]

@neetcode, any plans of launching pricing based on PPP so that it becomes consecutively affordable across all countries as well apart from US? Tried to send mail as well for the same, got no response

[@NeetCode]

Sorry, I might've missed your email - feel free to send it again.

[@sarathchandrareddy3033]

Yes. We really like your videos and content. But purchasing power difference is only making me to stop buying the pro. As a just grad student it's Ton of money other countries.

[@NeetCode]

@@sarathchandrareddy3033 Feel free to email me!

[@droidfever7267]

@@NeetCode Here in India, its a significant amount of money when converted into Indian Rupees. I really like your videos.

[@pinecedar180]

System design interviews seem to be totally phony. Looks like they might have originated at Google or something and then all other companies started copying this (humans arent smart so they just go with the flow; thats how human culture works). No one ever does system design in their software engineering jobs. There's always teams of people doing this for months. Whatever we can come up in 30 minutes is absolutely useless (was KZbin really designed in 30 minutes by a single person? no), so whats the point of testing something that is useless and something that we will never use in our real jobs? It makes ZERO sense. If something requires some kind of GRE/SAT preparation for something is not practical knowledge and people are trying to "train" for something that is just not used in their real jobs, thats another red flag. If they're trying to test intelligence or problem solving, they should stick to those kinds of questions. This system design stuff is just total BS.

[@intelliot]

It's a fun conversation starter though

[@69k_gold]

Considering they get most of the information from previous assessments, problem solving and technical rounds, this is the place where one can have a true SWE to SWE conversation. What you mentioned about how products are being ideated over weeks or months is true, and that's the point of system design interviews. Your idea will never become an actual product, but rather, this is a good way to test the real-time problem solving ability of a candidate. In theory, there are no wrong answers in system design interviews. Your solution might be flawless or it could have different flaws, but just the fact that you're able to adjust to changing micro-requirements and continue building is a good sign of a promising candidate.

[@TheMainManNorthOfEquator]

You cant design anything in 30 mins it is way to check how do people think and how much they can provide context and how they deal with trade off

[@CWaldman]

It's not about coming up with the ideal design. It's more about seeing if the interviewer understands various concepts and technologies + can communicate them well.

[@vhchoang]

It's definitely a skill you must acquire outside of real life software engineering. Almost like studying for an exam

[@krateskim4169]

Awesome

[@alloc7260]

I first thought Mr.Beast going to teach us youtube system But then I read the channel name 😅

[@muzamilahmed3127]

10:57 what if we store the only video metadata in the nosql while other data in sql?

[@xingyanglan6836]

Why no arrows from metadata to encoded object stores? How metadata go to CDN?

[@flowersideOrfa]

very good video

[@LogicTherapy]

Why would you use in-memory over distributed caching for your LRO? I can't wrap my head around that choice.

[@AnupamHaldkar]

Mr. Beast in the thumbnail OP!

[@SolarFlare090]

Ok, Hi, now I dont know if you may or may not reply to this, but I finally finished watching this video and would like to appreciate how informative this whole thing is. Now I do have a some questions if thats ok (I deeply apologize). So for starters, 50m uploads a day, is that true? Also, why divide by 100,000? You're also saying that within a min, that the encoder may need time to "encode" one batch of vids, but more may keep coming and coming and our need for more encoders increases? Does it keep growing or does it stop at some point. Now, you're also saying that the actual video is stored within a object store while its meta-data (like title, description, user) is within a database (could it be either sql or nosql)? Are the database/object store within their own server? also, could you define in layman's terms what is a app server, queue, cdn, and load balancer is and mention where they might be? Furthermore, why exactly do we need cache here? Lastly, you said that when we skip ahead within a YT vid, we're making http requests to load more chunks, right? If thats the case, does the same thing occur when we just let the vid play like normal? As I mentioned, I hope you do reply and I have to say how very insightful this whole thing is. Many Thanks.

[@danish6192]

so Mr. Beast was a software engineer so amazed by youtube architecture that he started youtubing

[@JShaker]

The justification for using NoSQL was kind of lacking.Relational database do just fine with high read throughput. I don't think it's necessarily a bad choice, I'd say either SQL or NoSQL would be fine.

[@kirillzlobin7135]

10:34 Why do we need to update all video documents if we just updated user image. User id will still stay the same

[@dibll]

Could you pls design a notification system and ad click event generation system.

[@pavelastraukh9905]

Is it really that better to use NoSQL and have a need to update potentially unlimitted number of records instead of having one user record and update it? I don't see how it is improvement over using relational DB. Yeah, it avoids join but I don't think that it is a big performance hit.

[@CookieCurls]

Yes NoSQL is a bad idea here. The joins are better to get all user data when needed, rewrites are actually somewhat frequent if you actually see how users make immediate changes multiple times on upload, the metadata content fields are going to constantly change and NoSQL could be a very expensive mass update vs SQL, storing profile pic urls in the video metadata is strange when a join could very easily solve this issue, it’s also potentially a many to many relationship… SQL just better here.

[@bhagyashri7729]

what kind of Q can handle data in GBs?

[@haidernaqvi87]

11:00 if a user updates his profile picture you don’t need to update it anywhere is the video metadata iff you have chose to name the profile picture as [uid].jpg in this manner you will always have the current profile picture

[@wesselbindt7589]

If a video takes 1 minute to encode, a worker can encode about 60*24 = 1500 videos a day. 50 million divided by 1500 is about 35000. You're making the math much more complicated than it needs to be in the section on the encoding workers.

[@thndesmondsaid]

Interesting, why does youtube send chunks of data via HTTP requests instead of using streaming?

[@thndesmondsaid]

oh wait just got a bit farther, and you're saying sending the chunks via HTTP is streaming. I see. I thought streaming required using a different protocol.

[@lasitha6251]

Bro was using an ad blocker on KZbin. 😎

[@yang5843]

Appropriate thumbnail as Mr Beast dominates YT

[@calvin6507]

oh my god i thought it was another mr beast skit until i saw the channel name

[@szefip]

Why is mr. Beast on the thumbnail? 😂 great vid btw

[@ruksads]

I guess, video upload should be 5million instead of 50 million. Ratio is 1:100 for uploading and watching users are 5 billion

[@asher4312]

I have a question people.. Why do we have to update the videos collection when the user updates a profile picture? Since the videos are referenced with the user via the id..and if the user updates their profile picture, only that user document should be updated? And since the video collection has the user via the id..and the next time a request for a video goes in, it will fetch the user via id and the updated profile picture will already be there in the users document. Why have you updated all the videos documents of that user asynchronously.. isn't this a bad practice..can anyone explain to me..I think I am missing something here..

[@AllInOnce318]

Are you and eggwick (here on YT) the same person? You guys have the same voice :D

[@dibll]

Neetcode - why we decided to use NOSQL for storing metadata rather than a relational DB and also how can we handle the use case you briefly touched upon where we don't want to upload the data again in case of a network partition during file upload?

[@vyshnavramesh9305]

Nosql used since uploads are frequent mandating sharding. But I don't think Mongodb is suitable to KZbin since it isn't good for Availability. There is a downtime when the master is down to elect the next master from slaves. I would prefer Cassandra due to its high availability (leaderless). But Cassandra isn't good when there are read queries on multiple attributes. Hence I would couple Cassandra with Elasticsearch inturn bringing in Fuzzysearch. If it were Netflix, I would have sticked onto a simple Mysql since writes are not frequent and provides enough querying. If fuzzy search required, I would still couple it with ES. (Would never use an ES as a primary db since unreliable).

[@dronedistrict4349]

Can you do a video about twitch?

[@ranjithragul]

Mr.beast thumbnail is soo good

[@challengeyourmind3937]

Why would you use an LRU cache when the most viewed videos are the most recent? You'd want the opposite of that.

[@NeetCode]

LRU means the least-recently used will be the first to be evicted - which I think is what you're also saying, unless I'm misunderstanding?

[@tal32123]

Where is the API gateway in relation to all this... Specifically in relation to the CDN?

[@muhammadsarimmehdi]

Why are you using noSQL and not SQL?