Master Youtube System Design

Whether you're building a video sharing site from scratch or want to optimize an existing platform, this video has everything you need to know to create a highly scalable and available video platform like KZbin. Don't miss out on this opportunity to level up your development skills and build better, more resilient systems.
In this video, we'll walk you through the process of designing a highly scalable and available video platform like KZbin. We'll cover all the essential pieces of technology you'll need to consider, including load balancers, caching systems, databases, and more. Whether you're a seasoned developer or just getting started, you'll learn practical tips and best practices to help you build a robust video platform that can handle millions of users and streams simultaneously.
System Design Interview Basics Playlist:
► • System Design Intervie...
AWS Certification:
►AWS Certified Cloud Practioner: • How to Pass AWS Certif...
►AWS Certified Solution Architect Associate: • How to Pass AWS Certif...
►AWS Certified Solution Architect Professional: • How to Pass AWS Certif...

[@Badral-Din_S]

Thank you

[@NaachWatch]

Complex design broken down just to the extent it should be. Very nicely put together and explained.

[@SunilKumar-jl6dl]

Assuming that we aren't allowed to use S3 for pre-signed URL for upload/streaming, the question arises how can the system be designed without these AWS services? How to use a distributed object storage? Can you also mention how to achieve service availability & reliability? Also the transcoding part is mentioned at small length here. Considering the number of uploads (in the order of millions), that would be a great piece of discussion on scalability. Using Message Queues, Async workers to run transcoding at scale, the number of workers required all that should be accounted in the design. Streaming using CDN also should be elaborated, cause that is another major requirement in the system design for scalability.

[@marcogbarcellos]

Thanks for the great work, helped a lot!

[@dheebanm3207]

Awesome explanation sir

[@rushikeswararaopolaki3602]

Great work

[@siyaram2855]

Nailed it 🎉

[@techthief3278]

WoW just awesome bro!!

[@chits006]

Nice Video, one clarification, why storage is needed for video watched ? Also CDN, I think for streaming of video, it should goto CDN ?

[@1985poorna]

Great video few questions for clarification: 1. since we have high through put why we didnt use something like Kakfa / Flink instances? It would have made the design more fault tolerant and efficient, right? 2. Instead of using a API for orchesting all Microservices, cant we just use a API gateway?

[@thakurakash321]

Nicely explained! Have few doubts though: 1. What is the difference between metadata service and video catalog? As you mentioned both are used to store and retrieve metadata about the video 2. At timestamp 12:43, you mentioned about pre-signed url generation and then that gets uploaded from ui to s3. Wanted to know why can't we directly upload this pre- signed url in s3 from video uploader service, that might save us some bandwidth and trips over server. Am I missing something here?

[@ByteMonk]

Thank you for the question. 1. The Video Catalog database is used to store the information about the available videos in the system, such as their title, description, and the user who uploaded them. This database also keeps track of the number of views, likes, and dislikes for each video. The purpose of the Video Catalog database is to provide a quick and easy way to access basic information about the videos without having to query the more complex Video Metadata Database. On the other hand, the Video Metadata Database contains all the information related to the video content, such as the video file location, encoding parameters, resolution, bitrate, and other technical details. This database also keeps track of the various versions and renditions of the video that are available for streaming. The purpose of the Video Metadata Database is to provide a centralized repository of all the video-related information that can be accessed by different microservices in the system.

[@ByteMonk]

2. A presigned URL is a URL that allows access to an object or a file (in this case, youtube video) in a cloud storage service for a limited period of time without requiring authentication and It is generated by the cloud storage service provider. In future I might do a follow up video on Presigned URLs, but here is how it works at a high level, which should help you understand the flow better. Step 1: The User Requests a Presigned URL When a user wants to upload a video to KZbin, they first request a presigned URL from KZbin's server. The presigned URL contains a set of instructions that the user can use to upload the video file. The server generates the presigned URL with the HTTP method parameter set to PUT, which means that the user can upload the video file to the URL using the HTTP PUT method. Step 2: The Server Generates the Presigned URL KZbin's server generates the presigned URL and sends it back to the user. The presigned URL is a unique URL that is valid for a limited period of time, usually a few minutes. This means that the user must upload the video file to the URL within the specified time limit, otherwise the URL will expire and become invalid. Step 3: The User Uploads the Video File Once the user receives the presigned URL, they can use it to upload the video file to KZbin's server. The user can use any tool or application that supports HTTP PUT requests to upload the video file. The presigned URL contains a set of instructions that specify the location and format of the video file on KZbin's server, as well as any other parameters that may be required.

[@srawat1212]

@@ByteMonk - won't It cause challenges while editing a video's title ? Since both the tables are in different dbs are different how do you do it atomically? Doesn't It makes more sense to cache the details of VideoCatalog table and keep everything in Video Metadata ?

[@saranshchhibber3431]

Hi great video! Im assuming for search, we would be using something like logstash in the video catalog service, to send the data to Elastic search?

[@ByteMonk]

Yes, using Logstash in the Video Catalog service to send data to Elasticsearch for search functionality is a common and effective approach. Logstash can be utilized to collect, transform, and enrich data from various sources, including databases like MongoDB or Cassandra, and then send that data to Elasticsearch for indexing and fast search. By integrating Logstash into the Video Catalog service, you can ensure that the video metadata is efficiently indexed and made available for quick and accurate search queries in Elasticsearch. This approach enhances the search capabilities of the system and provides a seamless way to keep the search index up to date with changes in the video metadata.

[@srawat1212]

Or we can use a message queue so that even other systems can consume. Other consumers could be something like IllegalVideoDetection, Analyser-service, Datalake etc.

[@vova_dev]

Great, thanks! Don't understand total metadata storage required per day. 4:05 Why 1 mln videos power day * 1KB is not enough? You add 100 mln*1KB to it. Why?

[@patrickmotaung5866]

Does it make sense to store the video in the Object Store, retrieve it from the object store, encode/transcode it and send it back to the object store?? Why not encode it first before sending it to the object store??

[@jwbonnett]

Uploading to blob from the UI shouldn't be done, it means anyone can upload what they want to your blob store, not only that checks on the data can't be cheched as local UI can be overriden. Always have an API forward to the blob. if you are getting data, that should be done blob from UI.

[@ByteMonk]

Thank you for pointing that out! Yes, it is always advisable to have an API that forwards the data to the blob store, this way, one can enforce proper authentication, authorization, and validation checks on the server-side, ensuring that only authorized and validated data is uploaded to the blob store. It adds an extra layer of security and control over the data being uploaded. Uploading directly to a blob from the UI can pose security risks and lack proper validation checks.

[@DK-ox7ze]

If we upload to our server first and then to blob storage then it will take double the time to upload, and it can be quite significant for large files.

[@jwbonnett]

@@DK-ox7ze No it won't as you can just stream it stright through... Though it seems that you have missed the point!

[@davidmataviejo3313]

I stopped watching the video as soon as I saw he was recommending uploading the file from the UI

[@pmkarthikeyan2382]

Hi, can you add a video for food delivery system design like uberEats, DoorDash or Swiggy? @ByteMonk

[@ByteMonk]

sorry for the late reply, I will work on it. Thank you for the topic!

[@AbiolaOgunsakin1]

Great work here. Can you share your contact with me? Thank you.

[@ByteMonk]

Hello! Our website is going thru an upgrade. Please email here: [e-mail]