How NETFLIX onboards new content: Video Processing at scale 🎥

[@adityatripathi1904]

I believe there is a variation to Gaurav's understanding on open-connect caching mechanism. A new uploaded video DOES NOT goes directly to the caching server, instead they are resolved to a 'a small Netflix DB replica" in that region, which in-term populates the caching layer for new videos or most frequently accessed videos. The distinction is important because otherwise the design is assumed to directly populate OC CDNs, which is not the case. So for example, if a viewer in Australia wants to stream a non-cached US content: Instead of directing all the Netflix traffic through under-sea cables, Netflix copies the video file (transcoded) from US-based storage to a storage location in Australia once during off-peak hours. Once the video has reached the continent, it’s copied to all the Open Connect servers present in the ISP networks. This is not to point out a flaw in the video, just a clarification.

[@gkcs]

Thank you for this amazing clarification 😁

[@adityatripathi1904]

@@gkcs I am glad. Keep making fun content😄

[@Nikhilajayk]

Content apart , the new animations that you've added are really nice. Extra efforts are appreciated! Keep em coming !

[@ayasswain]

Another informative video from you. Nice explanation. Thanks. Just wanted to add something on the caching part. Many websites take help of CDN providers like Akamai to do the caching on behalf of them. The CDN providers have the required infrastructure across the globe wherein they have placed their caching servers in most of the countries.

[@gkcs]

That's interesting, thanks!

[@tejassardana6266]

Don't be surprised if you hit 500K subscribers by the end of 2019. Your content quality is skyrocketing.

[@gkcs]

Thank you!

[@DiaryOfMuhib]

@@talesara74 It's system design and it's not for everyone to understand, what did you expect? People mainly use this channel for technical system design interviews.

[@pratsway961]

You said that from your perspective. But the sad reality is People really don't like good contents😅😂. They can watch a 3hrs movie, but couldn't watch this type videos for 10mins😑

[@letsrock7354]

Gaurav I love your content and soon I guess will be joining your InterviewReady platform...Please offer me some special discount bhai 😍

[@vinayak186f3]

It's almost 2023 and he still has 488k 🥲

[@shubhankar915]

Showing Gupt in local popular movie would make Booby Deol really Proud. Savage AF Gaurav! :D

[@natashasen2575]

Wonderfully explained. Everything is clear and simple. Love your content.

[@gkcs]

Awesome, thank you!

[@nehaambasta1191]

Man, you know how to teach with fun! I must say presentation and video editing is very well done in your videos.

[@adityapatange8]

What a beast! Loved your reference to Amazon S3 out of nowhere! 😎✔️

[@anupammangal8990]

Hey Gaurav, I was just wondering how the Open Connect or KZbin Red boxes are any different from the long existing concept of CDN (Content Delivery Network). The only change is, being big companies they have the money and resources to create their own personal CDNs and brand them which is kind of building your own house instead of renting one. Is there something I am missing here ?

[@gkcs]

They are very similar, yes.

[@samiles171094]

I was having the same question as how these Oppen connect boxes are different from CDNs.

[@akashthoriya]

holy shit! Just fall for these videos; Thank you from bottom of my heart! Please make more video on this topic!

[@amarchhabra2175]

Bro - Thanks for the great video! Learned a lot. Just subscribed. Keep the great content coming!

[@Iamjohnsmithindia]

Keywords = CDN , predictive content delivery , content pre-fetch and so on...

[@vinayshrivastava266]

Yes Content Delivery Network plays a really important role if the geographic location of physical server is far and also I think what these OTTs would be doing is what we are getting in recommendation might be based on geo location rather than your watch history , this way if one guy from ur location has seen some movie it gets cached to a CDN and then when u request for the same content , since it comes in ur recommendation its extremely fast

[@charanteja4632]

Please do some lectures on docker internals like Cgroups, namespaces, union capable file systems

[@sushruttabakade6088]

You have the most unique content on KZbin, man. No bs, pure knowledge. Keep going.

[@gkcs]

Thanks!

[@himanikumar7979]

Blown by the simplicity you explained such a complex topic!

[@user-zu2uh8zg6p]

Thanks for your descriptions

[@gkcs]

Cheers!

[@algoseekee]

It's amazing to see how Guarav improves the quality of his content, and I can tell you, guys, as a newbie tech KZbinr, it's a big deal. Keep it up, bro! ;-)

[@gkcs]

Thanks Victor!

[@walaaelsayed8764]

Very helpful and simple for anyone who is preparing for software design interview. Your way of explaining is interesting and funny 😊

The material you have in this playlist is gold! I did not expect to learn so much from it. Thanks a lot!

[@gkcs]

You're welcome!

[@prnk139]

great efforts thanks a lot ! i remember my childhood days after seeing the GUPT poster! :-)

[@andriidanylov9453]

Thank You for sharing. Very nice details and very good description

[@sankalparora9374]

Short, simple and full of interesting concepts. Thanks!

[@gkcs]

Thank you!

[@catchroniclesbyanik]

Moral : Steal that Open Connect Box

[@shubhamtech89]

Champion presentation. Just wanted to see a graphical representation of the adaptive bit rate ladder through Netflix's eyes.

[@Rishabh199494]

I just started watching your videos. Your content is Amazing.

[@shivkojha]

Learning a lot . Thank You gaurav for creating these masterpieces.

[@manishramesh5446]

Wow, excellent video. Love your unique style of breaking down the topics in chunks and explaining them neatly. Keep them coming.

[@pvrcs]

Can you explain some of the advanced concepts of video streaming too? For eg., Bandersnatch? I wonder how Netflix is providing conditional streaming of video chunks based on user selection.

[@vk21563]

ISP cache is really interesting, wondering how they have handled security on top of it across multiple geography??

[@INCREDIBLEGUYOFINDIA]

Have been working in this arena for a while now, you got everything correct man, other stuff you post usually goes over my head because I haven't dabbled in a lot of those things but for once, felt nice to already know what you were gonna say. Haha.

[@gkcs]

Thanks! Your replies on the comments are excellent btw :D

[@nadeemahmednady]

Video -> Fragmentation -> ABR -> DASH + HLS -> Packaging -> Cache On CDN -> Play

[@ShivamRaj-si5wt]

1:23 AWESOME GAURAV, perfect example of what you are explaining, Huge respect bro

[@shraddhapandey5442]

That was a nice one ! Thanks for the video.

[@vishnuthamizharasan]

To be honest learned something useful today, and willing to learn from you more like this kinda of topic!

[@tirthdoshi7463]

Gaurav, I can't tell how interesting this video was ! Thanks a lot ! I totally enjoyed learning this ! Keep making more such videos ! :)

[@gkcs]

Thanks!

[@amitjain-eth]

Nice and depth knowledge bro!

[@mrskwrl]

Love the energy. Your videos are great.

[@krishsbhanushali]

Can you make a video over difference between gRPC and REST architectures?

[@gkcs]

I'll add this to my list, thanks!

[@arunachalaramana1512]

Krish Bhanushali Check this out sensedia.com/en/apis-en/apis-rest-graphql-or-grpc-who-wins-this-game

[@ANKUSH225]

please add graphQL also

[@aadeshsharma0001]

this isp cache thing is just simply amazing,

[@uneq9589]

As i have some experience in this area, i would like to say that you delivered crisp content on the same. Good job!

[@gkcs]

Thank you!

[@ankitbagga5713]

Thanks Gaurav. I am not an engineer but always wanted to get some resource to learn the concepts. I would love to see more of your videos and on advance concepts.

[@gkcs]

Thanks Ankit!

[@kennethpchang]

I'm very impressed! Great explanation! I just subscribed! Keep going!

[@neeshespieces]

You’re awesome. Thanks 😁😍 this was so helpful

[@saurabhprasad4499]

Gaurav- I think what you say as cache or open connect are CDN. which are an old networking concept nothing created by google or Netflix. I would love to hear your thoughts and I am not 100% sure

[@UECAshutoshKumar]

Thank you 😊

[@nicoqueijo]

The intro scared the heck outta me considering I'm watching this in a playlist and the volume levels aren't normalized.

[@gkcs]

I'm sorry 😅

[@pavel9652]

@@gkcs Yeah, same here, do some audio compression (compressor filter) to ensure volume is roughly the same across entire video and even playlist. Great content otherwise ;-)

[@sahyadrikida]

Awesome bro ..!! Simple and very informative.. loved it .. keep it up 👍

[@BhageshDotIn]

Hi Gaurav, thanks for the video. Impeccable. Keep doing the great work!!

[@kajalpareek8291]

Very nice explained and made it interesting. Will be waiting for more good system design question.

[@ayanhalder4514]

Hi Gaurav, You mentioned that instead of segmenting based on timestamp, Netflix segments based on scenes that are a collection of shots. Do you have any understanding of how they divide each scene? Taking your car scene as an example: how does Netflix know from which point to which point (or to be more abstract: how many shots need to be collated) to create that scene and to download it in a single go?

[@nemanja.tonic87]

It's called Shot Transition Detection. Here is a starting point for researching it: en.wikipedia.org/wiki/Shot_transition_detection

[@videoguy640]

That bit at the beginning was hilarious 😂

[@CodeMode9313]

Paaji ek dum kamaal ka video hai ...even understoodable for non cs

[@lzone24]

Gaurav it would b great if you can go deeper into Netflix design and technology choices done by Netflix team. Overall a nice video 👍

[@Suryayyaar]

Dont know if this has been asked earlier. 1. When Indian producers want to upload new content to Netflix, say a new Hindi movie, does it happen at the servers (outside India), get broken into chunks and get written to the Open Connect box(es)? 2. Is the content duplicated on the original servers and Open Connect boxes?

[@yog2915]

Bro u are great am a 14+ software engineer really appreciate ur content

[@KomalSingh-bh8zr]

Thanks for this wonderful insight on the engineering side of Netflix, looking forward to more system design videos.

[@gkcs]

Thanks!

[@kashyap263]

Netflix subscribers : We are the coolest people living on this planet ! We watch netflix and chill Netflix Engineers : Hold my(our) Beer !

[@vimalsheoran8040]

Netflix is definitely does one of " the illest and the chillest" engineering out there.

[@austinkim8213]

Captions (in English, not sure about other languages) become unsynchronized at 9:23. Not sure if there's anything you can do about that. Great video. Very informative.

[@gkcs]

Thanks for letting me know, I fixed it :D

[@SA-bp5kw]

Not a programmer or anything. But I love your videos. It's interesting.

[@dineshbs6635]

I never knew data handling is this much complicated until I saw this video

[@Theartsygalslays]

Very well explained, thank you, Gaurav!

[@gkcs]

Thanks Veronica 😁

[@Ramesh-he7fy]

Thank you for sharing amazing content. I just have few questions here. 1) I am assuming that all the videos will be copied to all CDN (content delivery network) locations across the globe and each of these servers will have cache on top of it. 2) How OpenConnect is related to CDN in this case (or) any system design. Can you please clarify.

[@RishavMishraa]

OpenConnect is just like CDN. They serve the same purpose. However, OpenConnect is developed by Netflix.

[@RishavMishraa]

CDN simply stands for Content Delivery Network which is used to deliver the content to any ISPs. CDN in generally committed to deliver any sort of content like pictures, videos and other. OpenConnect is in house CDN build by Netflix. Netflix have installed OpenConnect appliances at ISP premises with each having copies of netflix content. This helped Netflix content to by pass multiple steps to let the content reach from CDN to ISP and finally to end user. OCA are specially desinged to deliver streaming video content and not like general CDN which are designed not specifically for Videos. Therefore, OCA is also a sort of CDN but more specifically designed by netflix to deliver streaming video content. OCA cost in terms of setting up and maintences is taken care of netflix alone. Netflix gets the advantage of better user experience as they are at the ISP premise and hence gives extreme low letancy. On the other side, ISP gets benefit of saving some bandwidth and money to access the content over network.

[@akshaygagrani2927]

Loved this video please create one on Google sheets also.

[@syjaf6615]

Awesome man! Simple and to the point. The best kind of online content.

[@utkrishtpatesaria5079]

Great video gaurav. Keep up. I guess open connect is a CDN, RIGHT?

[@gkcs]

I've to read up on this. Will let you know 🙂

[@siteshp]

I too have the same question. @Gaurav

[@utkrishtpatesaria5079]

Just read from one of your provided links it's a CDN!

[@jaikishanchudasama8579]

Awsome Video Just a question. Is the chunking and encoding happing in real-time as the user calls for a movie or is the pre-processed and then sent to the user as they watch the content ?

[@gkcs]

It's preprocessed.

[@logicboard7746]

Thank you for your reading on this topic...this is enlightening (at least for me)

[@djlyons90]

This was a very informative video! Awesome job.

[@KejriwalBhakt]

How is open connect different from CDN? CDN also stores static data and helps in reducing the request load onto the mother server. Moreover, if GOT is on UK's "Open Connect" does the mother server routes our request to the UK one? Or do we bypass the mother server? Because we often tend to watch series that are not local.

[@yashgandhi9698]

I was the first one to watch this video after him! The content had no views but a like!!

[@Lokesh_Karki]

That's a improper system design for KZbin, probably slow.

[@toabhijeetsingh]

The content of this video was very dense. The breather at 5:29 was very much wanted :-). You have very good audience engaging skills. Good Luck !!

[@sumankumarsubudhi]

Whoa!!! Thank you so much for making such unique and great content videos ;)

[@PyjamaLama]

Wow this is sweet! Super smart solution to the issues of video streaming, will be interesting to see where it is improved upon in the future too!

[@itzikkasovitch2081]

Thanks a lot for this series. I find it very helpful. One thing that comes to my mind with these OpenConnect boxes (and CDN) is how they are handling Digital Rights Management for content that is not on their servers. Do you happen to know?

[@vyshnavramesh9305]

Open connect is built by Netflix, hence their server

[@amitupadhyay6511]

Not gonna lie, you got me in first second, sorting in O(n) lol

[@manjunath2728]

Hi Gaurav Thanks for awesome video. What are the technologies Netflix uses to display the videos?.

[@sudeepsabnavis]

I guess Netflix also use AWS Edge location as well to cache the S3 content and serve local content as well.

[@Divinity1993]

Yes you are correct .

[@mrinalinimusic]

Your videos are pure delight to watch !

[@gkcs]

Thank you!

[@ramsinghization]

Let me get this straight ,the chunk theory is only applicable when we play a video, right ? or does it happen while we are not playing as well. For example - if there's an old video of 6 GB and now processing the same video requires only 1 GB. So, netflix does that while we are actually playing that old video or it's processed in the night (supposedly) and permanently changes and put it in the Amazon S3 ?

[@gkcs]

The chunks are made offline. The cache population is done at low-traffic times. Watch the video again for more clarity.

[@heysupratim]

Great video. Something I would like to point out. AVI MP4 - these are video containers. H.264 is a codec. Not a big thing to worry about in the grand scheme of a system design overview video like this but just wanted to point out

[@gkcs]

Thanks! That's a good point 😁

[@mennagamea4634]

thanks for the video, amazin but I have question u mentioned that each chunk is having one format one resolution, I didn't get it so one movie will have multiple formats for each part? where is the consistency for the user here? :/

[@gkcs]

A user watches only one format in one device. Their player gets consistent format feed.

[@scenescentral3004]

Fantastic explanation

[@anjalibelani5101]

This is great work Gaurav. Thank you for this!

[@gkcs]

😁

[@vinitmundra1920]

Can u suggest the best book for understanding and also learning System Design questions, which might also help in interviews Pls am in a great need for it....pls

[@ganeshaditya3042]

Designing data intenstive applications

[@imshafay]

Openconnect ---> UPFRONT (AWS) so basically s3 is storing the smaller chunks and they are being consumed using AWS UPfront to get them without lag. I basically, developed this architecture for my video optimizaiton algorithm :)

[@pulkitpriyam8749]

Do these open connects can be considered as CDNs or they are different totally?

[@manirajsivasubbu4623]

Nice explaination, Thank you so much.

[@B-Billy]

Gourav, thank you. Can you please make a video swiggy (example) payments and it handles different response like success, failed, or pending.

[@swagatpatra2139]

If we break into so many slices, won't there be extra overhead to keep rack of all individual slices? And we are sending them to different processors for processing? What if chunk2 is processed faster than chunk1? And regarding cache, will the ISPs do the mapping for each individual cache? Won't that increase the look up table size exponentially?

[@gkcs]

1) Yes, there will be an overhead of keeping track. The gains of chunking outweigh the overhead though. 2) This is addressed in the video already. 3) Doesn't matter, since it's happening in the background, and the chunks require consolidation after processing anyway. 4 and 5) You question isn't clear. Please read the links in the description.

[@shivnandantiwari7489]

You're a gem you should have a teaching startup like byjus

[@marchanselthomas]

For some reason, the sparse movies that I've watched were also A-rated.

[@anirbanchatterjee7766]

Legends understood what you mean.

[@sabihaali7962]

Hi Gaurav, amazing video 👏🏽👏🏽 Wanted to know who does the caching the isp or cloud front for s3 Also these different formats and resolutions , are they already done and stored in s3 or are they transcoded using a function in the cache area. Pls clarify. I can’t begin tell you how amazing and impressive your videos are Would really appreciate some beginner level containers video

[@gkcs]

The ISP has a deal in which they take a box from Netflix which stores most of their 'current' movies. Updating the cache is done by Netflix. They are done and stored to avoid compute during run time. I have something on containers in the works :)

[@nosouponhead]

Upvote for that t-shirt.

[@kashyapsreview]

Thanks for the great info I wonder why I like Netflix more than any other OTT

[@tiwaripranav32]

There are lot many more things happen. There can be more than one type of Caching - Amazon call it Edge servers or Edge computing. And there is CDN - natively Amazon has it but there are Akamai and CloudFliair of the world who specialise in CDN. There is proximity calculation by means of DNS and weightage is given to connection coming from different places. And you seems to have taken too much of ISP - it is called DNS which keeps a hostname Vs IP mapping (this is also quite complex as the name resolution is carried out by DNS server once a request is issued from users' machine.

[@polavenki]

Netflix open connect caching technique is like their home ground CDN approach? Was wondering how was it different from traditional CDN solutions?

[@anjarhusain921]

I found this video very interesting; Netflix architecture fascinates me a lot. The detailed explanation is very well put, but there is sth about the video quality that you should have fixed properly. Take the blurr at 1:21 and some disorder at 6:55 - 7:10 for example.

[@gkcs]

Thanks for pointing this out. The 1:21 blur was on purpose to show what happens with reduced quality. But the 6:55 one has been introduced by KZbin. I'll file a report on this. Thanks again 😁

[@shoaibzafar5663]

Outstanding 😨😱

[@luis96xd]

Excellent video, well explained!

[@lallu12343]

Hello Gaurav, can you tell me how caching at the ISP level different from CDN / Amazon Cloudfront ?

[@shubham277]

I think a content delivery network like Amazon CloudFront is a similar kind of concept. It would be great to hear from Gaurav about how open connect is different from Amazon CloudFront.

[@perfectlyfantastic]

Arent they different , i mean in case of CDN like Amazon Cloudfront you need to go the aws or any server you are using , hence one need to hop to server and check the cache , in case of ISP its more direct need not go to the server at all , shifting the problem i see it like reflex action , nervous system(spinal cord) providing some response rather than request going to brain(in biology terms) in short its faster than CloudFront