Scaling Hotstar for 25 million concurrent viewers

Gaurav Kamboj -- Cloud Architect at Hotstar -- explains why traditional autoscaling doesn’t work for Hotstar (Disney’s OTT streaming service). Hotstar recently created a global record for live streaming to 25.3 million concurrent viewers.
Gaurav explains:
1. The challenges in scaling infrastructure for millions and how to overcome it.
2. How Hotstar runs gamedays before actual games.
Details on hasgeek.com/ro...

[@HasGeekTV]

What bothers you more - monthly real estate rental or cloud cost bills? Plan your cloud infrastructure and optimize costs. Learn from experiences of enterprises and startups, on 26 November 2021 at hasgeek.com/rootconf/optimizing-costs-of-cloud-infrastructure/

[@ScrolltheNature]

Great !! wish he has covered the database scaling as well.

[@Namesake..]

@5:05 The point about the homepage API's taking a hit as soon as Dhoni gets out is INSANE. I wouldn't have imagined that scenario. Excellent talk.

[@v1r47]

That point is absolutely on point. Never thought about it in this way. Loved the talk.

[@chaitanyaatre4489]

Exactly I never thought this point as developer 😳 awsome

[@nikilkumar1000]

All these are usually observations from previous occasions. All predictions are always based on previous results.

[@bhaveshssharma8826]

There will be cache for home page content, why will it cause a hit, anything I am missing?

[@rajshukla865]

@@bhaveshssharma8826 You cannot use cache , as you need to change data of recommendation , and last watched videos , advertisements and so on! Hence we cannot use cache. It'll be a bad user experience!

[@ajaykumar-xy6pw]

I couldn't make my younger brother understood how big an achievement this is, he's still doing his engineering. I hope one day he'll understand what level of engineering in India specifically hotstar is doing. Loved the talk

[@deep_z]

Hi ajay, can you explain a bit about what is high level about this? i am not an engineer, but would like to know. Also, how does it compare with the engineering of other global hotstar competitors?

[@ajaykumar-xy6pw]

@@deep_z concurrency is the big thing here, how to handle these many people accessing same set of services without any interruptions, while in other apps you'll see different set of data will be consumed by different set of people, thats not true here.

[@arjunjmenon]

@@ajaykumar-xy6pw youtube , irctc is doing it all the time . isnt it

[@ajaykumar-xy6pw]

@@arjunjmenon KZbin yes, irctc I don't think so, and overall KZbin is backed up by big players like Google, which is one of the best tech companies in the world.

[@Phoenix-qo8kq]

@@arjunjmenon irctc lol

[@adityasadaphule2187]

The project I work on has 2M in a month and these guys are handling 25.3M in a day hope to work on something so big like this.

[@shashankshekhar8970]

Not even 1 day, its 25.3 M concurrently. Over a day it will be multiple magnitudes higher!

[@karun4663]

the product I work on has at max 64 concurrent transactions

[@garud1411]

@@karun4663 haha

[@vinitkumar2644]

Just now of my application broke down on 100 concurrent users🤣🤣🤣🤣🤣

[@ytadi9229]

Ind Pak game, they reached 100M recently

[@nayanagarwal1482]

And KZbin just decides this one fine morning, to recommend this video to all engineers !!

[@Jeevan-ve5sc]

its recommending on 2024 also.

[@AlanWake-c7y]

Here after 5.4 million in IND v NZ Semi-Final of 2023, crazy to think that the new record is DOUBLE than this!

[@cu7695]

What's interesting is they're not using CPU or networking for autoscaling. They're using concurrency and request rate. I think they already have scaled infra before the big game already. Autoscaling only kicks in when that baseline is exceeded

[@prasadj8676]

Yes u need to scale on golden metrics ! CPU based autoscaling is not very useful ! It should always be concurrency or latency!

[@Shivamyadav850580]

Yep. Proactice scaling > reactive scaling

[@RonaldDas42]

TLDR; Perform Load testing to get instance limits (in terms of cpu/ram/network) and than prelaunch more instances that would serve your expected numbers of concurrent users

[@abhishekahirrao4271]

I think the essense was well summarised. But the actual value lied in the process of the talk.

[@xtinctspecies]

Excellent level of logging, monitoring and observability. Loved the talk even though I am not a devops/ops person

[@Budgetbiker001]

Man same.i just felt this was interesting

[@XoPlanetI]

This is not just for devops or SRE folks.This is what core engineering team does

[@legowtham]

I've seen the concurrent viewers hitting up to 45M in the last ipl match. Science and technology never cease to amaze me.

[@swamysriman7147]

That's lakhs bro not millions I too thought the same and got confused😅

[@XoPlanetI]

And the skill of Indian engineers

[@pakhilnair]

This is amazing! I did not move from my chair for this 46 mins! In those tense moments of IND vs NZ, I used to wonder how these guys are keeping up with 25m users! That question has been beautifully answered after half a year! Thankyou Gaurav! You and your team are the real heroes!

[@wolverine047]

Wonderful talk by Gaurav and it was such a pleasure to work with him.

[@parastiwari6664]

This is absolute pinnacle of software engineering. Kudos to the team. I am still a student learning cloud platform and all, the whole presentation kept me glued to it.

[@asrajan55]

Wait...maybe I am not getting this...but isn't this all just configuring AWS?

[@parastiwari6664]

@@asrajan55 hi Soundar, afai recon, the scaling of so many resources, along with the risk management and maintaining concurrency without breaking down the APIs AT THIS scale, is itself a challenge. To mention the fragile nature of the viewership graph and working on a breakdown time of 30-60 seconds, all together makes this task an absolute marvel.

[@kbriaz]

They came up with their own autoscalling app - thats gangsta

[@Gearsofwartai]

Unfortunately hotstar management's doesn't want to spend more and let football streams be at 240p when it says Full HD.

[@8_adityasinghrawat319]

Fyi speaker isnot from any IIT,NIT,IIIT or BITS.

[@murali1790able]

This basic question is unanswered. 25Million concurrent viewers == 25Milllion TCP/UDP connections. are you creating the ~25 or higher TCP connections while performing the game day/load test ?

[@SivaChaitanyaReddySanivarapu]

I imagined how KZbin works perfectly with millions of calls per minute everyday

[@adnanahmed316]

more like per second

[@tell5g]

KZbin is more distributed bcz of that u will most likely face buffering watching videos less popular youtuber outside India.

[@niteshkumarmodi5363]

Kudos to hotstar team, handling traffic at at this level. Superb presentation to illustrate all concepts.

[@sushantwats]

I have an on prem application which i build , which is based on Event driven architecture and we handle around 14M msgs/day , whereas the max spike rate we have seen is 5M/hr, but our throttling and fault tolerance isn't as effective as this , loved the details and especially the depth of the presentor.

[@aaronbcj]

Interesting review. About using 8 AWS regions for distributing the bandwidth for pre-match load testing, how did it play out on match day since almost 20M of the 25M hits would have come from within India and hitting only 10 edge locations (2-3 regions) if I'm not wrong. Is that load testing strategy a good match for actual production load pattern?

[@saswatsenapati7870]

have the exact same question ....

[@ShivamSingh-sm2oy]

if your all traffic comes from india doesn't necessarily mean your all traffic would be served from indian AZ servers only. there are always resource limitations which can loose up the performance comparatively when entire traffic is being served from indian servers vs scattered global servers. but you would not want to take the risk of serving everything from indian servers on the cost of risking the whole system as a whole. that's why a lil slower and higher network bandwidth is accepted than pushing all the traffic to nearest servers and taking the risk of system downtown. its a "Trade-off". people will be okay with a lil lower loading but not with viewership disruptions. a lot of things come into play when you are dealing with such scale. *Engineering is beautiful*

[@pythonop7303]

yeah here after ind vs pak

[@yashodeepdhas8408]

Same bruh

[@abhayvaishnav]

@Sir Bradulkar Where have you seen commerical content on 60 fps sir?

[@aksh102]

Came here after India vs Pakistan match when 2 crore people watched the match without buffering.

[@--Anurag--]

The bitrate is still pretty terrible. I mean you can straight up see pixels on screen. Of all the streaming services, hotstar is poorest. Until they improve the bitrate and the fps, I will not be impressed.

[@rishikeshagrawani7920]

Awesome explanation of how the Hotstar tackled the load problem to handle huge traffic. Liked it. Thanks for sharing such a good info.

[@pankajkadwadkar2252]

I think now there should be part 2. Scalling Hostart for 4.7 cr concurrent viewers.

[@omishahyt]

The way he explained 👌🔥

[@sourendhoommail]

Now I know why we don't get to watch the match at 4K 60fps at least. Hope we reach 4K 120fps soon with premium and 1080 60fps on regular plans.

[@himanshubhatt9912]

I want to know how they are managing the aws bills specially when he mentioned c5 9x large cpu's

[@saneeshkumarkuruvath]

All the cloud providers have corporate plans were they provide huge discounts to their potential customers.

[@satishy5132]

Love love loved it❤...AMAAAAZING session...Every sentence is so full of substance and thought provoking...Gaurav absolutely killed it.

[@DhananjayNaik]

Apart from code, its just configuring AWS, I wonder, how AWS boasts about their metrics.

[@kapilmonadi7005]

Wonderful presentation. Lots of insights into how you handle resiliency and load. Many thanks for uploading this.

[@AnemUday]

Woww, wonderful explanation with right techniques along with use cases. Really liked it. Thanks Gaurav.

[@praveenchukka8194]

Madness! The scale issues are insane! how did they even manage such spikes.

[@sonicjetson6253]

So jist of 45 min talk is to use custom autoscaling metric like concurrent request instead of default cpu/mem metric provides by aws. Why don't u guys open source the auto scaling tool that u developed?

[@thoughtmachine6176]

Us humans can seemlessly serve 25 million people over the internet but when will we start making reliable projector connections???

[@SaurabhKatti]

It was indeed a great video. Thanks Gaurav for that wonderful presentation. I have one more question -> As you said for example if Dhoni is the next batsman we all know that the users are going to increase. But lets say if a batsman who is not performing well throughout the series, suddenly starts performing extremely well and again there is a spike in traffic. What is the plan to deal with such situations ?

[@sakalleydhruv]

The irony when you are talking about handling load for a streaming service and the screen says no signal (from the projector),

[@SaurabhPandeySF]

Would love to hear something like this from Netflix which handled Money heist and all very smoothly. One key difference could be getting live data vs streaming from CDN. Nevertheless great job #hotstar team. Proud of you guys 🙏🙏

[@venkatapavankumarsannisett2259]

Read about netflix openconnect they pre calculate the demand and preload the content at ISPs

[@healthinsurance59]

22:00 is the actual explaination starts

[@rixonmathew]

Great talk Gaurav. Some especially useful and practical tips to scale on public cloud.

[@jasper5016]

This is so amazing. No claps from the audiance? What kind of audience was there?

[@RakhiDhavale]

Bro, I was desperately trying to upgrade to premium during India Vs Pakistan. That day there were more than 90 Lakh online viewers. I was unable to pay online , payment methods were not working due to high traffic.

[@satyamgupta756]

I've learnt soo much from this video. ❤️

[@vs4vijay]

This is insanely huge! How come this has not made news.

[@brunowario5169]

Because they are more concerned with Aryan khan's drug habits.

[@asitha106]

Really good production level insights, thanks for sharing...

[@SiddharthKulkarniN]

What was the AWS bill?

[@CloudPathLearning]

Whatever the bill was hotstar with this kind of performance could have already earned it back

[@fevicoI]

5-6 crore each day during peak work load

[@shashankshekhar8970]

probably a million on a day like this

[@azr_sd]

@@shashankshekhar8970 a million dollars?

[@filestreams]

$1 Million on an Average

[@funnyclipz520]

3:25 Scaling is not slow in nature since kubernets .. if u follow MS arch a pod can be up and running in about 5 seconds This problem was solved long ago....

[@jainparv]

Nicely explained by Gaurav, there is a lot to learn from his talk

[@ashishsingh4693]

He's Hero of Cloud 🫡 waiting for pak vs india match case study .

[@saireddybonthu985]

12:15 Elastic Load Balancer but not really Elastic in nature!

[@cu7695]

To be fair no LB really is. The scaling by margin of 800 servers with high capacity is harder for AWS. They don't have hardware optimization GCP has. Although I do think this is hard for GCP too from networking perspective.

[@salman0ansari]

this is amazing but i am thinking about their AWS bill 🤔

[@harshitbajpai4942]

Saving this for my system desing interview

[@abhayvaishnav]

Yes 😂

[@roboticsprocessautomation7738]

Really awesome engineering by Gaurav Kamboj. I think after seeing this video Amazon will make some changes on autoscaling. Its my please to hear you

[@bonjovi5530]

i genuinely appreciate the stability of the disney hotstar platform during intense workloads but when you stream at 720p @25fps for 20 million vs 2160p @50fps to 2 million in uk germany, it is not a big achievement

[@kapilbhardwaj4680]

It's 1080p

[@bonjovi5530]

@@kapilbhardwaj4680 did you watch it in 1080p on hotstar? because i remember that the best quality was llimited to 720p 25fps for ind vs nz game

[@of5n307]

@@kapilbhardwaj4680 if you are a premium user only you will get 1080p! If you take hotstar mobile plan then only 720p resolution!

[@sanjeetbisht]

On TV app it was 1080p 22-30fps

[@bonjovi5530]

@@sanjeetbisht since 2020 it is 1080p 25fps, i was specifically talking about the ind vs nz 2 day match, i am pretty sure that it was 720p 25 fps for premium users for live events

[@jdfutura]

My coding abilities are limited to printing "Hello World" on python yet this is at the top of my recommendations lmao

[@godsonjoseph]

Its best if you start watching stuff like this,.. Makes you think a lot before writing a single piece of code.

[@mentix002]

@@godsonjoseph strongly disagree. It'll only serve to demotivate the beginner who will find all these keywords and abbreviations to be extremely complicated. Maybe after programing for some time and reading some system design books one would be comfortable with this video.

[@raushankumargupta7018]

All we here , After india vs pak match to know its database .

[@ganeshdeshmukh8020]

what a great talk, with details. would like to see these more in future.

[@amans6504]

5:05, I'm a newbie dev but can't we just temporary cache the homepage once the user open the app, and then show the same cached files when the user will return and because most users won't be consistent with the homepage, some will leave, it won't have a huge load on the homepage.. and because users will open the app in different times, it's easy to cache the homepage at different times

[@onlyjustin4u]

You will still make API calls to return from Cache. Depending on what level you are caching, you can shave CPU cycles but not the requests.

[@namanmehta4108]

Every user has similar API calls like fetching the video list, categories data but there are many which are personalized and specific to a particular user such as user profile and recommended. This is not that easy to cache and needs to be sent differently. I guess this is where the backend system is most impacted. I'm not exactly sure as I also just started learning this...

[@AbhilashMarichi]

This talk is amazing! Loved it.

[@arnabmukherjee5840]

At 27:00 should not we save personal preference in client cache ? For better user performance.

[@ItachiXSanjii]

Mainly its about the configuration on AWS to scale up that they have acquired using load testing. AWS need to be boasting about their cloud platform and services.

[@Lokeshsanapalli1729]

Great talk… thinking in terms of RPS and concurrency to auto scale is out of the box… I am bit disappointed to know they need to keep watching the event to scale, hope they’ve automated this now

[@adi_sekar]

Good talk, but most of heavy lifting is done by AWS. Hotstar is just configuring AWS services

[@mihirsheth2689]

And 1.2 Crores yesterday 🙌🏼🙌🏼🙌🏼

[@Real_Intelligence]

I feel hotstar’s Home API calls have been comparatively slower than other platforms out there. Every time I go back to home it buffers for some time always

[@sitikantapanigrahi8621]

I think they have done it intentionally

[@hitmeup7003]

I'm a fresher just placed. I'm happy I atleast understand what he's talking. Ssly a lot of efforts go into this.

[@mpn_family]

25:36 Chaos Engineering topics are nice and fantastic.

[@Narenjhala]

Had to stop it at 11:52 due to neurons scaling error.

[@malaybiswal]

about 13,500 8 CPU core 16 GB RAM load generators were used for load testing it seems, probably spot instance would be better in this case.

[@AtulSharma-vc2tg]

This was 2 years ago! wow..

[@vajahath7]

Hotstar: "Scaling for 25 million concurrent viewers" KZbin:

[@amanbhargava3164]

There’s a difference between streaming live video and just doing playback of existing files

[@techmad8204]

@@amanbhargava3164 even KZbin has live videos

[@jainlokesh]

KZbin is owned by Google. Which has it's own GCP.

[@ruturajdharav4675]

@@techmad8204 but not with 25M

[@sadhlife]

@@ruturajdharav4675 it's biased though. the number of viewers hotstar gets or cpus they run at any moment is positively drawfed by youtube numbers. the fact that all of those cpus are crunching the same livestream means very little.

[@ruffdmx99]

This video needs CC, the accent is a bit heavy to fully comprehend.

[@williammickeys]

Where are you from?

[@techmad8204]

There are captions

[@Sharz71]

This is awesome! I would like to know the current infrastructure also, they must have come up with new strategy.

[@arunkmrgarg]

12:20 elb not really elastic

[@techentre]

Thanks youtube for recommendation.

[@ArabianKnight63]

This is the mafia level of infrastructure management!! I handled 100k users qnd felt like boss. Well now I know what boss level looks like xd

[@omprasadsurapu955]

how much is your infra costing? Is it hosted on EC2 with ASG managed by ECS?

[@RahulChauhanart]

Not so difficult when app stream at 144p

[@harshparikh9554]

7:14 10 Tera Bytes per second or Tera Bits per second?

[@syedkamranahmed4854]

I think it would be Terabits and not Terabytes.

[@kaushik.aryan04]

bits as small b is used

[@siddhantchavan1370]

TeraBits lol, terabyte is denoted by TB

[@pumpkin7331]

data transfer is always measured in bits..

[@bournvita382]

Was going to go for a sleep. But then this happened. Always wondered how they scale it.

[@rakeshsambangi3876]

Again hotstar created another record of 1.3 billion viewer's.... Ind vs pak T20match 2021..They handled it very well...

[@voicevy3210]

what next ofter solving these problems 🤔🤔

[@RudhinMenon]

So inspiring to see home made apps doing so great

[@Polaris071]

Simply superb ... Learnt a lot of things 🙏

[@shivamannan]

It was more than 70million during ipl

[@nitinissacjoy5270]

you use 36 core machines for generating load? you're testing for huge amount of RPS maybe, but that ain't real life. Real life is TCP Connections.

[@rajivjha123]

Tata sky and cable/satellite network providers provide contents to billion users since years concurrently without any latency and issues :) . 25M is too less for them .

[@shivasundar4029]

Noob

[@raghunandan8162]

They don't stream via HTTP API calls

[@rajivjha123]

@@raghunandan8162 - yes they use dbv protocols for streaming , they also use http btw these days . Thanks

[@smithyanthaya]

And yet hotstar content moved to Hulu, Disney, ESPN in US

[@JaaOffendJaa]

Disney have bought stakes in it ? do u know ?

[@Babbuchinna]

KZbin never fails to surprise me

[@learn8896]

He is not talking about t20 wc India match

[@ashokrajur09]

liked the insights of ondemand scaling very much.

[@tilak231]

3-4 hrs watching rain on a screen!! That’s dope! 😂✊

[@swagatpatra2139]

they run highlights during this time, not rain drops falling.

[@jagadeeshmadhura3223]

Amazing tech talk 🎉

[@chetandange9866]

Yesterday it wasn't working on 1 mn viewers

[@MaheshPagar]

Scaling was a great feat but even after all these years , the UI and UX of Hotstar is absolute garbage.

[@brutef04c38]

zee5 has worse

[@alokprakash2017]

I never encountered this issue.

[@AbhiRitvik]

yes

[@nagendrakumar-tj1fe]

Great.. Very informative.. now I'm just wondered how autoscaling works for normal telivisions in traditional way thru satellites

[@suyashmittal]

Autoscaling is not needed in em communication

[@alokprakash2017]

That is called broadcasting. Any listener can get the packet and process it. If there were no viewers, in that case also satellite is going to broadcast. But in case of internet, bandwidth is limited.

[@theneedytechie2468]

Why is the channel moderator deleting my comments when I mention about how bad the actual experience is with hotstar Live matches. You can censor all you want but the recent play store reviews will tell you how bad it actually is with streaming Live matches in high quality. If it was not for the exclusivity deal with indian matches, it would not have users

[@billyean]

How do you define current viewers? Peak requests is 1M per second, how can you have 25M concurrent viewers?, also 10Tpbs seems low to me, if your data is correct 10Tpbs / 25M = 400K Tbps = 50k TBps, we know for low level, 480p, we need 1.2Mbps network bandwidth, I don't know how many people will enjoy this game if watching it lower than 480P. I am also not convinced this is more complicated than Tiktok/KZbin, many people watch the same event at the same time definitely is something, but since it can leverage cache and CDN. So the complexity is much simpler than people randomly watch what they want to watch.

[@DhanushSBabu]

I don't understand why you think 25 Million concurrent viewers is not possible. India has a population of 1.3 billion with over 560 Million who have access to the internet. Mostly through smartphones and cheap mobile internet plans, a country crazy about cricket having 25 Million concurrent viewers is not that surprising to be honest.

[@billyean]

@@DhanushSBabu Please read my question carefully, I am not questioning the possibility, just the fact of the data. Tell me why he has 1M peak requests per second but have 25M concurrent viewer, how do you define current viewer, is it something people login who don't watch anything?

[@devarpan]

He’s talking about backend services requests which happens lower than video requests (which goes to cdn provider). So when 25 million concurrent viewers are watching a video, Very few times per a interval requests made to backend services to check some conditions or log user session. So 1 million requests for 25 million concurrent users is possible.

[@sachinsarmal9308]

if you had watched hotstar's live stream you wouldn't have asked this question they indeed show 480p videos only, which are upscaled to FHD for their premium customers while still keeping bitrate of 480p. Their bitrates for 1080p are so fucking low, anytime theres transition in video everything become pixelated. imagine providing pixelated live stream in 2021 having such pride.

[@puneethshivalingaiah]

Thanks, Learnt few things 👏🏻👏🏻👏🏻

[@indianmemeswaala7653]

Jio cinema handled 2.2 cr (22 million) concurrent traffic for a normal ipl 2023 league match of CSK vs RR Wonder what will be the live numbers in IPL 2023 final