How Disney Hotstar Captures One Billion Emojis!

Get a Free System Design PDF with 158 pages by subscribing to our weekly newsletter: bit.ly/bytebytegoytTopic
This video is based on engineering blog post: blog.hotstar.com/capturing-a-...
Animation tools: Adobe Illustrator and After Effects.
Checkout our bestselling System Design Interview books:
Volume 1: amzn.to/3Ou7gkd
Volume 2: amzn.to/3HqGozy
The digital version of System Design Interview books: bit.ly/3mlDSk9
ABOUT US:
Covering topics and trends in large-scale system design, from the authors of the best-selling System Design Interview series.

[@mechwarrior4793]

1969: Put a man on the moon with the power of a calculator 2024: Process emojis with the power of multiple tech stacks, complex algorithms and the processing power of a multiple data centers

[@ReadThisOnly]

crazy

[@riyaddecoder]

Its all about the data size

[@pabss3193]

this

[@user-tj2we8ke8p]

This is my favourite video so far, please do more of these case studies!!

[@drhdev]

I just want to thank you guys for the content you create. Your newsletter has invaluable information that is hard to find elsewhere. Keep it up!

[@eduardoluissantos1359]

Love how you can share that much value in less than 5 mins!

[@redhood7105]

Consistently interesting content, consistently! Always learning tons from your content. Thank you!

[@pieter5466]

One of my favorite videos, particularly since I started a side project to learn Golang and Kafka!

[@juanitoMint]

What did you learn so far? Repo?

[@ToastRusk]

All these technologies serving emojis LMAO

[@LawZist]

They serve high load

[@tengahhidup]

It's not emojis, it's one Billion emojis

[@mm345-0]

The value clearly isn't in the emoji's, but the contextual data (not to mention the engagement). If advertisers can get real-time feedback on what happened and what the immediate sentiment is - it provides context that other channels (ie. purchases, forums/tweets/etc) can't. Individually one emoji's pretty worthless, but aggregated it seems like it would be pretty invaluable. I'd imagine the feedback loop for users would be valuable.

[@b-u-n]

This seems like a really expensive problem that sampling could've made cheaper

[@b-u-n]

you can't display every emoji so there is a % chance that someone will not see their emoji displayed. this means you can move that % to the frontend, reject similar submissions, and save 90% on your processing and 70% on your development costs.

[@shanehanna]

Cool and all but you could just sample a small percentage of the input to produce the output in a single service and nobody watching the emoji stream would be able to tell the difference. The vast majority of the reaction clicks could be dropped on the floor client side and the user experience will feel the same. Unless the gathered reaction data is the product you are selling to advertisers or something but even then just state it's sampled.

[@ricardoamendoeira3800]

This comment is the difference between intelligence and wisdom. Your solution would be much cheaper and simpler to run, maybe with a dynamic toggle to stop sampling if the number of events was low enough, such as a match with low viewership.

[@biaozhang1643]

Hotstar needs to calculate how "hot" the game is. Sampling wouldn't work.

[@ismbks]

why not? @@biaozhang1643

[@borstenpinsel]

Sampling works for TV. They say "x million watched the match last night" but they have no way to tell. At least that's how it works here on Germany. Selected homes have a box that recognises the channel and sends it over radio signal. You get paid a few bucks for your voluntary participation "​@@biaozhang1643

[@seannewell397]

I have a feeling they would've done that if it were clear and easy. There's something to spark being able to do those time slices for them I think. That being said, ofc in general build a simple one service thing and make it do the job, then see what breaks where cause it's probably the data model or data handling and if you can tune the data structure and/or algo you can run very very far with just one go process or a cluster of go node for example (still all one logical web server). idk wasn't on the team tho.

[@abhilashkr1175]

Spark has a lot of operational overhead

[@SreenathV]

In adddition to Spark's over head, we could use Kafka streams to calculate the emoji count (simple use case) and there by one less service in the architecture. For comlex computation, Apache Spark stands better as it is highly performant, clustered, statefull stats, windowing features etc.

[@seannewell397]

But the time slice is VERY good for this, basically aggregates for you over a sliding time window to do sentiment analysis.

[@faizandhami]

Great understanding

[@MilanVVVVV]

Honestly a weird stack with a lot of unnecessary overhead/complexity on multiple places.. Also don't know why they opted for such different technologies across the stack - i.e. literally a single Elixir/Erlang cluster w/ builtin batch processing (if necessary) could do the exact same job, only simpler and more resiliently.

[@dietsodalite3716]

I kinda feel the same way

[@ketaminefairy]

IMO it's not even the fact that it's so complex that makes it bad, it's the fact that they put so many engineers hours into building such a complex solution for a feature that at the end of the day pretty much doesn't make a difference to your users

[@dietsodalite3716]

@@ketaminefairy like the big value add seems to be aws emr processing

[@dputra]

​@@ketaminefairy not a big value for users, but it provides invaluable data for companies about users sentiments. We are talking about 1 billion emojis here 😊

[@ketaminefairy]

@@dputra You are right about that, I personally wouldn't think much of the data but I can see how they would love to gather it. That being said, their infra was built to feed this data back in real time to end users. If they only wanted the data for analysis, the go api + kafka and some processor like flink would have been more than enough IMO.

[@paulotcj]

This seems to be overcomplicated for its purpose intention. Additionally, while taking all data available would be highly desirable, missing reactions/emojis due to server bottlenecks would be barely noticeable.

[@juanitoMint]

Now that I've seen the vid I would implement it using a leaky bucket approach WDYT?

[@VaibhavShewale]

that was really awesome!

[@grkuntzmd]

Go Go! I learned Go in early 2016 and instantly fell in love with it. One of the languages that I learned very early in in my software engineering career was C and Go felt like C with many modern additions.

[@Kats0unam1]

Go is the way indeed.

[@markxavior]

​@@Kats0unam1rust and zig is coming

[@Kats0unam1]

@@markxavior I like zig too.

[@artursvancans9702]

Any reasons why HTTP requests and not websockets? Also, why python consumer and not golang if you already have a golang ingress service in the front?

[@truthprevails899]

@ByteByeGo 1:49 if the emojis are written in batches then while showing the order will be missed, right ? i.e. emojis won't be displayed in the same order it was received. right ?

[@akshaykumar-yk2ri]

This video was asked as a HLD interview question to me

[@sampath2054]

what is the use of Amazon s3 here ? do we storing the emojis ?

[@Su_Has]

Why can't we just use Kafka itself in the last pub/sub part? How is MQTT different?

[@soumalya]

I want a video on how hotstar handles massive live viewership in their platform

[@girishhalemani5947]

Does this added in bytebytego weekly newsletter?

[@pavelpikat8950]

Why run different vendors for messaging? If you already use and operate Kafka, why introduce MQTT as a separate pub/sub?

[@JvK20008]

Probably because it's designed for low bandwidth, low processing devices. It's fine to have a go service pick up a http post, because that's on their infrastructure that they control, but sending sentiments back to unknown devices, it's probably a little more reliable

[@HarshalRinge]

MQTT is used here as a replacement of websocket

[@AvinashRaj]

How will you create the topics in Kafka?

[@chigozie123]

A messaging queue would make more sense than Kafka. I would also get rid of Spark and the last Kafka.

[@shadabbahadara]

Which software can we use to make such presentations?

[@HarishIndia123]

Awesome

[@LawZist]

When vol3 is coming out?

[@_man.on.strings__]

create a video on how hotstart workks handles loads of request and viewoers with no lag

[@pennyether8433]

Assuming there are 1m emojis submitted per second, is Spark getting sent this many updates? It seems like a rather simple problem, given that the number of emojis is extremely limited. The main issue is handling the concurrency of millions of users, but beyond that aggregation is simple. I don't see why there is so much plumbing just to do a simple aggregation, especially one that is restricted to a window of time. You have to load balance anyway, so why not have each load balancing node do a simple aggregation across 100ms windows, then send their result (eg: a simple JSON of {timestart: , timeend: , smile: 1500, anger: 1,000, sad: 500, ... } to a single main aggregator capable of handling N requests per 100ms (where N is number of load balancing nodes)? From there it can send results to consumers, and also write to a persistent database.... any simple db could handle 10 (small) writes per second. I don't understand why it has to be as complicated as this is.

[@jyotiprakash6932]

you might have to replicate batch processing layer in every node(application layer under LB) instead why not do it centrally? there might be more metadata associated with emoji which you can aggregate centrally in a central layer?

[@atharvbhadange6871]

Well, in my opinion giving LB additional tasks apart from routing requests can hamper it's throughput.

[@pennyether8433]

@@atharvbhadange6871Disagree. The additional overhead is negligible: Incrementing a count in a hashmap (of size 20 or so), and shooting out a single message every 100ms. Even if you have to add another 10% more LB nodes, which would be among the absolute worse case here, it still saves you from the rest of the inflated architecture that was described.

[@DavidHarned1]

Why wouldn't they just semi-fake it? Its not like they're going to render millions of emojis client-side per second. Its not mission critical to make sure no emojis are missed server side. Just make it so that client side it always animates the emoji you've sent so that it *feels* like it actually sent. Then limit the backend to only accept a maximum amount of requests per second.

[@pennyether8433]

@@DavidHarned1 Excellent point! A relatively small random sampling would be sufficient.

[@cakitomakito3979]

how they can reach that performance by using python not other language can someone explain please? is there any specific reason to choose python

[@chansamonejk5379]

I like this presentation of you. I want a slide presentation of you. please

[@bforbiggy]

"System outages are a thing of the past" I like the optimism but... yeah we both know that's not true

[@catcoder12]

Things are a lot better now although. I don't remember the last time KZbin was ever down even after massively popular live streams or video releases

[@venkatamunnangi1287]

Why use spark over flink?

[@varshard0]

I think it's because Spark is built for batch processing while Flink is for stream processing. No need to process individual event as a stream. Imo, they are interchangably in this case as micro batches with Spark.

[@QckSGaming]

@@varshard0 Also it's highly likely they had everything listed here already running so they retrofitted existing solutions.

[@thecastiel69]

What is the usecase of S3 here? 4:09

[@steephengeorge]

Data store for sentiment analysis

[@zoravursingh5617]

they could just capture 1/10th of the emojis, generate 10x similar traffic with some noise added. ni one woukd be able to tell

[@Dr_Larken]

Free cookies! And you don’t need cache!

[@AshuSinghIN]

you had me at kafka, lost me at spark

[@mrluthfianto]

why not just use simple queues and tail the latest 100 emojis to the client, while aggregating the rest in the background?

[@airliners321]

Editing problem at 0:38?

[@MAtukulis]

same on 0:00 it says "day" instead of "today"

[@saffanalvy]

Dude! Why are you showing the picture of my Bangladesh's cricket captain getting stamped by Sri Lankan wicket keeper? Lol. This is awkward. Nice video btw. A question though. What will happen if the golang api service is replaced by a rust api service? I know rust is fast but how will it effect the entire pipeline based on performance? I'm assuming it'll increase. What's your opinion on this?

[@npcs]

I don't understand why they're using both kafka and pub/sub here

[@buzzprime93]

make one on how Instagram handles uploading millions of photos

[@sharinganuser1539]

I felt a lot of over engineering aspects ,but on low level there can be nuances and so the arch...but then we are just talking about high volume of emojis so idk.

[@mohali4338]

Seems the most inefficient design I have seen recently. Just put a redis and pub sub iot core or similar. Kafka, spark, EMR are good for analytics only

[@ankeshkapil3129]

I don't think this is very difficult to do as don't need to care about data consistency and nobody cares even if you loss track of few emoji's

[@morthim]

ive never heard of hotstar before. oh god oh dear god why?

[@_soundwave_]

Is kafka free?

[@aakarshan4644]

It's open source under Apache

[@user-gz6ml8ts3f]

free as in? you would obviously need the infra to host it. If you want to self host it, you should have a team with the expertise to ensure the availability of the cluster. There are cloud hosted services such as MSK on aws and confluent provides its own set of clusters and solutions, but its way too costly lol

[@dtvind]

Yes for local dev, not as production or cloud

[@user-gz6ml8ts3f]

@@dtvind You can run a kafka cluster which can be self hosted on any production environment. It's open source. LoL

[@vikingthedude]

Free as in freedom

[@jww0007]

guys look you haven't sat down with the problem to think & not even discussed with a team & you dont know their constraints (time, money, devs) & even if it's over engineered you don't know the org structures or incentives that lead to that, maybe it's a promotion thing youre missing too much context, dont act like you can watch a 5min video and critique it no matter how intuitive your suggestion is

[@user-kw9cu]

Golang 💪

[@UmeshKumar-iw7bx]

How reddit works????

[@andru5054]

they should use lmao-lang

[@jazeem10]

I think, its a perfect example of how to over engineer

[@skyhappy]

Why did they use Python instead of go? could use one language for everything

[@ricardoamendoeira3800]

Conway's law is probably at play here.

[@LSC132]

​@@ricardoamendoeira3800could you explain a bit more? How that applies here?

[@OmanDev]

One simple word *libraries*

[@skyhappy]

@@OmanDev you're making baseless assumptions

[@margs_aviation2153]

Probably it’s easier to connect the pub sub using python

[@MahmoudSeoudiCEO]

🥲

[@window.location]

Is this a joke ?

[@dietsodalite3716]

Why design a python consumer that will publish to pubsub?

[@Daniel-xi2eb]

I have the same problem

[@igboman2860]

So why not use go all through the architecture? Why did they choose python to read the aggregation topics?