Streaming a Million Likes/Second: Real-Time Interactions on Live Video

QCon London International Software Development Conference returns on April 8-10, 2024. Level-up on 15 major software and leadership topics including; The Tech of FinTech, What's Next in GenAI and Large Language Models (LLMs), Performance Engineering, Architecture for the Age of AI, Innovations in Data Engineering and more.
Learn the emerging trends. Explore the use cases. Implement the best practices.
Register now: bit.ly/3rRgsJV
----------------------------------------------------------------------------------------------------------------------------------
Video with transcript included: bit.ly/3nwZkBt
Akhilesh Gupta does a technical deep-dive into how LinkedIn uses the Play/Akka Framework and a scalable distributed system to enable live interactions like likes/comments at massive scale at extremely low costs across multiple data centers.
This presentation was recorded at QCon London 2020: bit.ly/2VfRldq
#LinkedIn #DistributedSystems #Scalability

[@vikram_saha7]

Perfect presentation !! The way in which he delivers the complex design choices in an easy and fun way is amazing!!

[@-indeed8285]

Enjoyed way he present.. (Proper slide, Great transitions, No breaks, engaged with audience, making jokes and all of that without losing the context.)

[@venkatapavankumarsannisett2259]

Brilliant articulation and to the point. Also for the resources in between. Thanks to infoq for such amazing real world scalable stories from the industry

[@Yunus0or1]

What is the raw computing power of one single Frontend machine? How many cores and ram?

[@visheshgupta4990]

Thank you so much I watched the complete video and it is very helpful 🙌

[@kanthipavuluri3874]

Totally loved this evolutionary styled presentation, brilliant story telling!

[@preetiirrothi744]

The best presentation i’ve seen!

[@natadilip]

Amazing Explanation. This is clearly one of the best videos on qcon.

[@RahulKumar-lp5jc]

One of the best talks I have seen. Hats off to you. I have a doubt here. In more than 10k connections case, why do we need an additional layer of frontend nodes? Instead, why can't we store the mapping of video and Real-Time Dispatcher node in the distributed key-value store. The Likes backend can query the key-value store to get the dispatcher nodes for sending the like object. Once it reaches to dispatcher node, the node has the in-memory subscription table to send to the corresponding end user.

[@amolnigade8830]

Insightful information on live video interactions

[@user-jt5nd3yq4u]

Amazing presentation, learned a lot from it!

[@naveenjohn3402]

Wonderful talk. Very engaging and beautifully presented

[@sk4152]

Great presentation. He missed answering one question about how likes at different times are stored and sent to the viewers. For real time watch, we don't need to store any time. But if we're supporting replaying the video and if we want to show the likes as it happened during live video broadcast, we'll have to store the time, say for example number of likes each second.

[@Secret4us]

Good video, thanks.

[@Rkrish2010]

Awesome many thanks 😍😍☺️👍😊🤗😱

[@codetolive27]

Amazing presentation. Loved the way he has presented.

[@YT-yt-yt-3]

Very engaging still calm and neat presentation. Insightful.

[@raymondsquared]

Really professional and easy to follow presentation. Loved it! Whats the benefit of using long polling vs websockets?

[@curiousspirit3947]

curious too

[@gxbambu]

You don't really need websocket here, cuz sending likes and comments don't need to be real time

[@rahulverma2523]

@@gxbambu Yea, but they are calling it the "realtime" platform. Also, the platform is used for multiple use cases and not just for sending likes.

[@ekamaloff]

@@gxbambu the whole point of this presentation was distributing likes and comments in real-time , which is required for live streaming

[@akashjain2990]

He kind off explained that in the first question from audience. Reason is : Long polling using server side events are basically plain HTTP requests; so support even very older devices; do not get blocked by firewalls etc. Both of these problems exists with WebSockets.

[@anandt8362]

Thanks for the video...With regards to Kafka, 1. When the client starts to watch the video server we can establish connection and store [key value ] the connection it. 2. We can have front end nodes subscribe to specific topics[videos] which it is gonna handle 3. Whenever a like happens we push into kafka and the front end nodes which are responsible for sending the like to all the subscribed connections. 4. they can get it from the stored connection key value store and send the like to the clients through akka actors . What is the issue in that ? Please clarify ?

[@yushutong722]

+ 1 this is the part I'm confused. A given frontend server does NOT need to subscribe to topics of every live video, but only to the unique ones that its connections are watching

[@gabbah79]

I was thinking the same thing. Just two years later. ;)

[@munjal55555]

I think what he means here is even a server do not have to read a message (because their is not user that is watching a particular topic), your server still has to read the kafka message and then may be discard it. so that one extra read will be their. and if their are lot of instance like this(lets say you have 1M different streaming happening at the same time and each have different set of user) , then it might slow down the system.

[@PuneetSethi89]

Great video Akhilesh! It was very clear, easy to follow and educational.

[@foreverursabhi]

1. How does the KV store get updated when a front end dies? 2. How do the dispatchers find each other? What happens when one dies? 3. How does dispatcher broadcast prevent infinite loops? (1 publishes to 2 and 3, 2 publishes to 1 and 3, and so on) Why not use gossip protocol instead?

[@lazypenguin3156]

I'm still learning distributed systems but here's what I think. 1. We need a time-to-live on the KV entry (I'd think of the KV as more of a cache than persistent storage bc the data is changing some much that I would not want to write to disk). 2. In the single datacenter model, dispatchers don't talk to other dispatchers. In the multiple datacenter model, I would probably use kafka to fan out instead of letting dispatchers talk to other dispatchers. 3. If I understand your question correctly, I do not think that dispatching an event will trigger another dispatch. I believe we only dispatch when a client sends a message.

[@sk4152]

For 1. When a front end server dies, client looses connection. So KV store need not get updated immediately. Clients need to reconnect to a different frontend server and that time new subscription call will happen, which will update the KV store. Until then KV store may have a node which is not active and dispatcher may fail to call. But it can just ignore and move on with other active nodes.

[@allmhuran]

How is the key value store scaling? Is it replicated? Presumably the publication of "likes" does not have to be guaranteed, so inconsistency between replicas is acceptable (ie, some likes are not published to active subscribers)? (Edit: OK, second part of this question is covered in the Q&A, verifying the assumption, but is that also the answer to how the key value store scales?) And is that same premise applicable to other elements in the chain? Eg, if a front end times out in response to a publication request, or a client times out in response to a publication request, is it simply dropped or does it need to be retried? Presumably the former?

[@umamahesh8774]

Very well explained. Great Talk. Thanks

[@javisartdesign]

Pretty interesting talk! I would really like to know how they manage idempotence and duplication for likes and messages.

[@deformercr6680]

Just have an ID for each like I suppose

[@javisartdesign]

@@deformercr6680 you do not want to store each like as a record separatelly, it must be just an aggregation or counter in database. I can think on a map-reduce operation in the stream

[@deformercr6680]

@@javisartdesign yes you don't, but you assign it and store it in the client

[@javisartdesign]

@@deformercr6680 that means you can use different devices and sessions for multiple likes... that is cheating.

[@munjal55555]

really a good presentation. I have a question on your comment that kafka will not be that good solution. the reason you have given is each and every likes has to be consumed by each frontend server, that will be problem only when you have a lot of low traffic videos which increase a chances that on each frontend server their is no user who has subscribed for that vedio/topic. what I think is kafka can also work here. yes the advantage of the presented architecture over kafka one is that yours will be usually a bit faster.

[@ledinhanhtan]

Does the dispatcher publishs data using WebSocket or SSE?

[@willemvdk4886]

It's amazing how everything in IT boils down to layers of abstraction. Whether you're talking about local device I/O or about large distributed systems. It's abstractions all the way down. Once you realize that it gets so much easier to comprehend.

[@forfun7700]

Great video + presentation. Thanks

[@muhammadnuri2910]

Very useful and clear presentation.

[@iamvasu2698]

Really nice.. but what about DB scaling or Key-value store mapping..

[@chaitrakeshav]

Great talk! So informative. Thank you!

[@marvinalone]

I think we have 10x viewers on 11/11 when thousands of sellers selling their products online thru live video

[@abhijit-sarkar]

How does a client find a frontend server?

[@aysarferas2253]

macam mana nak buat?

[@srcmake]

Really great presentation.

[@yashwanthnerella9935]

Awesome 👏

[@zhakunigor]

Why do not include "likes" and "comments" into the stream itself letting client not only show video, but also handle other type of messages in the stream? In this way it's possible to reuse existing video-stream infrastracture fully

[@nikolay6700]

i do not understand if they use Kafka

[@karansaluja3315]

In the multi Datacenter scenario, how do you ensure that a node is subscribed to only one dispatcher ?

[@_overide]

It doesn't have to, node can subscribe to any number of dispatcher. Dispatcher in DC1 simply broadcast likes to all peer dispatcher in other DC and then it get handled as usual.

[@afrozalam5389]

Great talk dude

[@ankk98]

Thanks

[@greyreynyn]

lmao, man do they ever feel goofy being like "we have dedicated infrastructure and engineering for a fucking thumbs up button"

[@greyreynyn]

"I get paid 600k USD a year to let people hit a like button" idk seems goofy to me

[@avisheksanthaliya3714]

More like solve the problem "because its a nice engineering problem" than "because it will help the world". TBH I don't care how many people are reacting to a live video.

[@sahilkharidia563]

Why not add a Dispatcher between client and Frontend Server during Kafka ?

[@viczsaurav]

The problem will remain. The dispatcher will be bottleneck then and cannot be scaled as each dispatcher would have to consume all the events emanating from all kafka streams. Letting dispatcher decide which events to pull for breaks down the data chunk for it to process through thus providing the multiplicity we see at the end.

[@lazypenguin3156]

why introduce the dispatcher? Why not just broadcast to all frontends?

[@lazypenguin3156]

at 47:53 he explains it is a matter of scaling.

[@travelingDeveloper]

Yeah we can broadcast all likes to all frontend nodes but that won't be efficient, as there might be some frontend nodes in which connections does have only red live videos watchers so sending green live video likes to it doesn't make any sense.

[@likoarreola8874]

God and Jesus always