Streaming a Million Likes/Second: Real-Time Interactions on Live Video

QCon London International Software Development Conference returns on April 8-10, 2024. Level-up on 15 major software and leadership topics including; The Tech of FinTech, What's Next in GenAI and Large Language Models (LLMs), Performance Engineering, Architecture for the Age of AI, Innovations in Data Engineering and more.
Learn the emerging trends. Explore the use cases. Implement the best practices.
Register now: bit.ly/3rRgsJV
----------------------------------------------------------------------------------------------------------------------------------
Video with transcript included: bit.ly/3nwZkBt
Akhilesh Gupta does a technical deep-dive into how LinkedIn uses the Play/Akka Framework and a scalable distributed system to enable live interactions like likes/comments at massive scale at extremely low costs across multiple data centers.
This presentation was recorded at QCon London 2020: bit.ly/2VfRldq
#LinkedIn #DistributedSystems #Scalability

[@vikram_saha7]

Perfect presentation !! The way in which he delivers the complex design choices in an easy and fun way is amazing!!

[@-indeed8285]

Enjoyed way he present.. (Proper slide, Great transitions, No breaks, engaged with audience, making jokes and all of that without losing the context.)

[@venkatapavankumarsannisett2259]

Brilliant articulation and to the point. Also for the resources in between. Thanks to infoq for such amazing real world scalable stories from the industry

[@umamahesh8774]

Very well explained. Great Talk. Thanks

[@Rkrish2010]

Awesome many thanks 😍😍☺️👍😊🤗😱

[@chaitrakeshav]

Great talk! So informative. Thank you!

[@codetolive27]

Amazing presentation. Loved the way he has presented.

[@visheshgupta4990]

Thank you so much I watched the complete video and it is very helpful 🙌

[@srcmake]

Really great presentation.

[@naveenjohn3402]

Wonderful talk. Very engaging and beautifully presented

[@kanthipavuluri3874]

Totally loved this evolutionary styled presentation, brilliant story telling!

[@natadilip]

Amazing Explanation. This is clearly one of the best videos on qcon.

[@AssassinoYunus]

What is the raw computing power of one single Frontend machine? How many cores and ram?

[@RahulKumar-lp5jc]

One of the best talks I have seen. Hats off to you. I have a doubt here. In more than 10k connections case, why do we need an additional layer of frontend nodes? Instead, why can't we store the mapping of video and Real-Time Dispatcher node in the distributed key-value store. The Likes backend can query the key-value store to get the dispatcher nodes for sending the like object. Once it reaches to dispatcher node, the node has the in-memory subscription table to send to the corresponding end user.

[@NK-ju6ns]

Very engaging still calm and neat presentation. Insightful.

[@forfun7700]

Great video + presentation. Thanks

[@muhammadnuri2910]

Very useful and clear presentation.

[@allmhuran]

How is the key value store scaling? Is it replicated? Presumably the publication of "likes" does not have to be guaranteed, so inconsistency between replicas is acceptable (ie, some likes are not published to active subscribers)? (Edit: OK, second part of this question is covered in the Q&A, verifying the assumption, but is that also the answer to how the key value store scales?) And is that same premise applicable to other elements in the chain? Eg, if a front end times out in response to a publication request, or a client times out in response to a publication request, is it simply dropped or does it need to be retried? Presumably the former?

[@PuneetSethi89]

Great video Akhilesh! It was very clear, easy to follow and educational.

[@amolnigade8830]

Insightful information on live video interactions

[@user-jt5nd3yq4u]

Amazing presentation, learned a lot from it!

[@iamvasu2698]

Really nice.. but what about DB scaling or Key-value store mapping..

[@preetiirrothi744]

The best presentation i’ve seen!

[@yashwanthnerella9935]

Awesome 👏

[@javisartdesign]

Pretty interesting talk! I would really like to know how they manage idempotence and duplication for likes and messages.

[@deformercr6680]

Just have an ID for each like I suppose

[@javisartdesign]

@@deformercr6680 you do not want to store each like as a record separatelly, it must be just an aggregation or counter in database. I can think on a map-reduce operation in the stream

[@deformercr6680]

@@javisartdesign yes you don't, but you assign it and store it in the client

[@javisartdesign]

@@deformercr6680 that means you can use different devices and sessions for multiple likes... that is cheating.

[@afrozalam5389]

Great talk dude

[@anandt8362]

Thanks for the video...With regards to Kafka, 1. When the client starts to watch the video server we can establish connection and store [key value ] the connection it. 2. We can have front end nodes subscribe to specific topics[videos] which it is gonna handle 3. Whenever a like happens we push into kafka and the front end nodes which are responsible for sending the like to all the subscribed connections. 4. they can get it from the stored connection key value store and send the like to the clients through akka actors . What is the issue in that ? Please clarify ?

[@yushutong722]

+ 1 this is the part I'm confused. A given frontend server does NOT need to subscribe to topics of every live video, but only to the unique ones that its connections are watching

[@gabbah79]

I was thinking the same thing. Just two years later. ;)

[@ankk98]

Thanks

[@raymondsquared]

Really professional and easy to follow presentation. Loved it! Whats the benefit of using long polling vs websockets?

[@curiousspirit3947]

curious too

[@gxbambu]

You don't really need websocket here, cuz sending likes and comments don't need to be real time

[@rahulverma2523]

@@gxbambu Yea, but they are calling it the "realtime" platform. Also, the platform is used for multiple use cases and not just for sending likes.

[@ekamaloff]

@@gxbambu the whole point of this presentation was distributing likes and comments in real-time , which is required for live streaming

[@akashjain2990]

He kind off explained that in the first question from audience. Reason is : Long polling using server side events are basically plain HTTP requests; so support even very older devices; do not get blocked by firewalls etc. Both of these problems exists with WebSockets.

[@foreverursabhi]

1. How does the KV store get updated when a front end dies? 2. How do the dispatchers find each other? What happens when one dies? 3. How does dispatcher broadcast prevent infinite loops? (1 publishes to 2 and 3, 2 publishes to 1 and 3, and so on) Why not use gossip protocol instead?

[@lazypenguin3156]

I'm still learning distributed systems but here's what I think. 1. We need a time-to-live on the KV entry (I'd think of the KV as more of a cache than persistent storage bc the data is changing some much that I would not want to write to disk). 2. In the single datacenter model, dispatchers don't talk to other dispatchers. In the multiple datacenter model, I would probably use kafka to fan out instead of letting dispatchers talk to other dispatchers. 3. If I understand your question correctly, I do not think that dispatching an event will trigger another dispatch. I believe we only dispatch when a client sends a message.

[@karansaluja3315]

In the multi Datacenter scenario, how do you ensure that a node is subscribed to only one dispatcher ?

[@_overide]

It doesn't have to, node can subscribe to any number of dispatcher. Dispatcher in DC1 simply broadcast likes to all peer dispatcher in other DC and then it get handled as usual.

[@willemvdk4886]

It's amazing how everything in IT boils down to layers of abstraction. Whether you're talking about local device I/O or about large distributed systems. It's abstractions all the way down. Once you realize that it gets so much easier to comprehend.

[@marvinalone]

I think we have 10x viewers on 11/11 when thousands of sellers selling their products online thru live video

[@zhakunigor]

Why do not include "likes" and "comments" into the stream itself letting client not only show video, but also handle other type of messages in the stream? In this way it's possible to reuse existing video-stream infrastracture fully

[@abhijit-sarkar]

How does a client find a frontend server?

[@aysarferas2253]

macam mana nak buat?

[@nikolay6700]

i do not understand if they use Kafka

[@sahilkharidia563]

Why not add a Dispatcher between client and Frontend Server during Kafka ?

[@viczsaurav]

The problem will remain. The dispatcher will be bottleneck then and cannot be scaled as each dispatcher would have to consume all the events emanating from all kafka streams. Letting dispatcher decide which events to pull for breaks down the data chunk for it to process through thus providing the multiplicity we see at the end.

[@greyreynyn]

lmao, man do they ever feel goofy being like "we have dedicated infrastructure and engineering for a fucking thumbs up button"

[@greyreynyn]

"I get paid 600k USD a year to let people hit a like button" idk seems goofy to me

[@avisheksanthaliya3714]

More like solve the problem "because its a nice engineering problem" than "because it will help the world". TBH I don't care how many people are reacting to a live video.

[@lazypenguin3156]

why introduce the dispatcher? Why not just broadcast to all frontends?

[@lazypenguin3156]

at 47:53 he explains it is a matter of scaling.

[@travelingDeveloper]

Yeah we can broadcast all likes to all frontend nodes but that won't be efficient, as there might be some frontend nodes in which connections does have only red live videos watchers so sending green live video likes to it doesn't make any sense.

[@likoarreola8874]

God and Jesus always