System Design Interview: Design Live Comments w/ a Ex-Meta Staff Engineer

[@TimothyZhou0]

The image of a comments service screaming out into the ether is somehow very funny. Liked and subscribed :)

[@hello_interview]

😂

[@siddharthchaudhary2183]

Great first video for the year! I was waiting for this for a while.

[@hello_interview]

Going to start to picking up pace on the videos to start the year!

[@calvincruzada1016]

We are so back

[@hello_interview]

😂

[@anuragtiwari3032]

Way to go. Couldn't have had a better start to the year 🎉

[@alihussainladiwala5929]

The best content for System Design!

[@hello_interview]

💙

[@Anj2409]

Thanks for the great video. The framework has been really useful and I am incorporating the same Excalidraw boxes in my prep to stay organized. Love love the Guided practice on the website

[@chengzhihu1742]

It's super helpful to read the material on the website as well as seeing this well explained video! Kudos!

[@hello_interview]

Awesome! That's the goal. You're ding it right. Now try it on your own and see how you do :)

[@user-kb5vl8hj4e]

Really enjoy the content! Just have one question for the redis pub/sub. In the previous video design messenger we went with the redis pub/sub approach, and for each user we sub to a topic, that’s also a large number of topics I suppose. But in this video reason we don’t do it for video ids, but rather do it on a courser granularity. Is this because there are way more videos live streaming than live user in chat services?

[@billbach21]

Thank you so much for your videos, I'm a huge fan! 💙A small ask. Could you add smaller sections to your videos? I think it would be great to see all the great questions and topics you discuss in the sections so we can return to them and refresh our memory. For example, instead of having a huge section "Deep Dives", split it up to smaller chunks with specific topics. Again, thanks a lot for making this content! 🙏

[@19pend]

Have my FAANG interview coming up very soon. I got premium on the website (very worth it btw!) but I also find these videos very helpful. Thanks for them!

[@hello_interview]

Our pleasure! Good luck with the interview, you’ve got this!! 💪

[@plankton383]

I just found your channel. This is beautiful content. Thank you! ❤

[@Godfathercat]

Loved this video! Thank you

[@princepatel6852]

Babe wake up!! New video from Hello interview is here :)

[@hello_interview]

😅

[@tabeebyeamin9986]

I just finished my E4 onsite for Meta. No idea exactly if I passed the system design interview yet but I felt confident the whole time. It was thanks to your content and your mock interview services. I studied your videos and articles over and over until I could identify patterns. Every time the interviewer asked me something about a component or why I choose to use or do X it felt easy and I could answer it. Even if I don’t pass I’m so happy with how far I’ve gotten in learning system design (and the system design interview specifically) from almost 0! Thank you so much for all you do. I hope that I will be able to use your content to pass senior+ interviews in the future! EDIT: GOT THE OFFER!

[@arneishprateek6444]

Did you pass? :)

[@tabeebyeamin9986]

@@arneishprateek6444 still waiting lol

[@tabeebyeamin9986]

@@arneishprateek6444 Got an update today. Got the offer!

[@wil6956]

Could you touch more on your strategy for enabling persistent sse connections via aws api gateway? I didn’t think that was possible and thought you needed to directly connect to a backend server. Thanks for all the amazing content! Absolutely the best site for interview prep

[@nickname8668]

Just curious why cannot we adopt a similar approach in WhatsApp design, instead of mapping from video to see connection, mapping from user Id to connection. keep a record of how many users are in a video comment section （equivalent to a chat room), then every time user post a comement in video, find all the users and push to their respective servers. In this case you wouldn’t need a zookeeper to track which user is connected to which server, as it no longer dependent on video Id but on user id

[@robert23kim]

Is it true that initially we had a read heavy design, but then our design became write heavy because most of the read volume now come directly from the comment service instead of our comments db? Due to read heavy nature of the use case initially, some people might argue that a sql db with read replicas might serve well here. Is that true or is Cassandra also able to support high read volume as well?

[@hello_interview]

Yah this is fair, but really a polling solution is not passing here. Not even at mid-level. So a passing design requires a solution that considers the write throughput over the reads

[@omarmeshaal5995]

What happens if a few real-time servers go down, causing new messages to be lost due to Redis Pub/Sub's lack of message persistence? In such a scenario, should the client implement polling for a certain period to ensure it receives any missed comments?

[@hello_interview]

They'll lose their connection and when they reconnect, we'll fetch only messages from the DB and restart the stream!

[@LuisDiaz-jo6vo]

Great videos! The videos led me to pay for premium! Thanks Evan!

[@hello_interview]

Appreciate that, glad you like the content!

[@je_suis_onur]

Very good video. Thanks for posting it. I really appreciate it. Couple of additions, if I may: You can’t really use anything other than redis pub/sub really. You need something in-memory (with a passive replica(s) waiting to hot swap) to meet that 200ms requirement. Regular pub/sub systems won’t cut it. They’re too slow. Redis pub/sub doesn’t have a topic limit. The limit is the available memory. But realistically you’d hit the connection limit before topic limit in modern high end machines. For requesting earlier video comments, I feel like having them by user is a bit wasteful no?Wouldn’t there be a lot of copies and db lookups (per user)? Wouldn’t it be better to have one copy per video (sorted list by timestamp)? Do you anticipate that there would be very few who would need this and they wouldn’t need it for too long? Those sorted lists need to be ttl’ed too of course and it can be trimmed from the older end when inserting new comments (maybe an overkill). Another thing is, you’d need to mention that createdAt field is the cluster key for Cassandra again for quick fetching of the comments from db into the cache. You mentioned that there needs to be some indexes but with the volumes we’re talking about it is best if there is no extra index at all. I’ll leave it to the reader on how to make the cluster keys unique per videoId partition key. Reply with @, if you need hints or the actual solution. Finally you probably need a write optimized db here given the amount of incoming data. So an LSM tree based system is going to be needed for real-time comments. So postgre probably won’t cut it, however you partition it (unless you’re ok doing huge scatter-gathers to fetch the comments which defeats the purpose). So Cassandra is again a good choice here but saying that other sql dbs would be fine is probably problematic especially if you’re going for Staff.

[@VikasSingh84527]

Awesome content. Thanks a lot. If possible, please do some more highly scalable systems like view/like counter and reddit threads. Thanks Again.

[@jihanzhang5527]

One thing didn’t mention in this video is that t the the pub sub is extremely difficult to maintain when you constantly having users disconnect and reconnect meaning each server needs to periodically to check whether it should continue to sub a topic

[@nickname8668]

I don’t think so as the topic is partitioned based on video ID not user id. Suppose video is also partitioned across servers, then the server will continuously subscribe to that topic as long as it is till “responsible” for the video id.

[@YeetYeetYe]

There goes my hero, watch him as he goes

[@hello_interview]

😂💙

[@Ahmad3d1]

@hello_interview, can you talk about how the real-time comment instances are initiated (initially, or after a failure)? Lets say instance n is hosting video1 {conn1, conn2} and video2 {conn3, conn4}; and it crashes. How is the connection passed over to another instance? Is there a coordinator, or a leader that helps with that? If so, why this leader component is not essential to be called out in the design interview?

[@Veejay-h2o]

Liked the differentiation b/w SSE and WebSockets and when to use which one. I was also looking for scrolling forward/backward from the current position, which you briefly touched upon. Isn't caching comments per user & video occupy lot of memory, when there are thousands of viewers avg? Since the comments are common for all the viewers (which deviates from the Tinder, since the suggested matches are personalized), why not cache comments just per videoId? Appreciate if you had gone bit deep on this requirement.

[@hello_interview]

The key is that the TTL is pretty tight there. You could absolutely cache per videoId, not per videoId:user pair. This makes more sense for a system like this with no user based filtering. I was just illustrating pagination techniques in general, but admit that was not as clear as it could have been

[@flyingpiggy741]

Should we only use zoo keeper to store the configuration between and videoId and ip address? I am wondering if DynamoDB with DAX could be a good candidate here (DAX is not strong consistency though)?

[@abdo83-k96]

Great design! Is it possible to use DynamoDB CDC streams to send the new comments to the pub/sub channels instead of sending directly from the comment service? That way we only show comments that we are sure are saved in DB? Otherwise the state in db and what is showing might differ slightly in case of errors (unless we add something like SAGA pattern to handle that? which will only make it more complex). Or is it safe to assume this won't happen (or we don't care if it happens very rarely)?

[@hello_interview]

Yup, great call. DDB streams can certainly work. Otherwise, just ensure the DB write is committed before broadcasting. If the stream goes offline for some reason, we just need to force a client refresh to get back to a consistent state.

[@jinghongmiao9963]

+1 I have the same feeling. It seems after using Pub/Sub + SSE, the 2nd API is not needed(or can be simplified). So probably the DDB data is not needed during real time but for other offline processing or post live replay? And great video as always!

[@7gurubhai]

Hi Evan, Great video!! Can I have a feature request. 1. Could you please remove the duplicates from voting page and merge their votes. 2. Automatically vote up the topic requested by a new user if it already exist in the list. The page will look cleaner for the audience and it will help you prioritise the next topics.

[@hello_interview]

Yah totally, need to get to this. Plus a search. Will do this week :)

[@kushpathak4097]

Great video! Slightly off-topic, but is there a way to display keybinds as you press them? I was practicing system design on Excalidraw, and not knowing shortcuts felt limiting. I could look them up or practice more (admittedly lazy), but seeing the keys as you use them would be super helpful.

[@kevinzhang1335]

Quick question. I agree with the approach where you have single video_id per server or group of servers so that you can have 1 connection to redis per server. One question i have is this poses somewhat of a fault tolerance issue. I.e if the server goes down, this can be catastrophic for all users subscribed to that video. What are some approaches to handling this issue? Is it to have a backup sse connection on a passive server that can failover?

[@codecrafter-y2k]

Thanks for great video! How would you handle popular videos?

[@Curious-b2x]

Great video , basic question on apis, is there a reason why nothing about users and their info was spoken about? We do have userId in comment object but should there be an indication of that info being obtained from auth or how that would be captured and mapped to right caller ?

[@hello_interview]

Yah, i talk about this in each video so think it just slipped my mind. You have a session token or JWT which has the authenticated users info. You wont put that in the post body or path

[@robert23kim]

Great video as always

[@hello_interview]

💙

[@chinmaymaheshwari9998]

How can one real time connect server(machine) can connect to 1M SSE connections at a time? because as far as i understand a machine only has 65,535 ports so it can't handle more than this as we need to block each port for each SSE communication? if this is the case how can we scale it to 1M videos with 1000 concurrent users? Also great content really helpful.

[@go4knk]

@hello_interview I often get confused when to use uni-directional vs bi-directional arrows. could you pls clarify. Is it based on data flow?

[@hello_interview]

Based on the read/write ratio! If a given user is writing as much as they’re reading (like a chat app), then it’s bidirectional (websocket). If it’s mostly just reads, with few or no writes, then unidirectional (sse)

[@Jimmy-j7z]

Hi Evan, noticed you're wearing a black flower while recording, hope everything is going well. Thank you for this post, learned a lot from it!

[@hello_interview]

Aw you're too nice. It's actually just a goofy mic :)

[@rostislav_engineer]

thanks for such gold video

[@hello_interview]

Cheers!

[@19pend]

For scaling the database, you mentioned having VideoID as shard (partition) key and createdAt as sort key. This means that you can only have one unique comment per videoID per timestamp. This doesn't really seem correct since you can have multiple comments coming in at the same time, as unlikely as that is, for a video. Should it be a multi-column partition key, like PRIMARY_KEY( (videoID, commentID), createdAt)? I think this would work since it'll partition based on the first part of the partition key, but also maintain uniqueness per videoID/commentID pair, and then it'll be sorted by timestamp for fast range queries? If my concern is valid, how would this be tackled in dynamoDB which doesn't support multi-column partition keys? I guess you could just use commentID as partition key and createdAt as sort key, but also have a secondary index on the videoID?

[@hello_interview]

yah this was loose from me. sort key would be commentID which would need to be monotonically increasing id like cuid. Thus, allowing you to sort by time

[@jieyin4169]

Hi Evan, thanks for the video. I remember the previous FB live comment system design goes a lot of details of cursor/pagination etc. Can I ask why opt out that part in this video?

[@hello_interview]

Not as interesting as persistent connections and scaling them and aim for around 1 hour. The pagination stuff is pretty general to most apps

[@jieyin4169]

@ thanks

[@billwang4844]

How does the realtime connect service know that we dont need to send the comment to a specific client anymore? Like, the user left the video.

[@joaop10]

When the user disconnects, the real-time service responsible for that connection will just update its internal hashmap, so "video1: [conn1, conn2]" will just be "video1: [conn2]". So, when that service receives the pub/sub message, that connection will simply no longer exist

[@mythilikarra7297]

Would love a video/premium AI question option on News Feed! I think Metas been asking that a good amount recently

[@hello_interview]

We gotchu!

[@Veejay-h2o]

In the NFR, is it "millions of videos" or "millions of viewers"? Awesome content, btw!

[@hello_interview]

Millions of concurrent videos. Thousands of viewers each (avg). Most with none or few, some with many millions

[@Veejay-h2o]

@@hello_interview Thanks for the reply. Liked the differentiation b/w SSE and WebSockets and when to use which one. I was also looking for scrolling forward/backward from the current position, which you briefly touched upon. Isn't caching comments per user & video occupy lot of memory, when there are thousands of viewers avg? Since the comments are common for all the viewers (which deviates from the Tinder, since the suggested matches are personalized), why not cache comments just per videoId? Appreciate if you had gone bit deep on this requirement.

[@AGuilmon]

Should we be talking about time series databases here? Or is that out of scope? Probably more for an analytics OLAP followup...

[@hello_interview]

No need! No use case here, but even if there was: www.linkedin.com/posts/evan-king-40072280_its-always-surprised-me-how-often-candidates-activity-7283206383368380416-kVhZ?

[@AGuilmon]

@hello_interview wow you just posted that! How coincidental :) Seems better to keep it simple as possible with proven tech than just shouting buzz words. It's tough to decide when you haven't had a chance to work at a large scale. Thanks for the answer and awesome content, it's the best I've seen. Hoping to ace my loop next week with Amazon thanks to all your prep and premium materials!

[@tusharsrivastava4328]

Great video as always. Please make video on top k leader board problem and any recommendation system.

[@hello_interview]

We have top k KZbin videos already! Leetcode has leaderboard

[@laxman1225]

request to make a video on fb internationalisation?

[@hello_interview]

Vote for it here please :) www.hellointerview.com/learn/system-design/in-a-hurry/vote

[@gremlan6419]

I hope I don't get an interviewer who is myopic and focuses on the back of env calcs, being a product of the US school system and slightly dumb, I suck at arithmetic and would like to avoid it at any cost. Thanks for the video, there are plenty of us out here waiting for them in 2025!

[@hello_interview]

Lots more to come to start the year!

[@jeevteshsingh7227]

Can you do online multiplayer gaming system design? PS: thanks for all the content, I am a premium subscriber!!!! You are the best.

[@hello_interview]

You rock! Upvote it here please. Will try to get to it :) www.hellointerview.com/learn/system-design/in-a-hurry/vote

[@JudgeHolden]

please do a video on design of ebay

[@hello_interview]

We have online auction written up on the site for premium users. Check that out in the meantime!

[@JudgeHolden]

@@hello_interview premium? bruh...

[@hello_interview]

Don't sweat, still tons of free stuff and always will be. Just extra for those who want to go deeper

[@andrewzhu-h2g]

@@JudgeHolden this ain't charity bro

[@Molson31]

KZbin live comments use basic polling, so.. if it's good enough for them..

[@hello_interview]

engineering.fb.com/2011/02/07/core-infra/live-commenting-behind-the-scenes/

[@systemdown96]

Do the realtime comment services absolutely need to have the state in memory? Would keeping the connection state in Redis, then doing something like GRPC unidirectional from comment service work instead?

[@hello_interview]

Why would be my question. They have to have the connection so you can’t escape them being stateful. No reason to add a new service/component to manage. Plus then you need them to be consistent

[@Jimmy-j7z]

I think the idea (author proposed) is that the "granularity" the message is forwarded (between Comment Service, Pub/Sub, Realtime Comment Service) is per "video id", and once RCS (Realtime Comment Service) received the message, it then sends message to each individual clients (who participated in the live video and and connected to it). Of course, you can adjust the granularity to "per client", in this case, the Comment Service will load all the user list of the video, then send request to related RCS which will forward to client.

[@jeevteshsingh7227]

I have E4 software engineering Infrastructure role interview, should i be focused more on these top 5 or questions like design cache, message queue, etc?

[@hello_interview]

These. But cache does come up in meta E4 with some regularity too

[@avawilliams2704]

Would dynamoDB be an acceptable DB?

[@hello_interview]

Totally!

[@andrewzhu-h2g]

a single server can handle a million SSE connections? This sounds like will be limited by the hardware i.e., ram etc Do you have article links that can back this up? Most online source suggests somewhere in the range of 10k-30k. Im assuming it also needs some sorta of kernel bypass to scale

[@hello_interview]

Yah this is correct. I should have expanded here. This takes careful optimizations. I believe Netflix famously got to 1M here. But will need to dig up the article (on mobile). In any case, the scaling challenge is the same

[@kamal9991999]

Can you please design for E-commerce

[@hello_interview]

Which part of an Ecom site? We have both Ticketmaster and gopuff

[@kamal9991999]

@ ok ..

[@kamal9991999]

@@hello_interview actually looking something like Amazon or Apple Store etc .. let me go thru ticket master first .. don’t see gopuff

[@yashgarg9458]

​@@hello_interviewComplete e commerce like amazon

[@jeevteshsingh7227]

If this is second most popular, can you list top five at meta?

[@hello_interview]

leetcode, this, ticketmaster, top k, web crawler (variant)

[@robert23kim]

I recently had top k Spotify songs per user and twitter search (was more like design elastic search) for meta staff level

[@dhiraj5721]

@robert23kim how did you prepare? Could you share any particular resources which you found useful apart from this? TIA

[@robert23kim]

@@dhiraj5721 read Alex Xu 1 and 2, ddia, mock interviews and hello interview site. Most important I would say is identifying most asked questions, practicing them and doing mocks. Don’t let mock feedback fool you too. In my experience feedback I got was better than reality.

[@dashofdope]

wow. 1B concurrent users. so 1/8th of the world is pausing to tune into our sick app we just mocked up

[@hello_interview]

lol. 1k viewers per video is def high. But you’d have a Pareto distribution for sure.

[@hello_interview]

Should out to our live Q&A this Thursday which will probably have 2B.

[@dashofdope]

@@hello_interview should come?

[@dashofdope]

@@hello_interview when's the Q&A?

[@hello_interview]

@@dashofdope kzbin.infoFbEKrBcrLiI

[@ricardofilipegomes4396]

Amazing one Evan :) What are your thoughts on using kafka as the pub sub pipe for Staff level, is it passable ? Recently had one Systems design where i used kafka as my consumer/producer for websockets real time connection, i feel that if you need to deliver things, you can use actors (which i really like for this solutions), the inbox is kafka topic and how you send from actor A -> B, it can distribute pretty good, but i felt i didn't articulate well for a E6 interview, especially on meta, feel so strangled for time, that i almost felt that i still didn't had a time to really explain every trade off tbh I did 2 (since i jumped from the EM role to the IC), my EM one went really well, and i got E6 rated like, but the whatsapp on think i bombed for the IC stream, still have one left :) When my interviewer prompt me for cassandra strangled, i promptly said we could divide cassandra into hot/cold storage and use zookeeper to keep it smaller, play with the amount of nodes and also the read/write quorum, he seemed to really like the approach and bought in. But lets see if i can get back to the E6 in the next, but i also feel most times i reach the end of the interview with some impostor syndrome, do you feel sometimes staff level candidates brezze trough some of the topics ? Or they absolutely nail every single detail and every explanation ? With meta especially i fell 35m is quite harsh in terms of time to showcase and think :) Btw your content is pure gold!!! This is so amazing, and helped me scale as a professional as well ass in my interviews!

[@ricardofilipegomes4396]

Premium is also pure gold, and support this guys, 40$ a year is nothing for the content they are putting up for us :)

[@hello_interview]

Dang thats so awesome to hear. You rock :) Thank you!

[@hello_interview]

Kafka itself is fine and not disqualifying by any means. It just depends on how well you can talk about scaling it and answer interviewer's inquiries.

[@ricardofilipegomes4396]

@@hello_interview thats brilliant, that part i felt i covered well, but it is also a bit on time management and giving the signals imo, which is 35m in meta is quite tricky, i really value the framework you guys put in, it makes it way simpler to laser focus on cleaning all FR first, and then going into the deep dives in a structured way, to make sure you can show how can make a workable simple system, and then kinda of solve scale problems in 2 distinct phases, which helps load imo :) Again thanks a lot for the brilliant content, this is incredible, hopefully i can work with you guys as a coach somewhere in the future :)

[@thegt]

There a people who apply for infrastructure positions. Listening to your dismissal of some parts “infa” is scary. What should these people do?

[@yogeshrnaik]

Very good video! I also came across this video from LinkedIn that is also very good. Streaming a Million Likes/Second: Real-Time Interactions on Live Video: kzbin.info/www/bejne/r6LGZIOGoq2pqKM #akka