Notification Service Design Deep Dive with Google SWE! | Systems Design Interview Question 22

Рет қаралды 15,506

Күн бұрын

Time to get a women @ google shirt next, need to add to the collection
00:00 Introduction
01:18 Functional Requirements
02:08 Capacity Estimations
02:51 API Design
03:44 Database Schema
04:45 Architectural Overview

Пікірлер: 77

@cambriandot3665 Жыл бұрын

5:30 Stream processing 6:14 Message brokers 7:30 Duplicate messages - 2 Phase commit, idempotency - bloom filters 9:55 Queues - partitioning, zookeper 10:43 Fan-out to users 13:23 Notifications database - Hbase 14:23 Topics database - MySQL 15:02 Architecture

@fangao9977 Жыл бұрын

Questions here: I'm assuming that the publishMsg API will finally trigger the service to push SSE msg to the Frontend Users, so the consumer servers as the ones who pushes the messages, but how does it know that the client is connecting to itself, what if the notification consumer picks up a message and find that the client is not connected to it ? Is it being controlled by the UserId Sharding queue ? so that the queue always know which consumer to send to ?? (and what if a consumer is down, does the queue know it and take care of the reconnected consumer ?)

@jordanhasnolife5163 Жыл бұрын

Yeah you basically said it. We're partitioning based on userID. If the consumer goes down, consistent hashing will ensure that those messages and the client frontend will connect to a different consumer, but the same one.

@raghavkaushik4986 Ай бұрын

Awsome video dude! Just one thing missed CDC from TOPICS table to Flink using message queues partitioned on userID.

@jayepeg4388 Жыл бұрын

How would you deal with spiky traffic - e.g. load increases by 10-100x because of some special event (black friday sales or something)?

@jordanhasnolife5163 Жыл бұрын

Well at the end of the day your only option for something like that is to add more resources, right? Assuming you're doing this in AWS I'd imagine you'd have some autoscaler to give you more when traffic increases.

@sanampreetsingh9638 Жыл бұрын

Great design ! Of restricting user to just one web socket connection with server by queuing messages ! Couple of questions ! 1-What’s the implication of user subscribing to topic (topic to user list mapping I mean ) topics are internal to our notification service ! For example user May be interested in getting notified for some product X coming in stock , so our general topic can be product in stock but user is only interested for product X in stock notification ! ? Thanks

@jordanhasnolife5163 Жыл бұрын

I think that the consumer node would basically just have to go ahead and read it and determine whether it needs to send the message to any users.

@AnkitGaurav-qt1hs 4 ай бұрын

What changes would you do in your design if a requirement comes in where notifications have different priority, like P0 notifs need to be sent before any P1 and P2 notifictaion?

@jordanhasnolife5163 4 ай бұрын

Probably just ensure that they're going to the same kafka queue via some sort of partitioning key. Kafka queues are consumed one at a time, so that should ensure the ordering for me. If you mean that you expect the P2 to be delivered after a P0 but it is published first, well I think you'd have to clarify your requirements a bit more. Clearly you'd need to buffer the events for a bit and use some sort of priority queue to occasionally poll one and send it to the world.

@imutkarshy Жыл бұрын

Following up on the user id shards. Do you mean using user ids as partitions for 2nd log based queue? If yes, then will it not be unoptimised to have 1 billion partitions (as per capacity planning) even tho there is no limit on the numb of partitions?

@jordanhasnolife5163 Жыл бұрын

I think that you're maybe slightly misinterpreting - being sharded by userId doesn't mean that there will be only one userId per partition, but rather that we just partition on the hash range of the userId field - there could potentially be many userIds per partition, it's just important that all the messages for a given userId are sent to the same node so that we can keep a persistent websocket connection on it

@imutkarshy Жыл бұрын

Thanks, @@jordanhasnolife5163 for the quick reply. It's making sense now. These videos are super helpful. Keep them coming. Love from India

@vanderer_prashant 6 ай бұрын

I see only Flink in between topics message queue and user message queue in the diagram. Who puts message from topic MQ put into user MQ?

@jordanhasnolife5163 6 ай бұрын

Yeah very possible I forgot it here, you'd probably want some other stateful consumer that is aware of which queue a message from a given topic should be routed to.

@sanampreet3045 Жыл бұрын

Hi Mate , can u please elaborate more on local state management using flink here ? Like consumers are sharded on hash of user id and topic has list of users to notify , how state management is happening here !

@jordanhasnolife5163 Жыл бұрын

Basically the consumers know for each userId, which topics they are subscribed to, so they can easily send messages to the appropriate queues. We just get better data locality out of doing this than having to consult a database for this information. It effectively is just a cache.

@dibll Жыл бұрын

Why do we need to use Zookeeper to store metadata like partitions in a topic , IP Addresses of those partitions etc. Why can't we use a DB for that purpose? What are the advantages of using Zookeeper over a normal key value DB

@jordanhasnolife5163 Жыл бұрын

Zookeeper is strongly consistent which is very important to have when dealing with actual configuration of your architecture so you don't make stale reads Recommend watching my ZooKeeper video

@VivekMahadevanv Жыл бұрын

Followup question.. Why do you think a userId sharded 2nd level of MQs' are necessary? In most practical applications such as say chat for example, the notifications still go through multiple mediums based on what the user subscribes to (say push notification, email but not SMS). This information could be made part of the payload as tags that the broker can aggregate on for easier consumption. For any medium, it would then make sense to have a storage mechanism to hold messages that were not delivered. I don't think the current design saves any resources on the server(threads?) on the server either since you probably need to maintain a connection with each device the user has in order to send the message. Thoughts?

@jordanhasnolife5163 Жыл бұрын

Yep, the point was actually to optimize for the client device. If a client has to maintain an active connection for every single topic that they are subscribed to, they need to maintain potentially hundreds of active connections, which to me at least seems excessive. I'd agree that while perhaps a second set of message queues aren't necessary, it seems pretty clear to me at least that the client should only be pulling from one device as opposed to actively maintaining many TCP connections. For a phone at least, that could really use a lot of data/drain battery life.

@VivekMahadevanv Жыл бұрын

@@jordanhasnolife5163 interesting. How many of these connections must one hold ? Unless it's real time (chat or something of that nature) isnt it mostly a fire and forget (email/sms) . Either ways worth having a conversation with the interviewer I guess ..

@jordanhasnolife5163 Жыл бұрын

@@VivekMahadevanv If it's fire and forget I agree - however I've seen notification service talks that assume you're actually directly pushing to the service, in which case all the messages for a user have to go to the same place

@simin2371 Жыл бұрын

@@jordanhasnolife5163 isn't the second queue also necessary to ensure the order of delivered messages for a given user? the first queue is topic-based, this is necessary as it enables per-topic heavy post-processing right after the first queue, which is quite likely. Then the second queue makes sure messages for each user are in the same queue and shard, hence ensuring the order of message consumption. Finally the notification sender server is 1-to-1 mapped to a user partition so to have only one websocket per mobile client. Am I right?

@jordanhasnolife5163 Жыл бұрын

@@simin2371 That was my logic, thanks for helping to communicate it!

@sanampreetsingh9638 Жыл бұрын

@jordanhasnolife5163 Жыл бұрын

answered above

@Glashutte1900 Жыл бұрын

The problem is that we have a chat app and when the app is in kill state in android or ios the push notification does not work so we go for FCM and we dont want to use firebase as google data issiues so do you think it is possible to build your own push notification in kill state like fcm. i will be looking forward for your answer

@jordanhasnolife5163 Жыл бұрын

Not sure - I imagine there's a way but from what I know about mobike dev most people outsource their push notifications to FCM, one service (I think it's called), or SNS

@Glashutte1900 Жыл бұрын

@@jordanhasnolife5163 Thanks for your kind reply

@SlashTagged Жыл бұрын

On using bloom filters for idempotence: Wouldn't we start getting a lot of false positives at some point (due to hash collisions)? That would lead to erroneously rejecting non-duplicated notifications, which seems counterintuitive to idempotency. We're essentially being so "aggressive" that we're actually rejecting "good" (read: non-duplicated) notifications.

@jordanhasnolife5163 Жыл бұрын

While this is true, you can always just occasionally reset the bloom filter, say every hour. Then you no longer have to deal with this issue.

@SlashTagged Жыл бұрын

@@jordanhasnolife5163 I had thought about that but was worried about the case when you clear the BF but then process an actual duplicated notification. Probably an edge case but worth mentioning nonetheless. Appreciate the reply and thanks for putting in the time to make all these videos!

@jordanhasnolife5163 Жыл бұрын

@@SlashTagged I think in the worst case scenario you'd just check if you've seen it before - remember the bloom filter can only tell you with certainty if you HAVEN'T seen something, so we have to check again if there's a chance we have seen it anyways

@TheImplemented Жыл бұрын

Did you intentionally skipped Subscriptions table (user_id, topic_id) ?

@jordanhasnolife5163 Жыл бұрын

I included a parameter userlist in the topics table that does that but yeah this is probably better

@Gerald-iz7mv 7 ай бұрын

what about celebrity problem? ... would you do pull vs push in this case? you also mention user queues - how do you batch the messages for a user? ... imagine the user doesnt want to receive 20 messages per day - instead 3 or 4 message might be a better user experience...

@jordanhasnolife5163 7 ай бұрын

Yeah I think using dedicated celebrity queues which the user connects to makes a lot of sense there. If you did need to batch messages, you could use stream processing to do some sort of time windowed joining (see the stream processing concepts video)

@Gerald-iz7mv 7 ай бұрын

@@jordanhasnolife5163 how does the celebrity queue work - one queue per celebrity? how does the message get to the 100 million followers than?

@prateekaggarwal3305 9 ай бұрын

Can you desribe a bit more about how Configuration service is helpul here, like what information it holds and how that information is used.

@jordanhasnolife5163 9 ай бұрын

Mostly important for your sharing config. Each device is connected to one notification server, so we need to always be aware of that mapping.

@povdata 9 ай бұрын

Thank you for video. I still have one question. How to to avoid sending duplication messages? You say that unique id is attached to each notification message. How to use it then? It which step?

@jordanhasnolife5163 9 ай бұрын

A consumer will receive messages on a message queue, it will then cache the unique IDs locally. If it's seen that I'd before, don't handle the message again.

@povdata 9 ай бұрын

@@jordanhasnolife5163 So, where is cached? Because it might be some consumers (backend) and each one caches locally, so one consume does not know about cach of another, right?

@povdata 9 ай бұрын

and how is possible to get duplicate message if message is fetched from queue can not be added back?

@jordanhasnolife5163 9 ай бұрын

@@povdata if the message gets delivered to a node but the node is unable to send an ack back to the broker due to network issues the broker may deliver it a second time

@povdata 9 ай бұрын

@@jordanhasnolife5163 Thank you, but I did not get about how to send message using SSE to concrete client user? Could you explain please?

@user-se9zv8hq9r Жыл бұрын

how about temporal for that 2 phase commit

@jordanhasnolife5163 Жыл бұрын

Can you elaborate on this?

@jayshah5695 Жыл бұрын

we should probably add a notification acking mechanism from client back to service?

@jordanhasnolife5163 Жыл бұрын

I think this is definitely useful from the perspective of not dropping notifications, though if it's okay to drop the occasional notification then it isn't a huge deal if we don't acknowledge received notifications and stick them in a database somewhere.

@jayshah5695 Жыл бұрын

@@jordanhasnolife5163 we should discuss delivery rate and delivery SLA as part of requirements I guess.

@jordanhasnolife5163 Жыл бұрын

@@jayshah5695 Very fair point

@TOKYODRIFT00000 10 ай бұрын

I am trying to implement an API for one of the notification services. Let me explain the architecture. There is a gateway service that has methods for creating configurations. A person can interact with the service and create a configuration for sending a message via email, for example. Then the person calls the Notify method, which accepts the following DTO: 1. Consumer information 2. Transport configuration 3. Message: [type (simple, broadcast), content] The Notify method looks at the transport and understands that the message should be sent using the email service, so it sends the DTO to the email service. Ok, so I described how gateway and email transport service work. However, there is a transport called "client" that has not been implemented yet, and it needs to be designed. The main idea is that the client can be a web application, some desktop application, etc. It is clear that delivery will happen through a message broker, but how will it all look? I don't quite understand

@jordanhasnolife5163 10 ай бұрын

I'm confused: you're just asking how the message gets to the client after someone tries to send them a notification?

@TOKYODRIFT00000 10 ай бұрын

@@jordanhasnolife5163 maybe I’m bad in explanation and my english also bad but yes! Lets imagine we have rabbitmq broker with different topics. We want to send notification to a user so what topic to use? Maybe you have an example api?

@jordanhasnolife5163 10 ай бұрын

@@TOKYODRIFT00000 You can make a topic per user

@TOKYODRIFT00000 10 ай бұрын

@@jordanhasnolife5163imagine user is offline, we keep all notifications in the database, but what if there are a lot of them? if a person becomes online and calls fetch_notification, they will immediately receive a lot of notifications and this is bad. how to solve this problem?

@ariali2067 4 ай бұрын

Curious why we use SSE/websocket here instead of long polling? I would assume notification is more like uni-directional communication instead of bi-directional here.

@ariali2067 4 ай бұрын

Also not sure whether I missed anything, is HBase use single leader replication? It mentioned in this video but iirc this is the first time I heard of this.

@jordanhasnolife5163 4 ай бұрын

Long polling is uni directional.

@jordanhasnolife5163 4 ай бұрын

Yeah H-Base replicates in the same way that HDFS does which means that there's typically a single place where writes first go to for a given file.

@ariali2067 4 ай бұрын

Yeah but isn't notification more like uni directional pattern which push notification from server to client? So why we don't use long polling here?@@jordanhasnolife5163

@PerfectSylex 9 ай бұрын

hello everyone. this is application servar and web server. which one do you like more? web servae is my crush

@xmnemonic Жыл бұрын

Good video. Can hear siren in background at 1:55.

@jordanhasnolife5163 Жыл бұрын

That's NYC for ya :(

@RinayraMotwal Жыл бұрын

Can you please also make video on distributed messaging queue

@art4eigen93 Жыл бұрын

wow! that's so fast I literally had to watch in 0.75x :p

@jordanhasnolife5163 Жыл бұрын

The ladies also always tell me how fast I am :(

@art4eigen93 Жыл бұрын

😂

@Elena-kx5lw 7 ай бұрын

Ппц сложно

@nocode659 Жыл бұрын

Start a paid course.. ITs worth it

@jordanhasnolife5163 Жыл бұрын

Haha no need truthfully - I'd rather put my videos out there for people to see, as I think in the long run it's better to get the exposure. Plus, I'm not really qualified to be charging money for this stuff as I'm mostly just ripping it off of other people for the time being. I appreciate the view!

@sanampreetsingh9638 Жыл бұрын

@jordanhasnolife5163 Жыл бұрын

1) There will likely have to be some logic by a consumer to figure out which messages are relevant for which user 2) There's not 1 queue per user, there's just 1 user per queue. You can have many users getting messages through the same queue