Photon - Exactly Once Stream Processing | Distributed Systems Deep Dives With Ex-Google SWE

Рет қаралды 2,727

Күн бұрын

Пікірлер: 13

@mark-6572 3 ай бұрын

Hi Jordan! On the topic of streams, I've seen both your videos on flink. But also realised the amount of times it's been used in systems design questions and felt a deep dive on what you can do with flink (instead of how it works under the hood) would be really useful. I learned more about flink from the deep explanations you've given in the system design questions but felt a separate video would do it justice. You've made system design really enjoyable, thanks for all your hard work. Loving the channel!

@jordanhasnolife5163 3 ай бұрын

Thank you Mark! Hope to get to a whitepaper for that one very soon!

@ready-mcready 3 ай бұрын

Hey Jordan, this is several weeks overdue, but just wanted to say thank you from the bottom of my heart for your content. Late last month, I was able to land an E4 role at Meta and I strongly believe the system design portion of the interview would've been a lot harder without your System Design 2.0 series. You rock man!

@jordanhasnolife5163 3 ай бұрын

Congratulations dude! Really happy for you, thanks for the kind words and go kill it in the new role!

@reddy5095 3 ай бұрын

Batch writes to paxos can end in failures. Say T1 is writing 1,2,3 ids and T2 is writing 2,4,5 . Both the writes will fail or one of them needs to fail. My understanding is this case is rare since they went ahead and used batch writes. I wan't to know why this is rare or less likely to happen

@jordanhasnolife5163 3 ай бұрын

To be clear, the batching process resolves all conflicts within a batch because all of the writes are ordered, so one of them would be aborted before the actual paxos commit.

@reddy5095 3 ай бұрын

Batches are created by joiner and written to some db via paxos, across joiner they are not ordered . At paxos they are. How often do joiner batch writes fail due to this problem

@jordanhasnolife5163 3 ай бұрын

@@reddy5095 "Although each client may be able to bundle multiple key commits into one RPC request, when there are a large number of clients, client-side batching is not always effective. We describe two mechanisms to achieve high scalability with the IdRegistry: server-side batching and sharding. " "As shown in Figure 3, the IdRegistry server has a single background thread (called registry thread) that dequeues client RPC requests, translates them to PaxosDB transactions, and sends the RPC response. The registry thread dequeues multiple requests and batches them into a single PaxosDB transaction. Within a batch of requests, the registry thread performs application-level conflict resolution. Consider the case where multiple requests try to insert the same event id into the Id580 Registry. If the event id does not already exist, the registry thread will insert the event id into PaxosDB, and respond with success to one client which marks the event as joined; other clients will receive a failure message which must drop the event. We take advantage of multi-row transactions in PaxosDB (updating multiple key/value pairs atomically with condition checks) to implement this behavior." "Some stream processing systems [26] recommend grouping individual events into a bigger batch. This works well only if all the events in a batch can be processed at the same time. For our case, it is very common to see a click event before its corresponding query event. If we group multiple clicks into a batch, then the whole batch cannot be committed until the corresponding queries are available. Hence, system designers need to make a careful decision on whether to batch or not." They're only doing server side batching here.