Ask Confluent #6: Kafka, Partitions, and Exactly Once ft. Jason Gustafson

[@tlberglund]

Confirmed: calling me "young man" definitely makes my day. :p

[@jessyndaya9487]

🤣😂

[@sachindeokar4798]

Than you Gwen and Jason for answering my question :-)

[@semragenima7587]

Thanks for your explanation, especially the exactly-once semantic, but I still have some doubt on how to choose the partitions number, I still have some customers splitting the same data flow in to different topics, where topic equal to client.

[@asht7290]

Dang, I got distracted trying to figure out how she could write backwards with such fluidity and ease! LOL 😆

[@kevinfleischer2049]

I had the same question.... but than you look on the logos and everyxthing clears up.

[@sachintiwari1579]

I believe you misunderstood last question (Abhishek) completely. What is asking is that latency of events getting processed in separate output streams. In simple words, update occurred in DB but cache is yet to be updated. Just to be clear, exactly once is not going to solve this problem as it is about parallel processing of events in output topics.

[@mazenezzeddine8319]

So Exactly once processing is guaranteed under fault tolerance? For instance, say one of the servers in the producer, kafka cluster, kafka stream cluster or consumers is down can Exactly once guarantee recovery (e.g, from a savepoint) with exacly once guarantee?

[@mazenezzeddine8319]

So given a stream of events e1, e2, .... that traverses the whole pipeline (producer, kafka cluster, kafka stream cluster, consumer) at time t1, t2, under at least once guarantee, the stream of events will traverse the whole pipeline at t1+~100ms, t2+ ~100ms under excaly once processing? is that accurate? or i shall reintepret the conclusion otherwise?

[@terafl]

I would really like to start using EoS in my producers, but unfortunately, it's not yet supported by librdkafka, my client library of choice. It would certainly help with the adoption of this feature since a lot of other clients depend on librdkafka so much (e.g. CGO-based Kafka client for Golang). Thanks for a great show!

[@Tamirallamiro]

excellent explanation jason and Gwen

[@αλήθεια-σ4κ]

One person speaking and other writing in little distracting...

[@AleksandarT10]

I still did not understand if Exactly Once works 100% on Consumer Side, i guess it does not work unless you handle the commits and processing of the message on the Consumer Side with some logic (like Inbox pattern)

[@rajaraodv]

good topic but super distracting that that one person is writing and the other person is talking

[@sabside]

Great conversation but the eagerness of the lady to write notes while the gentleman is explaining is very distracting, even distracting to the person trying to explain.

[@srikanthjnr]

Seems like a very uncomfortable setting to do a Technical Q&A?

[@rahulgoti3864]

Thank You