Data Plane - Apache Kafka®'s Replication Protocol

[@ajaykanojiya4054]

excellent.. But I couldn't understand, what is epoch. It will be great if you light more on epoch.

[@stepander007]

Leader's epoch is something like President's elections. First president, Second president and so on. All the time leader is changes(new president elected) you have this number increased.

[@cronus663]

Just complementing the previous answer, the epoch is just to mark what leader was responsable for writing which data, this is important to run the reconciliating flow, it's important to know the offset of the last commited record from the previous leader!

[@TracyZhang-j1l]

What does the "follower caught up all the way with the leader" mean? Is it the last time the follower has sent a Fetch request successfully with a right epoch and offset?

[@PooyaWDG]

When does a follower know to start fetching the new data? is it a polling approach?

[@ragupathia2316]

Hi, Suppose broker 103 elected as new leader, it has epoch value 1 till offset 3. But broker 102 has epoch value 1 till offset 5. Due to reconciling, 4,5 will be deleted, it means those 2 data will be ignored to commit. Is that producer data lost ?

[@marc-andrelabrosse5379]

That means that we lost 2 record in the reconciliation between the lader and the follower is it not? Why did the election did not choose the broker 102. to not loose those records?

[@CarterChen]

because the 2 records are not yet considered as committed, since they are only replicated to one follower of the ISR (102, 103 both are in ISR), so 103 is a valid candidate for competing leader

[@quizforces]

@@CarterChen So for those two messages would the client know that the messages could not reach the kafka cluster and they need to be sent again ?

[@fanzhang1370]

this depends on the producer config of ack. if not set to all there can be data loss

[@samson279]

4:07 it's very weird that the leader advanced it's high watermark after receiving follower fetch request from 1 follower, even though there are 2 (!!) followers in ISR. This video doesn't explain this.

[@pra86sat]

HighWatermark is advanced by leader after it receives the fetch request from follower with the updated offset 3:45: broker 102 request for offset 3 and broker 103 request for offset 3. Now broker 102 and 103 are not aware of the HighWaterMark, until the leader propagates this info via the response for the fetch request.

[@РоманБондарь-ы8щ]

Does not really explain what preferred replica is and what it does. The part about leader epoch is also unclear

[@martykate]

the replica that was the leader when the topic was originally created. It is preferred because when partitions are first created, the leaders are balanced between brokers

[@maaruiusagi7400]

What if broker-102 is elected as the new leader? When broker-103 sends a fetchRequest with offset=3 and lastFetchEpoch=1, will broker-102 know that it needs to drop events 3 and 4 since they came from epoch 1 (the now-failed broker-101) but never received by broker-103 (thus they are probably never acked), or will it still send them to broker-103 though they were never acked?

[@DivijVaidyaIndia]

The latter. In Apache Kafka's replication protocol, the following invariant always holds true: - the log of the current leader is never truncated as long as it is the leader. - all replicas will reconcile to a state where either their log is empty or the record (including epoch) at their (endOffset - 1) is present in the leader log. In the situation that you mentioned, broker-103 will send a fetchRequest with offset=3 and lastFetchEpoch=1. Broker-102 (new leader) will send the records 3 & 4 to broker-103. On receiving the records, broker-103 will add the records (3 & 4) to it's local log, and increment it's Hw to match that of leader i.e. 3.