Kafka: A Modern Distributed System

QCon London International Software Development Conference returns on April 8-10, 2024. Level-up on 15 major software and leadership topics including; The Tech of FinTech, What's Next in GenAI and Large Language Models (LLMs), Performance Engineering, Architecture for the Age of AI, Innovations in Data Engineering and more.
Learn the emerging trends. Explore the use cases. Implement the best practices.
Register now: bit.ly/3rRgsJV
--------------------------------------------------------------------------------------------------------------------------------------
Video with transcript included: bit.ly/2ZAyNum
Tim Berglund covers Kafka's distributed system fundamentals: the role of the Controller, the mechanics of leader election, the role of Zookeeper today and in the future. He looks at how read and write consistency work, how they are tunable, and how recent innovations like exactly-once semantics and observer replicas work.
This presentation was recorded at QCon London 2020: bit.ly/2VfRldq
For more awesome presentations on innovator and early adopter topics check InfoQ’s selection of talks from conferences worldwide bit.ly/2tm9loz
#ApacheKafka #DistributedSystems #SoftwareArchitecture

[@adityasoni1207]

If Ted Lasso is giving a tech talk, you have to watch it for sure! Amazing! Thanks for sharing this! 10/10 Awesome Talk.

[@tone_deff]

This is how I imagine, Ted Lasso would give a tech talk. 10/10 Excellent Talk.

[@satvistayou]

Tim as always a great speaker.

[@abderrahimhaddadi4023]

Very interesting talk for beginners ! Thanks !

[@rupeshjha4717]

Loved the talk! Summarised all the concepts very well! Loved ISR high water mark lol 🤣

[@smitrp]

Great speaker and great talk.. I would love to see him give more talk about this topic.

[@xrvgamers1189]

Best Kafka talk!

[@julianxe]

Exceptional talker, good stuff

[@dibyaranjanmishra4272]

Excellent talk!

[@pengdu7751]

excellent talk

[@NealZhang]

good enough

[@rum81]

When a leader for a partition goes down and a new leader is elected. 1) In between that time what happens to read and writes for that partition . 2) what happens to messages above high watermark

[@magsteel9891]

If the leader is down client reads and writes fail. The client has no connection and must reconnect to an available broker. If the broker is up but is not the leader the client gets "not the leader" exceptions. The recovery action is to find the new leader and connect to it. The more current apis hide these details.

[@abhiitechie]

Mag has answered the 1st question, I will answer the 2nd. The messages above the high watermark will get lost. This may seem data-loss but the Producer is responsible to tune the durability of their messages during the write with more replica acks, so with leader + 1 replica ack the producer will have durability with at-most 1 node failure, with leader + 2 replica ack the producer will accommodate 2 node failures, with just leader ack there can be data-loss even when 1 node i.e the leader dies

[@satishmhetre5542]

41.25

[@aaronchowdhury2706]

Not impressed by this dude's speaking style. Less dry humor, more content please.