Jun Rao, Confluent - Kafka Controller: A Deep Dive | Bay Area Apache Kafka® Meetup

Bay Area Apache Kafka® Meetup hosted at Confluent HQ (confluent.io) in Palo Alto on April 29th from 6:00pm. Jun Rao, cofounder of Confluent, a company that provides a streaming data platform on top of Apache Kafka. In his talk, Jun discusses how the controller is the brain of Apache Kafka. A big part of what the controller does is to maintain the consistency of the replicas and determine which replica can be used to serve the clients, especially during individual broker failure. In this talk Jun will outline the main data flow in the controller-in particular, when a broker fails, how the controller automatically promotes another replica as the leader to serve the clients, and when a broker is started, how the controller resumes the replication pipeline in the restarted broker. Jun will then describe recent improvements to the controller that allow it to handle certain edge cases correctly and increase its performance, which allows for more partitions in a Kafka cluster. Previously, Jun was a senior staff engineer at LinkedIn, where he led the development of Kafka, and a researcher at IBM's Almaden research data center, where he conducted research on database and distributed systems. Jun is the PMC chair of Apache Kafka and a committer of Apache Cassandra.
ABOUT CONFLUENT
Confluent, founded by the creators of Apache Kafka®, enables organizations to harness business value of live data. The Confluent Platform manages the barrage of stream data and makes it available throughout an organization. It provides various industries, from retail, logistics and manufacturing, to financial services and online social networking, a scalable, unified, real-time data pipeline that enables applications ranging from large volume data integration to big data analysis with Hadoop to real-time stream processing. To learn more, please visit confluent.io

[@AchillesGeng]

Very helpful video, zombie requests has bothered me for a while. I have determined to upgrade to 2.2 in these days.

[@vipulgoyal2370]

Just wondering how much partition metadata could be in terms of size, that controller node needs to move?

[@raghavops146]

Thanks for the excellent video. Is it possible to move the controller to broker of choice