Why did Redpanda rewrite Apache Kafka? (with Christina Lin)

[@thomasmizrahi]

This is the second podcast episode I watch from you, I'm 10 minutes in and I'm already FASCINATED

[@miscbits]

This was a great product to hear about. I currently use Kafka in my day to day work and am always looking around at the general ecosystem of the space. I really like the philosophy of Red Panda and my immediate reaction is I want this to succeed in the space.

[@abhinavsinghal5486]

Amazing talk by Christina. It sounds exactly my previous employment minus I never made it to world of Redpanda. Amazing talk. Keep it up!!

[@AFenderson]

Kafka is moving away from having a separate consensus protocol and bringing it internal? I thought KRaft was a separate cluster of quorum server nodes that make up a "quorum controller service". I understand that it can possibly run co-located on the same physical broker nodes but wouldn't that not be supported in production?

[@DeveloperVoices]

I believe it's much more tightly integrated than that - KRaft is very much a part of Kafka, and it runs in the same logical process as the broker. You can choose to run Kafka as "just" a broker, or as "just" a metadata-controller, or both at once. Either way, it's just configuration - you're not spinning up a separate service in the way you would Zookeeper. That said, yes, there are operational advantages to having some instances dedicated to brokering and other instances dedicated to quoruming: "The key disadvantage [of running in combined mode] is that the controller will be less isolated from the rest of the system. For example, it is not possible to roll or scale the controllers separately from the brokers in combined mode. Combined mode is not recommended in critical deployment environments." - kafka.apache.org/documentation/#kraft_config I read that as more an operational issue than a problem with combined mode per-se. 🙂

[@AFenderson]

@@DeveloperVoices is KRaft production ready? It wasn't back in sept at Current

[@DeveloperVoices]

Yes, the release notes say go for it. 🙂 It was deemed production ready for new clusters back in late 2022. Migrating existing clusters from Zookeeper was declared production ready with the release of 3.6, which happened on 10 Oct 2023: kafka.apache.org/blog#apache_kafka_360_release_announcement

[@andydataguy]

Wow, between Fluvio and Redpanda Kafka is getting rekt