Point-to-Point vs Publish/Subscribe | Designing Event-Driven Microservices

[@ConfluentDevXTeam]

Wade here. My first experience with Publish/Subscribe was actually using a custom-built messaging platform. This predated Apache Kafka. There were other solutions available at the time, but they lacked some of the features we wanted (such as a persistent log). A few years later, I worked on a project that used Apache Kafka instead. I have to say...I realized at that moment how many mistakes we made in our custom system. Apache Kafka solved many of the problems we had in a much more elegant fashion. I've been a fan ever since. Since those early days, I don't think I have built a system without Publish/Subscribe as the backbone, and most of them have used Kafka.

[@Milandor]

Thanks for this material, from a colleague transitioning to a fully distributed system.

[@ConfluentDevXTeam]

Wade here. No problem. Glad you found it useful. Distributed systems aren't easy. There's a lot of gotchas hiding in them. But, if you can get past those, there's a ton of pretty awesome benefits.

[@randaldurayhim]

what if the kafka broker fails, isnt it a single point of failure ?

[@ConfluentDevXTeam]

Wade here. Yes and no. Kafka should be considered a critical part of your infrastructure, the same way that your database is. So yes, if Kafka fails, then you might have a problem, much like if your database fails, then you have a problem. Having said that, there are a few things to consider that make it not really a single point of failure. 1. Kafka shouldn't be running as a single node. You should have multiple instances running in a cluster so that if one fails, the others can take over. If you have enough brokers running, the entire cluster failing should be almost impossible. 2. Depending on your system design, even if Kafka fails, it may not be an issue. For example, if you are using tools such as Event Sourcing or the Transactional Outbox pattern to publish events to Kafka, then the events are probably stored in a separate database. In that case, if Kafka fails for some extreme reason, then you can always publish the events later, once it comes back online, so no data is lost. Meanwhile, as long as your system is designed around asynchronous events it should be able to tolerate a short delay while you get the Kafka cluster back online.

[@AdamSouquieres]

A must see ! Flink and Kafka Streams are two Epic processing framework that could implement this.

[@ConfluentDevXTeam]

Wade here. Glad to hear you enjoyed the video. And yes, I agree that Flink and Kafka Streams are good solutions for doing stream processing in these types of architectures.

[@walidmatta1]

I am planning to implement Cognite Data Fusion in a refinery where a real-time historian, MES system and ERP will be integrating with each other. Any idea if Cognite supports MQTT protocol

[@ConfluentDevXTeam]

Wade here. I would suggest reaching out to Cognite with your question. You could also see if they have a community forum or similar where you could post your question.

[@zenluiz]

Great video

[@ConfluentDevXTeam]

Wade here. I'm glad you enjoyed the video. Make sure to check out the other videos in the playlist and follow the channel for more like it.

[@cestlacroix]

these 90% of all these patterns are just a theories and philosophies, it won't apply for 90% of real-world cases;

[@ConfluentDevXTeam]

Wade here. These are definitely not just theories and philosophies. Point-to-Point and Publish/Subscribe are patterns that are being used in the vast majority of systems today. If you can find a major software system that is not using one, or both, of these patterns, I'd be quite surprised.