Wade here. My first experience with Publish/Subscribe was actually using a custom-built messaging platform. This predated Apache Kafka. There were other solutions available at the time, but they lacked some of the features we wanted (such as a persistent log). A few years later, I worked on a project that used Apache Kafka instead. I have to say...I realized at that moment how many mistakes we made in our custom system. Apache Kafka solved many of the problems we had in a much more elegant fashion. I've been a fan ever since. Since those early days, I don't think I have built a system without Publish/Subscribe as the backbone, and most of them have used Kafka.
@Milandor4 ай бұрын
Thanks for this material, from a colleague transitioning to a fully distributed system.
@ConfluentDevXTeam4 ай бұрын
Wade here. No problem. Glad you found it useful. Distributed systems aren't easy. There's a lot of gotchas hiding in them. But, if you can get past those, there's a ton of pretty awesome benefits.
@randaldurayhim4 ай бұрын
what if the kafka broker fails, isnt it a single point of failure ?
@ConfluentDevXTeam4 ай бұрын
Wade here. Yes and no. Kafka should be considered a critical part of your infrastructure, the same way that your database is. So yes, if Kafka fails, then you might have a problem, much like if your database fails, then you have a problem. Having said that, there are a few things to consider that make it not really a single point of failure. 1. Kafka shouldn't be running as a single node. You should have multiple instances running in a cluster so that if one fails, the others can take over. If you have enough brokers running, the entire cluster failing should be almost impossible. 2. Depending on your system design, even if Kafka fails, it may not be an issue. For example, if you are using tools such as Event Sourcing or the Transactional Outbox pattern to publish events to Kafka, then the events are probably stored in a separate database. In that case, if Kafka fails for some extreme reason, then you can always publish the events later, once it comes back online, so no data is lost. Meanwhile, as long as your system is designed around asynchronous events it should be able to tolerate a short delay while you get the Kafka cluster back online.
@AdamSouquieres7 ай бұрын
A must see ! Flink and Kafka Streams are two Epic processing framework that could implement this.
@ConfluentDevXTeam7 ай бұрын
Wade here. Glad to hear you enjoyed the video. And yes, I agree that Flink and Kafka Streams are good solutions for doing stream processing in these types of architectures.
@walidmatta17 ай бұрын
I am planning to implement Cognite Data Fusion in a refinery where a real-time historian, MES system and ERP will be integrating with each other. Any idea if Cognite supports MQTT protocol
@ConfluentDevXTeam7 ай бұрын
Wade here. I would suggest reaching out to Cognite with your question. You could also see if they have a community forum or similar where you could post your question.
@zenluiz6 ай бұрын
Great video
@ConfluentDevXTeam6 ай бұрын
Wade here. I'm glad you enjoyed the video. Make sure to check out the other videos in the playlist and follow the channel for more like it.
@cestlacroix4 ай бұрын
these 90% of all these patterns are just a theories and philosophies, it won't apply for 90% of real-world cases;
@ConfluentDevXTeam4 ай бұрын
Wade here. These are definitely not just theories and philosophies. Point-to-Point and Publish/Subscribe are patterns that are being used in the vast majority of systems today. If you can find a major software system that is not using one, or both, of these patterns, I'd be quite surprised.