What is the Transactional Outbox Pattern? | Designing Event-Driven Microservices

[@ConfluentDevXTeam]

Wade here. I spent a lot of time trying to solve the dual-write problem in silly ways (such as writing the event first). Eventually, I settled on Event Sourcing as a good solution (which it is). But, while event sourcing is a great solution with a lot of advantages, it definitely has a learning curve. A few years back, when I discovered the Transactional Outbox pattern, it felt like a bit of a breath of fresh air. Here was a solution to the problem that was relatively simple to understand and implement. If you are building applications with a relational database model, you don't need to re-architect your entire system just to solve the dual-write problem. That put the transactional outbox high up on my list of solutions.

[@thestroke82]

Another great advantage is that it can be used almost anywhere, not only when architechting green field solutions. Consider, for instance, the shift from a traditional to an event-driven architecture in a legacy application fraught with complex and tipically messy business logic. In such cases TOP complements the strangler pattern exceptionally well

[@schallereqo]

Hi Wade, Thank you for the great video. This new series of videos is amazing. I am a customer of Confluent and use the Event Router product to implement the Outbox pattern. The issue is that this product feels incomplete compared to some of your other well-established connectors. We have made a few attempts to contact Confluent regarding fixing minor things and adding Protobuf support to this connector but with no luck. I'm glad you are aware of the Outbox pattern but wish you gave such a product more emphasis to help your customers make use of this solution.

[@ConfluentDevXTeam]

Wade here. I'm glad you are enjoying the videos. Unfortunately, I can't speak specifically to the issues you are having with Event Routers, but I will pass on the feedback.

[@mehrdadk.6816]

I've been using Outbox pattern, and it works nicely. I also used Kafka transactions and it works even better. Kafka transactions are available to link to database transactions.

[@ConfluentDevXTeam]

Wade here. The purpose of Kafka transactions isn't to link with database transactions. It is to ensure that if we write multiple events to Kafka, either they all get written, or none of them do. It solves the dual-write problem for the specific case where all of the writes are to Kafka. It makes no guarantees about writing to external databases. The problem is that there is no built-in way to commit a Kafka transaction and a database transaction in an atomic fashion. You have to commit them separately, one and then the other. As long as you do that, there is a chance that a failure leaves one committed and the other uncommitted, and now you have hit the dual-write problem. You might want to double-check that your logic isn't suffering from the dual-write problem. You can find more information in this video: kzbin.info/www/bejne/fKGviXZ4p5yomaM.

[@quanta9236]

Hi Wade, thank you for the great video. I have a question is what is the differences between CDC and outbox, I think they are quite similar with respect to know what was changed in a database

[@zhoulingyu]

Great series. I have some experience with streaming. I find this very easy to understand.

[@chengchen9032]

Hi wade, your video is very well-made! I have been implementing event-driven architecture for some years, the way I dealt with the dual problem has always been event-sourcing + listen-to-your self pattern. However, it turns out that whenever we deliver messages through kafka, there might be an unstable delay. In scenarios like user signup and change of information, it could be quite a problem since user might panic if they don't see the change being applied immediatly, so I feel like the outbox pattern could be a solution for this. The reason I intentionally avoided the outbox pattern is that I am worried about periodically scan the database for events and simutaneously modify them, i had some bad experience with situation like this which eventually led to dead-locks and the whole DB just shut down.

[@ConfluentDevXTeam]

Wade here. The outbox pattern doesn't eliminate the eventually consistent aspect of Event-Driven systems any more than event sourcing does. Your events still have to run through Kafka and that can still take some time. Having said that, when using the outbox pattern, you can potentially write more complex queries against the database which is being updated synchronously. That means you don't have to use a CQRS read model to access all of the data which can eliminate some of the Eventual Consistency. Of course, the write model for event sourcing is often strongly consistent as well. Used carefully, it can reduce the need to deal with eventual consistency. It all depends on how you design it.

[@chengchen9032]

​@@ConfluentDevXTeam Thanks for you detailed explaination wade. What I was trying to say is, in a situation where you need to immediately read the information you just wrote to the database, such as user login immediately after they signed up, if you send the information to the MQ first, user might not be able to log in successfully. This is why I think event sourcing might not be the best choice in this case. it seems like the outbox pattern+kafka connector would be the best choice(like you mention in the video). whenever you update your entities in database, you write the related events in eventstore(a table in your database), then just let the kafka reads CDC and publish events.

[@nithinnambiar]

I prefer to keep it simple, in the scenario where the application receives the message from the broker and processes it and then got to write to the db and kafka, the app wouldn't return till it successfully does both the writes. If the db commit is successful and kafka publishing fails the message would be processes again but it wouldn't write to the db since an entry already exists, it will then proceed with kafka publishing.

[@ConfluentDevXTeam]

Wade here. The simple case you describe is actually quite complex because it has a lot of assumptions built into it. You are assuming that the incoming message comes from Kafka. But what if it doesn't? You are also assuming that it is acceptable to retry the entire operation multiple times, but what if it isn't? You are also assuming that the destination for the messages is Kafka, but what if it isn't? The advantage to the Transactional Outbox pattern is that you can use it even in cases where Kafka isn't involved, or where multiple database write attempts might be problematic.

[@kevinding0218]

Thanks a lot for the explanation of the Outbox Pattern; I think I grasp the workflow, but I'm trying to get a better understanding about what benefit it brings. For benefit of outbox pattern, compare to if the original microservice updates the state in the database first and, only upon success, proceeds to publish the event with a retry mechanism. I sense that a benefit of using the Outbox Pattern might be 1 1) Un-blocking the original service while having to let it hanging there perform the retry if event produce failed. 2) Record the event in a different place so it won't be lost if original service goes down after only persisting the state in DB 3) Isolate the event produce with at-least-delivery manner while having to respect eventual-consistency is there anything missed by using the Outbox Pattern?

[@ConfluentDevXTeam]

Wade here. You suggest using a retry mechanism to publish the event. But how do you know what events need to be retried? You'd have to save them somewhere, like perhaps in the database, because if they are only in memory, they are prone to getting lost. You need to save them in a transactional fashion because otherwise, you encounter the dual-write problem. Essentially, you've now implemented the Transactional Outbox pattern while working on retry mechanism.

[@AmanNidhi]

My understanding is same as @kevin's. @ConfluentDevXTeam Do you agree with the 3 pointer he has mentioned?

[@JamesHarrisonHaribo]

This sounds great! My problem is I have microservices that use both Postgres and Mongo, depending on the kind of state being changed. Of course, rarely does it write to both on the same command, but can happen. So in those cases, I don't feel like I can rely on a transaction inside one database, or an outbox that lives in one of those databases. I need an external transaction, that's where it gets a little messier.

[@ConfluentDevXTeam]

Wade here. It's definitely a tricky problem. Obviously, I can't speak to your exact situation. There will be plenty of nuances that I don't understand. But in those kinds of situations, I start to question whether I have drawn my boundaries correctly. Maybe the data that lives in two separate databases should actually live in one database. However, assuming that the boundaries are drawn correctly, the next thing I start to look at is whether these things are truly transactional in nature. Is it that either one could fail and that needs to trigger the other one to fail? Or is it that one causes the other. If one causes the other, then a transactional outbox might be a good solution. You implement the outbox in the "cause" and then use that as an event to trigger the "effect". And failing all of that, you might have to start looking at alternative solutions such as the Saga Pattern.

[@paulcalinovici8808]

I am not really sure that I understand how the data is stored in the outbox table. Is it some kind of event store from CQRS pattern, where we store events in a serialized format(json as example)?

[@ConfluentDevXTeam]

Wade here. To some extent, it's up to you. A serialized record would certainly be a valid option. Alternatively, if it makes sense, you could break the data into a columnar format. It will depend on what type of data you need to put into that outbox table. If you have a single event type, with a limited number of fields then columns might make sense. But if you have multiple event types, or data with many complex fields, then serializing it might be simpler. The other factor to consider is that you probably don't really need to query this table in any complex way. In that regard, keeping things as columns might not matter all that much.

[@saikrishna8429]

How Delivery Process updates Outbox table after publishing event to kafka. I want to understand what if Delivery process fails to send event to kafka (As publishing event happens asynchronously) but deletes the event in Outbox table.

[@ConfluentDevXTeam]

Wade here. I'm not sure I fully understand the question. I think you are asking what happens if the process that is supposed to send the events to Kafka fails and deletes the event from the Outbox table. That shouldn't happen. The way this process should be built is that you publish to Kafka, and you only delete the event after you get a confirmation back from Kafka saying it was successfully received. If you don't get a confirmation, you don't delete the event and you instead try again.

[@gabrielpiazzalunga5540]

Hello, i currently have an implementation of the trabsactional outbox pattern. But I'm considering moving to an in-memory outbox (like mass transit) what do you think about it?

[@ConfluentDevXTeam]

Wade here. I hadn't heard of Mass Transit before, so I had to look it up. The biggest issue with a true "In Memory" solution is that if the memory is lost, so is the message. I found an article from Mass Transit that was attempting to explain how they handle such failures using an analogy of taking out the trash. I'll be honest, I found the article kind of vague and I didn't leave it feeling like they had a solid solution for this problem. That's not to say they don't. The article just didn't do a good job explaining it. So, without a deep enough knowledge of Mass Transit, all I can really say is this: If you decide to go that route, make sure they have a good solution for handling failures without losing your messages. If the entire system crashes for some reason, are your messages gone? Unless your message delivery doesn't need to be reliable, but in that case, you wouldn't need an outbox anyway.

[@gabrielpiazzalunga5540]

@@ConfluentDevXTeam hey wade thanks for replying. Yes, I was thinking the same. My proposed approach as an option (with some trade offs of course) is to have an in memory outbox, but still store the state and the events in a DB to have a consistency. I feel this approach is more optimistic than the transactional outbox. The main downside is that it relies on being always a consumer that as an output produces an event. This in memory pattern doesn't work for if the producer of the event is not a consumer of a previous one

[@mhetresachin717]

Hi Wade, Thank you for the informative video. I have a few questions: 1. Why is this pattern necessary when we already have CDC and DB connectors? 2. If I'm manually handling the Kafka publishing by reading data from the outbox table and publishing it to Kafka, how should I manage the scenario where the Kafka publish is successful but deleting the entry from the outbox table fails?

[@ConfluentDevXTeam]

Wade here. 1. You've made an assumption that you have CDC and DB connectors. What if you don't? Now, for the sake of argument, let's say you do. What is the CDC process emitting? In a traditional architecture, you don't save events to the database. You save database/domain objects. Your CDC connector could certainly emit every change to a particular table, but that's not actually the same thing as an event. The event itself could span many different records in the database and may contain information in a different format than how it is stored in the database. It might filter out information, or include extra information such as more detailed context about the change. Now, CDC + Outbox is a pretty handy combination. 2. If deleting the entry (or marking it complete) fails, then you have to retry the entire process. And yes, that means you will get a duplicate event. Duplicates are always a risk when you are dealing with at-least-once delivery guarantees (Note: the same risk exists if you use CDC).

[@dinhthaile8648]

Hi Wade. Thanks for the great series. I have a question. What if the event has been emitted, but the deletion in outbox table has been failed? Once again, we still want it works in a transactional fashion, and we can't ensure the consistency.

[@ConfluentDevXTeam]

Wade here. The only part of the Transactional Outbox pattern that is actually transactional is the recording of the event and the updating of the data. Those two parts are done as a transaction to guarantee that either both happen, or neither does. However, once that's done, the rest isn't transactional. If you emit the event then fail to record that it has been emitted (either by deleting, or with a flag of some kind), then you will end up emitting the event again. However, distributed systems deal with duplicate events all the time. When you consume an event from the topic, it's possible for you to fully process it, but then fail to send the update to indicate that it was fully processed. At that point, you are going to get the same event a second time. As a result, event-driven systems typically implement deduplication techniques, or they ensure that the events are idempotent so that receiving them a second time won't matter. The delivery guarantees for the transactional outbox pattern are at-least-once.

[@dinhthaile8648]

Thank you. That's clear for me

[@Pentatonic_Hardcore]

didn't understant will try to find another video, thanks.

[@deependrachansoliya106]

Hi Wade, Outbox pattern may cause performance issues for high volume?let's say system at peak hours can handle 16K QPS but after implementation of outbox, database performance degrade. What's better to use a JDBC connector or CDC like debezium in my case?

[@ConfluentDevXTeam]

Wade here. Yes, in some implementations, the Outbox pattern may cause issues at higher volumes. The example I give in the video involves a specific database I worked with in the past that had issues with Tombstones, but there definitely could be other issues. As for JDBC vs Debezium, either one is a valid option. What works for your specific case is a little outside the scope of what we can answer in a KZbin comment.

[@Andron4iKTV]

​@@ConfluentDevXTeam , as I understand it, implementing a method that uses, for example, debezium (any kafka connector) does not guarantee consistency with, for example, postgres (because it uses wal). If kafka connector and kafka is down wal can go ahead and miss some data . Do I understand this correctly?

[@ConfluentDevXTeam]

@@Andron4iKTV Depending on your choice of technologies, there definitely could be issues you need to watch out for. Best to keep an eye on the official documentation or drop into the community forums for those specific tools.

[@combatLaCarie]

I'm a bit confused. If my code wants to write to the db and then kafka it's already the case that I wouldn't write to Kafka if the db entry fails. ``` success = write_to_db_with_transaction() if success: write_to_kafka() ``` Maybe in this video we're assuming the db write is async?

[@ConfluentDevXTeam]

Wade here. No, we aren't assuming an async DB write. You might want to watch the video on the Dual-Write problem to understand the issue better. kzbin.info/www/bejne/fKGviXZ4p5yomaM

[@SamreenZeeshan-t1c]

I want to implement it

[@ConfluentDevXTeam]

Wade here. By all means. I'd love to hear about your experience if you do. Drop another comment when you have something up and running to let me know how it went.

[@asifanwar7942]

Would a suitable solution to dual write problem be - setting appropriate unique constraints on the DB then either creating or updating or doing nothing to a record ? E.g. Use if the record already exists on the DB as a way to de-duplicate when writing to the database

[@ConfluentDevXTeam]

Wade here. If I understand your question correctly, then this wouldn't be a solution to the dual write problem. Database constraints can certainly be used to deduplicate messages in the database and ensure it stays consistent. However, the dual-write problem isn't really about message duplication. It's about a lack of message consistency between disconnected systems. In that case, database constraints aren't going to provide any benefit when the second system is something like Apache Kafka, or an email server. Meanwhile, if you are only writing to the database, and not to a separate separate system, then a database transaction is the right solution. To solve the dual-write problem, you need to break apart the process. You need to separate it so that your database write, and your Apache Kafka write, are not happening in the same process. With the transactional outbox, that means writing first to the database, then having an async process that can retry the Kafka write until it succeeds.

[@MarcelBornancin]

Yes, if you can make the database operation idempotent or make it throw an error for duplicated requests (and just ignore the error), you can safely resend the command to your service until you receive a successful response from it. There's the possibility of sending the message to the service bus multiple times (requiring deduplication there), but both these problems (command request retries + at least once semantics) also exist within the outbox pattern. So it is also a viable solution

[@sohaagarwal7336]

What if write to outbox table fails?

[@ConfluentDevXTeam]

Wade here. If the write to the Outbox table fails, then the entire transaction fails. At that point, you are just dealing with normal failures of your commands. You still have to potentially deal with the failure, but you won't leave yourself with the inconsistency caused by the dual-write problem.

[@WagnerBianchi]

Yet, you have the data already delivered and consumed to Kafka, and you can start over from the point it has failed.

[@ConfluentDevXTeam]

@@WagnerBianchi Wade here. If the write the outbox table fails, then the data never gets to Kafka. The write to Kafka requires the initial write to the outbox table to be successful.

[@prakharsahu7145]

Thanks a lot.

[@ConfluentDevXTeam]

Wade here. You are most welcome. I'm glad you enjoyed it.

[@patronovski]

How different is KIP-892 with this?

[@ConfluentDevXTeam]

Hello, Wade here. The Transactional Outbox Pattern is a design pattern that can be used in a variety of situations, including situations where Apache Kafka is not involved. For example, rather than publishing an Event to Kafka, you could instead be sending an email. The Transactional Outbox pattern would still apply in that case. Meanwhile, as I understand it, KIP-892 is dealing with how Kafka Streams writes to local state stores such as RocksDB in a consistent fashion. It is very limited in scope and uses a different approach from the Transactional Outbox Pattern. The Transactional Outbox Pattern is a solution to the Dual Write problem which is a general distributed systems issue. Meanwhile, KIP-892 is a solution to a different, Kafka specific, issue. For more details on the dual write problem, check out this video: kzbin.info/www/bejne/fKGviXZ4p5yomaM

[@larryhall1772]

"Promo sm" 😒

[@ConfluentDevXTeam]

Wade here. I can't deny that I work for Confluent, and that this is solving a problem that might be encountered by our users. Having said that, the technique outlined in this video is applicable across a broad range of technologies. I'd be more than happy to hear that someone took this approach and applied it to any use case, regardless of whether it specifically touches a Confluent product or not.