Transactional Outbox Pattern | Microservices Architecture Pattern | System Design Primer

[@aniruddhkhera510]

Thank you for diving into the Txn Outbox pattern along with an example. This is really helpful. I was wondering do we really need to manage outbox tables? Why can't we simplify the design by implementing the binlog consumers of the actual table? Example: When we are performing CRUD operations on the ORDER table, we can setup the consumers of the binlog of this table which guarantees whenever there is UPSERT on the ORDER table we also publish an event to the subscribers.

[@na2599]

Yes we need to create separate table for that. Because why we are giving burden to the same table

[@mousumisaha4525]

You are just awesome. Please keep Posting this kind of videos.

[@JaNaMSoNi]

extremely valuable

[@TechPrimers]

Cheers buddy

[@TheEntium]

Amazing content brother.!

[@TechPrimers]

Glad you liked it

[@mikedqin]

Great update, do you have Orchestration Saga Example? Thanks.

[@neel3297]

Tx.beginTransaction Db.Read(select order_id from outbox_table where sts = PENDING) PublishToQueue() Db.write(update outbox_table set sts = COMPLETED where order_id = :id) Tx.commit() Is this how it will be? If yes, then here too we are dealing with 2 different systems i.e. queue and db within TX boundary

[@TechPrimers]

Yes Neel but it's not tied up with a API call to create a new orderId. In our case we were generating a new OrderId, persisting and publishing. We cannot have both of them in a single transaction.

[@saiaussie]

A bit confused here. Wont a simple transaction management + exception handling be sufficient here? If any of the function fails, the DB txn is rolled back? @Transactional myFunction() { try { persist(); publish(); } catch(Exception ex) { throw ex; } }

[@TechPrimers]

Look at the bigger picture here Sai. Here our priority is creating order. Publish that to the downstream is not tied up to a user creation and user should not be made to wait because of that. The code above works only when we consider both as one transaction. In our case, downstream should be notified only after order gets created, which means order data is committed. Isn't it?

[@ildar5184]

​@@TechPrimers It's a choice we can make, whichever is best for our case. Some frameworks like Spring Transaction allow several logically connected operations to be wrapped into a single transaction scope, including sending a message to another system. It's not really a crime to commit only after we've done all necessary operations in the code after DB update, considering that sending an event is a required operation in order for the whole system to function correctly, not a secondary one (in this case). Regarding orderId - we don't have to commit in order to get it, e.g. ORM framework can take the next id out of the id sequence instead of delegating that to the DB upon commit. If a language and the current framework don't allow opening and using transactional scopes like this, then there's really no choice and we have to use Transactional Outbox/CDC tools.

[@ildar5184]

@@TechPrimers Oh BTW, even if we use Spring Transaction, there's still a case when we have to resort to Outbox/CDC - if sending an event/message to another system is not the last operation in the list of operations in a transaction. E.g. if after sending an event we need to do another DB update, which might fail and result in rolling back the whole transaction, that intermediary event send won't be rolled back, since it's not a DB change. Maybe it's really a better option to just stick to Outbox, rather than always trying to be aware if we implemented sequence of operations correctly and no such situation is possible in our code.

[@manojBadam]

How can we scale the outbox table publishers? Let’s say we are inserting 100k records per minute. If I have one service reading it will take forever. If I have multiple services, all of them end up reading the same set of records and publishes multiple events for one DB commit?

[@TechPrimers]

Great question. It's a separate topic in itself. If you see in the video I mentioned about 2 approaches for the publishers. 1. Transaction log based publisher 2. Polling publisher. The 2nd option doesn't scale for 100k msgs. However using option 1, you can read the log n publish an event into a queue like kafka and process messages off of that. This way you don't have to read the data from the table again. You can read about that more in this article medium.com/trendyol-tech/transaction-log-tailing-with-debezium-part-1-aeb968d72220

[@manojBadam]

@@TechPrimers Thank you!!

[@ashishranjan4597]

Can you give your input on designing common configuration server for Microservices. I don't have to use Spring Cloud Config server.

[@sairanganath9293]

To make more resilient, Batch job running to validate xyz_outbox and the orginal table of there is any discrepancy it will be notified or collected then sync those two table which have and successfully created order id. Any thoughts?

[@na2599]

But what will happen if after queue try ro update the table status goes down..

[@Saravanan-lj9so]

Thanks.Will you do coding for this?

[@TechPrimers]

Yes, soon

[@manojBadam]

What is the recommendation for publishing multiple entities(let’s say order, orderLineItem entities) through outbox pattern? Have a separate outbox table for each entity or use one outbox table for all entities ?

[@TechPrimers]

Take a look at "The Scale Cube" video which i made few days ago. It will help you take a decision on scalability using 3 different patterns. Transactional Outbox is helpful in individual persistence + publishing part. Scalability is a different usecase altogether. If entities are split, obviously we should have different outbox. then there is no point in splitting the entities in the first place. However, there is no hard and fast rule :)

[@mikedqin]

In my opinion, you can have a single outbox table to store Order JSON object which contains OrderLineItems. The published Event in this case is a Domain Event that carries state transfer information.

[@mukarrampasha7729]

Instead of persisting order in the DB first, we can first publish an event and "listen to ourselves" by consuming the same event and inserting in the DB. What do you think?

[@TechPrimers]

thats not a clean design since we need to create an orderID first and how will maintain the order sequence. Also, what if DB is down? then the message will be present in the queue and the order would have never got created in DB and all consumers would have got the data. The requirement itself is publishing only after committing in DB, so its ruled out Mukarram