The Pattern You MUST Learn in .NET

[@bscamba]

I want to know what is your other approach to doing this!

[@AlexBroitman]

+1

[@ahupond]

The "listen to yourself pattern" is more scalable in my opinion

[@dand4485]

Another way, back in the day you could have another program you run and register a row change listener in SQL, you can receive an event any time a row in a table is added/change/delete. This registered call back could place the "outbox message" into the outbox table for example, and then i've see signalR used for pushing events to subscribers.

[@neotechfriend]

+1 I normally use cdc direct from DB transport events to message bus or queue, also good when you re happen to change the data directly on the DB you want miss the messages

[@ian_hammond_cooper]

@@ahupond Listen to yourself works, provided it does not cause timing issues with a consumer knowing about the message before the producer. If the Db supports it another trick is to listen to its transaction log, and write to your outbox from that. The issue there is that your code that knows how to create the outgoing message has to be able to interpret the outgoing message from the Db transaction. The reason why Outbox is so common is it doesn't bump into these issues.

[@smikkelbeert]

Hey Nick, great video as always! One thing was a bit unclear to me, which was the total message flow, so I did a little research. From the video it seems that it is only one pattern you are using and that both services still need access to the same database. But in fact, you are using two patterns: the outbox and inbox pattern. The message is first saved to the outbox in the database. Then behind the scenes, MassTransit picks it up and sends it on the message queue (and removes it from the outbox table). Finally, the message is received by your worker which will save it to the inbox table. Then the worker reads from the inbox and processes the messages. So the worker and API can be running as separate processes (on different nodes) and they can work with a different database. The worker only touches the inbox table and the API touches the outbox tables. Correct me if I'm wrong, cause I only did a quick 5 minute research to clarify for myself :)

[@foxred2380]

I have heard about this pattern twice for past 2 weekends on interviews, great topic

[@SayWhaaaaaaaaaaaaaaaaaaaaaaat]

yet another hyped nonsense as previuos ones....Fashion changes every few months..

[@sevansoft]

I'd love to see the "Nick" approach to this, you tease 😊

[@GlassScissors]

Hi Nick, thank you for sharing. Please give us some more insight on your experience dealing with the alternative method for the outbox pattern.

[@mitar1283]

"Hello everybody I'm naked" once you hear it you can't unhear it 😂

[@Kitulous]

what have you done to me

[@tedchirvasiu]

Hellovrybody

[@mrcarlpaton]

Clowns 😂

[@andreikashin]

You can't hear 'hello everybody' here

[@SchattenPV]

Yes, please also show your approach to this problem.

[@Great_Critic]

Hey Nick, Could you please tell us your favorite way of solving this problem, And also tell us about how we can guarantee message is handled only once in the consumer? Also wondering how to preserve correct order of the messages.😊

[@PeterBojanczyk]

Excellent video as always. I'm interested in hearing about your other approach that you mentioned at the end of the video. Cheers

[@adriaandavel9540]

Atomic vs Self-healing is always a fun conversation. I've had good success with self-healing designs where atomic was not feasible. Could you do a video on self-healing design patterns?

[@Twyzz91]

A common issue with the polling worker is that if you scale your app you would then have multiple workers processing and publishing the same messages. That's why people would usually process the outbox before acknowledging the original message. If the publish fail then that original message would retry, you would see that an outbox already exists and then only process the outbox instead of the entire message.

[@anvogel99]

You can deal with it by adding a field like "ProcessingStartedAt" which will prevent messages from being processed by other workers until the one who picked them will complete processing and delete the messages. Having a date instead of bool will allow you to re-process messages which became stuck because of a crushed worker.

[@dsvechnikov]

Yo dawg, I heard you like queues so we added a queue for adding messages to a queue so you can add message to a queue while adding message to a queue!

[@MilYanXo]

literally. I am looking and thinking, wth? The whole point of MQ was its simplicity of use, even replacing a MQ engine is not such a big deal that it reauires this massive MassTransit abstraction on top of it.

[@Astral100]

@@MilYanXo Yeah, the problems you run into using MassTransit for a while are not worth it in my opinion.

[@TheWoodad]

Love the abstraction over message queuing that MT offers. And this feature you've demonstrated is definitely something I'm keeping on the radar. Last project that had something similar was using MSSQL ServiceBroker, but that was ages ago and that technology has become a long forgotten relic. Very cool stuff, Nick.

[@888emce]

There is no need to configure the consumer side. Doing so only makes this example confusing (as people are already asking if it uses a database instead of a queue). The consumer can be also configured to use an outbox pattern (which actually combines inbox and outbox), but it's not obligatory.

[@mrcarlpaton]

Glad im not the only one using paint to explian things! Brother id love to see a video on the other approach you mentioned to solve for atomic operations over a database and queue!

[@dk95ns]

One question regarding the worker. Considering the behavior we're trying to solve is usually part of a microservice architecture, or some fully separated systems, doesn't the fact that the worker needs access to the outbox kind of defeat the purpose? What I mean is that if I have access to the outbox, I have access to the DB. If I have access to the DB I can create a trigger for example in the DB, or a scheduled task that gets the info and handles it. I don't see a purpose of the queue in these cases, because I can just create an artificial one locally.

[@Sirozha1337]

I think the idea is that you create another service that has access to both DB and MessageBroker. It reads data from the outbox and passes it into the queue. Other services consuming from the queue don't need connection to DB since they can use message acknowledgement to avoid duplicate processing of the message.

[@ProfessorCodemunkie]

I think the example in the video was a bit confusing but that's not how it works. MassTransit automatically configures a Background Service in the API that polls the Outbox and sends the messages. The consumer application does not change, it still reads from the queue.

[@billy65bob]

If you're using the database as the message queue/broker, you should be aware of SQL Hints, particularly UPDLOCK, HOLDLOCK, and READPAST. UPDLOCK will make it take the necessary exclusive locks to modify the row, HOLDLOCK will make it hold the locks until the end of the transaction (e.g. shared locks tend to get released otherwise), READPAST makes it skip rows that are already locked. Combine this with a SELECT TOP 1 read, and you've got yourself an easy and fairly robust (though not idempotent) queue. Can run as many competing readers on it as you like (to diminishing returns), and they won't step on each other's toes. That all said, I'm sure MassTransit has stuff built in to deal with any idempotency concerns.

[@BTimelessC]

I don't think the worker has access to the DB in the example

[@piotrkowalski3460]

consumer reads messages from the queue (not db). It can use inbox table for duplicate detection feature and it can have it's own database.

[@1235663]

Interesting to hear about pettern for reverse situation. When you read from queue, and something happens with database, or the next queue sending

[@BrunoDPO]

Thanks once more, Nick! Very interesting pattern! I want to know about the other approach using native cloud products!

[@danielmischiu511]

a very scientific tool...opens up paint. Great content man. Many tanks

[@ian_hammond_cooper]

it is worth noting the trade-off here. By assuming you are on EF Core, Mass Transit can avoid you needing to inform it about the "unit of work" that the Outbox needs to participate in. For Brighter we take a different view, and support multiple approaches to your DB: Dapper, EF Core, DynamoDb etc. but you need to pass us a unit of work. We will create this for you, via config, so that you can just use DI to inject it into your code. You then use the unit of work to create the transaction for your Db and the publish to the queue. As ever, trade-offs, but it removes the hard dependency on EF Core for us.

[@chrismantonuk]

Yes I’d really like to know your preferred solution to this problem. I’m not a big fan of this approach, it smells a bit “automagic” - i.e, it isn’t obvious when looking at the service code that the queue message isn’t published until the save changes transaction. I can imagine picking up this codebase and scratching my head as to why the message isn’t in the queue when it looks like it should be.

[@tanglesites]

Awesome video on Information Systems. I think this was a great introduction to the development side of Information Systems. I think you touched on it, but did not say anything specifically, but what about Event Sourcing. And I vote yes on a video detailing your alternative method of solving this problem.

[@PauloWirth]

A good follow-up for this would be a video detailing the Inbox Pattern, for receiving messages.

[@BrunoBranco-kd3ir]

Hey Nick, I would like to congratulate you on another excellent video. I have a question regarding to the need to add EF Core packages to Consumer. The way I was assuming it work is following: - publish push the event to the outbox table - a background process (on the publish side) is responsible for scanning the table, pushing de event to the queue, and the update de status of the event row as processed - the consumer still consumes messages from the queue with no changes needed What am I missing here? 😅

[@ZuvielDrama]

This is nice. I solve things like this just with a Task / Thread Process in the background and a messagequeue as a command/message store to process dependend tasks like this. It is simple and easy and only as many features in this construct that i need for my problem that i have to solve. But i really love this content, because it validates my way of doing things, if big companies walk same ways in a bigger radius of problem solving.

[@julienraillard3567]

Love you're "very scientific tool" as you mentionned :D

[@FSEAirboss]

I'd be interested in know what you have actually implemented and used in practice for you previous requirements.

[@foonlam7134]

We have a legacy system that implements something like the outbox system. Some action will write a message to our custom outbox and some separate background app will check that outbox and send the message. But it doesn't have retries and no idempotency.

[@nickniebling]

Love videos like this - explain useful pattern and show a good way to implement it using OSS 🙌

[@joshuaodugbemi5864]

Instead of using entity framework in this case, I would manage the db in my code. I would start a transaction, stage the update, send message to the queue. If successful, I commit the transaction, if not, I rollback.

[@BrendonParker]

I think I like the simplistic of save changes the fire the message. Having a relay poll the database and then enqueue the message seems complicated and oddly familiar to using the database as a queue. Maybe my problems aren’t enterprise-enough to warrant this.

[@Adiu72]

But in enterprise scale of micro services each service has its own database and common queue. So it is not the same as DB will be pulled only by one service just to publish messages to the queue while in your solution all services would need to access single DB and pull it.

[@jeremy3264]

In this approach is there a behind the scenes polling of the outbox? If so, that could be costly from a cloud perspective. I've seen patterns that inert into a db, then add a service bus message with a reference to the record in the db, then ack to the client. You need to have a cleanup process if there is a failure between db and service bus, but there would be much less db polling. Not quite as elegant as the Listen To Yourself pattern, but it works with large messages that might not be suitable for service bus

[@MeMeNaMi]

Please create a video for your preference for dealing with this issue, thanks

[@Eatyaatbreakfast]

I've been seeing this pattern a lot lately. Many teams implementing it at work. What is the other approach that you suggest (besides two-phase commit, obviously)?

[@ismailm123]

Another fantastic video, definately interested in the other option.

[@darthrainbows]

How does duplicate message detection work, particularly if the subscriber has many instances and does not have access to the DB?

[@petermihalik9453]

The subscriber must have access to the database

[@binoypatel09]

So far I used Azure Functions but this looks interesting, I also wanted to know what other approach do you suggest, thanks for the great turor

[@rakeshkumarreddymudda]

12:32 I am confused about how id created and is passed to outbox table which was done before save changes. Does the masstransist handles to update id and send to outbox table?

[@ealeykin]

In EF you can implement interceptor for SaveChanges operation - so this is how MassTransit, most probably, does that

[@rakeshkumarreddymudda]

@@ealeykin got it thanks

[@georgekopadze791]

Hey Nick! Great video, thank you. I have a question though. What if I have 2 micro service with its own database and want to implement this pattern, how do we do it? In this video, you have single database and publisher/subscriber pointing to the same database. Thank you

[@RaitLaast-Laas]

Nice! Hey what's that high-tech uml tool you using?

[@peterk4694]

I use outbox all the time. What's your preferred way to do it? We use masstransit, SQS, RMQ & rdb but not the built-in masstransit outbox.

[@StefanoTempesta]

Would you send the message to the queue AFTER saving the record to the database? What if you get a DB exception and the message was already propagated?

[@RohitMoni0]

Hey Nick, you mention the sponsored AWS course is free from the link in the description, but there is no link? And the course has a price tag on the website? Interested, thanks!

[@hobbit125]

TL;DR - Have your services store messages in a database table and then have separate service that sends them from there to your message broker.

[@DJHightower77]

I like to let you know that i am interested to learn how you solve that problem without a outbox. Is it the build-in transaction-support in some messaging frameworks (i think azure service bus supports that).

[@dbohdan]

I am so excited to read this comment

[@marcobaccaro]

The outbox pattern is applicable for some use cases, in your example it is not the case though. You added extra complexity to your design and pooling in the database. A DTC with retries would simpler, atomic and efficient.

[@mauritsvdp406]

Nick, with the pattern you ensure the message is published exactly once, but how can you make sure the message is also consumed exactly once?

[@ladrillorojo4996]

It includes a "lock" column which should be enough for queries to ignore it if it's not filled with zeroes. I didn't see it but it must also have a "processed" column or something too.

[@Revocdeb]

The message queue should support locking so a message can only be consumed by a single subscriber.

[@Great_Critic]

He didn't explain this. From what I read, one solution is to save message id's to be able to identify already handled message. If already handled - then skip.

[@alfonsdeda8912]

Hi Nick, very nice video! One question: In api methods that work with related entities(aggregates), when for example I want to update all data in related in main and related entity, is a good idea to pass the related entity id or should get directly with the main entity id all the childs and update then? Sorry, maybe the explanation is not the best... The question is if should I send directly to the api related entity's IDs or should I obtain them from the main aggregate? Thanks in advance.

[@paulkeating9259]

Defo want to hear the other solution, currently building this very thing at work and we lol'd becuase your released the vid as soon as our meeting about it was done

[@Th3MoL3]

I'm really spooked because this literally happened just after a meeting I had which spoke about the outbox pattern

[@A-JesusWlkr]

Is there already a video out which is Nick's favourite approach to this ?

[@nickchapsas]

I'm working on it

[@A-JesusWlkr]

@@nickchapsas Thanks. Blessings to you in your work!

[@abdulrafeyartist5025]

how does mass transit know that the event shall be tied to customers table ? what will happen when there are multiple queue calls ? perhaps storing data into multiple tables etc.

[@jonasbarka]

I want to know your favorite way of solving the problem!

[@markmahowald7866]

Pub sub is the best. I demo this in paint for our new hires.

[@as10046]

This video is confusing since is not showing how to implement the design pattern, its showing how to use a black box that already has the design pattern.

[@nrmagalhaes]

Idk if I miss something about DI injection magics but who guarantees that Publish will work after inserting to database? Isn't the same problem as the beginning?

[@lukegordon4734]

I believe they're both added to the changetracker which EF will ensure happens atomically (they either both are created or neither is). So it's not one then the other. It's either both or neither.

[@nickchapsas]

The Publish method is hijacked behind the scenes and it turns into a database operation that is part of the transaction

[@nrmagalhaes]

Thanks! @@nickchapsas

[@GiovanniOrlandoi7]

Thanks for the video!

[@Locustfiretree]

At 16:37, you use AddAsync() to prepare adding a customer to a database. Why? Normally you would only need to use the Add() method unless you have some reason to use AddAsync()? The documentation says as much (e.g. SequenceHiLo).

[@MilYanXo]

Its so funny to see the resurrection of MSMQ spinoffs now that cloud providers decided to sell (err, rent) them. I've been there 3000 years ago when MSMQ with DTS was used for all this and we didn't call it outbox pattern :D

[@MeinErsterKanal]

Please let me know how you are dealing with distributed transactions.

[@PedroPabloCalvoMorcillo]

This is a very informative video. Kudos!

[@romanpelikh1862]

Awesome! Thank you 😊

[@abeljr88]

use function app with cosmos db trigger, cosmos db is your outbox

[@az6876]

But what happens if there is no consumer? Is there a timeout and api returns error?

[@DNI404]

I'm also curious about the other approach

[@EivindGussiasLkseth]

Yes, I wan't to hear your preferred way of dealing with this problem.

[@timseguine2]

7:30 When you said "Don't log like this", to me it was slightly unclear what you were referring to. Do you mean we shouldn't have log lines that change the semantic behavior of a function? Or do you mean abusing the message queue to log something that the creator of the message should have logged? It's hard to tell exactly what you meant, and I can think of multiple reasons why it might not be a great idea.

[@888emce]

He was referring to the concept called "Structured Logging"

[@NicolaiSkovvart]

He meant don't use the object .ToString() as a log message template. Why he didn't just do it "right" is beyond me though, would have taken less time.

[@tuberklz]

actually i need less auto-magic way to implement this pattern. i have a legacy app that i cannot add dependencies, and uses oracle database. so maybe i would just manually write to manually configured tables in oracle, and on the client side, expect to get a similar result. this is about payment system that i need to be sure more than one independent services are processed the request successfully. and, i should not be a bother to writes of main system. any suggestions on that would be great. if not i will try my luck so, maybe i am late and lucky for already you have publised second video you have metioned. if not, please do consider

[@RussellGuthrie]

I too would like to know the alternative approach, especially if it's cloud centric.

[@Mec2005]

@Nick, is it possible to use with LiteDB?

[@Iingvarka]

The only issue I can see consumer should have access to publisher database or outbox configuration should be enough on publisher side only?

[@nickchapsas]

It doesn't need to have access. It depends on your setup

[@ThalesAlves-o6o]

Anyone else not getting the email when registering in the dometrain course?

[@tomthunderforest1681]

I want to know what is your other approach to doing this!

[@claucometa]

Why use await to publish to the queue?

[@ShaneMontgomery32114]

Could someone clarify why there is a service that reads from the outbox table and pushes the message to a queue to be read and dealt with as opposed to just resding from the outbox table and dealing with the messages at that point in time? IE read and process directly from the outbox table and cut out the message queue.

[@Adiu72]

Other microservices probably have its own DBs. Still if you have a single DB for all then you need to pull it even more often because each service will be pulling it. And you loose built in support for topics etc for queues.

[@Nouriishly]

But does the worker has to connects to the same database?

[@nickchapsas]

The worker doesn't need to be aware of the outbox. There are multiple types of setup you can have and one of them is for the worker to be agnostic of the db stuff

[@Nouriishly]

thanks for replying@@nickchapsas, I really liked it the concept and started exploring it in a demo project.

[@sikor02]

Too bad that there is no easy way to stop listening for messages from the queue. Or maybe there is? What if my app received some message that starts up some heavy process and I don't want to listen on new messages like this from the queue until I'm ready?

[@bluecup25]

Thanks bruda

[@ИванИванов-я5э9к]

7:25 what is the right way to do the logging?

[@xtazyxxx3487]

Using templating to achieve structured logging

[@Michel000000001]

Aren't you just shifting the problem? Initially, you had the issue that you couldn't guarantee that both the database transaction was successful and that the message was placed on the queue. You have now solved that problem. However, there is now another, additional process that also needs to read a message from the database, place it on the queue, and then mark in the database that the message has been placed on the queue. Doesn't that process have the same problem? That it might place the message on the queue but fail to mark the message in the database? Isn't it still an issue that you want to perform both a database transaction and a queue transaction?

[@dbohdan]

Thanks for covered topic, but for me seems like overkill. Interesting how everyone deal with this problem, from my side I don't see any problems with creating transaction, save changes, publish message , complete transaction. Interesting about other opinions.

[@TheMikernet]

What happens if after you publish the message but before you complete the transaction, any of the following happens: a network outage, app pool recycles, hardware failure occurs, or web process is shut down?

[@dbohdan]

​@@TheMikernet Got it, thanks,I understand the problem. To implement all this staff you need background worker which read DB (possible bottleneck) and push messages to queue. And background works can have the same problem (network, hardware) so we have to add some logic to prevent problem with background worker and logic to prevent duplication of messages in queue. Maybe it's better avoid using the outbox pattern and just add logic to subscriber of this event where we check that operation completed. Or if it is so critical to use Service Bus transaction. Thanks why I think that this pattern is overkill.

[@deddykosasih387]

Why does the working need to use DBContext? didn't it just listen to sqs and process?

[@thiagomatu]

It seems it deleted "physically" the event after processing. Can we soft delete instead?

[@jancg1972]

Good video. I don't like the 'behind the scenes' aspect. Especially the invisibility of what is actually happening.

[@levimatheri7682]

How does this compare with Sagas? Or is it an implementation of Sagas?

[@JH-qe3fu]

Saga involves taking counter measure if the message has not been delivered is not it?

[@ThomasKardaridis]

Hey Nick, I wan't to hear your preferred way of dealing with this problem.

[@avineshwar]

The videos where you are introducing us to libraries and all that is great. Outside of that, in my opinion, there is a consistent lack of disconnect between your intent and what you are really trying to accomplish through pattern-type videos. You need explain a lot more technical background via system design and diagrams without even touching the code. Of course, this is not to a suggestion to make hour long videos. At least share system examples you have worked on and why in the hindsight you think you were right or wrong. I do not really care you talking about food delivery service or ride-hailing service. That would be so because you have not worked on them. Any heavily used enterprise system would pose massive technical challenges, so there should be enough to discuss on that itself. If only seeing through the complexities of the system would be easier.

[@nothingisreal6345]

When you need to write the message to the DB - why not simply pull the DB via an IP or push some notification to a consumer of the "mesage" service that sometging int he DB has changed

[@JKd3vLD]

How would we configure something like this if we were trying to queue multiple transactions between multiple microservices, each one independent of each other... I wonder... 🤔

[@2SHARP4UIQ150]

No... I Will not mix messaging with my application data layer.

It's a resiliency layer. The messaging is still handled by a messaging service.

[@2SHARP4UIQ150]

​@ I am not concerned about how or what handles the service but the implementation. If you watch the video, you will see that the implementation becomes part of your data access code. Nick briefly makes an excellent and concise explanation using a diagram. I prefer a separate service to handle messaging; it is not part of my application and deploys separately.

@@2SHARP4UIQ150 I see your point but there's probably not a good way to create a transaction that would affect both the data and the messaging service. The goal is to ensure that either both or neither of the writes happen. So yes, messages are temporarily (probably for less than a second) saved in the database along with the other data. It's a trade off but makes sense to me.

[@pureevil379]

I write to a table have a status column and my service reads from it :/

[@davidbrinton1526]

It would be great to hear how else you deal with this.

[@harryprince9556]

I wanna know your preferred approach very much

[@MeMeNaMi]

Is there a way to do this in DynamoDB? Does EF support DynamoDB? As I saw there's still no support for dotnet8 in AWS Lambda! :(

[@fredericguertin3364]

Theres something really close to EF for dynamo that I used its called poco dynamo, and you could use cdc + lambdas in dynamo instead of this.

[@ian_hammond_cooper]

Brighter supports an Outbox for DynamoDb

[@ealeykin]

What about Debbezium or CDC ?

[@thanzeeljalaldeen]

whats ur approach?

[@qweqwe123qewweqwe]

Why there are three tables, and what is inbox?

[@jesussantos2934]

13:52 Somebody knows what console it is?

[@sebastianh7712]

Why does the Worker also need the Outbox pattern and Entity Framework as a dependency? Please watch 15min.

[@nanvlad]

Did I get it right that it uses db as a synchronization source instead of messaging queue? Can it be done without using db? As I know message queues have their own persistence layer, so the messages can be saved to a file and then be restored in case of failure

[@dy0mber847]

What if request modifies data in your database which other parts(microservices) of your system are interested in?

[@local9]

@@dy0mber847 Not only that, but I'd rather write to a database instead of a file... those are locks and queues I don't want to get into.

[@T___Brown]

File scales better than db. In microservices you have to create a new table for every db to make this happen or you get into dtc. And scaling reading from 1 table transactionally is very slow. I was originally doing this via triggers and the inserts were fast but i couldnt get my transactional reader to read/queue/commit fast enough. Instead, when queue write fails i write to disk with a prefix of the app name so that later i can rename it then read it and resend it to the queue. This is much more consistent during networking or rmq outages.

[@mrplatypus0]

@Nick Chapsas - I would like to know your approach, please.