The Pattern You MUST Learn in .NET

[@bscamba]

I want to know what is your other approach to doing this!

[@AlexBroitman]

+1

[@ahupond]

The "listen to yourself pattern" is more scalable in my opinion

[@dand4485]

Another way, back in the day you could have another program you run and register a row change listener in SQL, you can receive an event any time a row in a table is added/change/delete. This registered call back could place the "outbox message" into the outbox table for example, and then i've see signalR used for pushing events to subscribers.

[@neotechfriend]

+1 I normally use cdc direct from DB transport events to message bus or queue, also good when you re happen to change the data directly on the DB you want miss the messages

[@ian_hammond_cooper]

@@ahupond Listen to yourself works, provided it does not cause timing issues with a consumer knowing about the message before the producer. If the Db supports it another trick is to listen to its transaction log, and write to your outbox from that. The issue there is that your code that knows how to create the outgoing message has to be able to interpret the outgoing message from the Db transaction. The reason why Outbox is so common is it doesn't bump into these issues.

[@smikkelbeert]

Hey Nick, great video as always! One thing was a bit unclear to me, which was the total message flow, so I did a little research. From the video it seems that it is only one pattern you are using and that both services still need access to the same database. But in fact, you are using two patterns: the outbox and inbox pattern. The message is first saved to the outbox in the database. Then behind the scenes, MassTransit picks it up and sends it on the message queue (and removes it from the outbox table). Finally, the message is received by your worker which will save it to the inbox table. Then the worker reads from the inbox and processes the messages. So the worker and API can be running as separate processes (on different nodes) and they can work with a different database. The worker only touches the inbox table and the API touches the outbox tables. Correct me if I'm wrong, cause I only did a quick 5 minute research to clarify for myself :)

[@mitar1283]

"Hello everybody I'm naked" once you hear it you can't unhear it 😂

[@Kitulous]

what have you done to me

[@tedchirvasiu]

Hellovrybody

[@mrcarlpaton]

Clowns 😂

[@andreikashin]

You can't hear 'hello everybody' here

[@foxred2380]

I have heard about this pattern twice for past 2 weekends on interviews, great topic

[@SayWhaaaaaaaaaaaaaaaaaaaaaaat]

yet another hyped nonsense as previuos ones....Fashion changes every few months..

[@sevansoft]

I'd love to see the "Nick" approach to this, you tease 😊

[@GlassScissors]

Hi Nick, thank you for sharing. Please give us some more insight on your experience dealing with the alternative method for the outbox pattern.

[@dk95ns]

One question regarding the worker. Considering the behavior we're trying to solve is usually part of a microservice architecture, or some fully separated systems, doesn't the fact that the worker needs access to the outbox kind of defeat the purpose? What I mean is that if I have access to the outbox, I have access to the DB. If I have access to the DB I can create a trigger for example in the DB, or a scheduled task that gets the info and handles it. I don't see a purpose of the queue in these cases, because I can just create an artificial one locally.

[@Sirozha1337]

I think the idea is that you create another service that has access to both DB and MessageBroker. It reads data from the outbox and passes it into the queue. Other services consuming from the queue don't need connection to DB since they can use message acknowledgement to avoid duplicate processing of the message.

[@ProfessorCodemunkie]

I think the example in the video was a bit confusing but that's not how it works. MassTransit automatically configures a Background Service in the API that polls the Outbox and sends the messages. The consumer application does not change, it still reads from the queue.

[@billy65bob]

If you're using the database as the message queue/broker, you should be aware of SQL Hints, particularly UPDLOCK, HOLDLOCK, and READPAST. UPDLOCK will make it take the necessary exclusive locks to modify the row, HOLDLOCK will make it hold the locks until the end of the transaction (e.g. shared locks tend to get released otherwise), READPAST makes it skip rows that are already locked. Combine this with a SELECT TOP 1 read, and you've got yourself an easy and fairly robust (though not idempotent) queue. Can run as many competing readers on it as you like (to diminishing returns), and they won't step on each other's toes. That all said, I'm sure MassTransit has stuff built in to deal with any idempotency concerns.

[@BTimelessC]

I don't think the worker has access to the DB in the example

[@piotrkowalski3460]

consumer reads messages from the queue (not db). It can use inbox table for duplicate detection feature and it can have it's own database.

[@PeterBojanczyk]

Excellent video as always. I'm interested in hearing about your other approach that you mentioned at the end of the video. Cheers

[@wimlotz713]

Again another wonderful video, I watched a few before I watched yours about the outbox but yours explained it very well, thanks!!!!

[@SchattenPV]

Yes, please also show your approach to this problem.

[@adriaandavel9540]

Atomic vs Self-healing is always a fun conversation. I've had good success with self-healing designs where atomic was not feasible. Could you do a video on self-healing design patterns?

[@Great_Critic]

Hey Nick, Could you please tell us your favorite way of solving this problem, And also tell us about how we can guarantee message is handled only once in the consumer? Also wondering how to preserve correct order of the messages.😊

[@Twyzz91]

A common issue with the polling worker is that if you scale your app you would then have multiple workers processing and publishing the same messages. That's why people would usually process the outbox before acknowledging the original message. If the publish fail then that original message would retry, you would see that an outbox already exists and then only process the outbox instead of the entire message.

[@anvogel99]

You can deal with it by adding a field like "ProcessingStartedAt" which will prevent messages from being processed by other workers until the one who picked them will complete processing and delete the messages. Having a date instead of bool will allow you to re-process messages which became stuck because of a crushed worker.

[@TheWoodad]

Love the abstraction over message queuing that MT offers. And this feature you've demonstrated is definitely something I'm keeping on the radar. Last project that had something similar was using MSSQL ServiceBroker, but that was ages ago and that technology has become a long forgotten relic. Very cool stuff, Nick.

[@dsvechnikov]

Yo dawg, I heard you like queues so we added a queue for adding messages to a queue so you can add message to a queue while adding message to a queue!

[@MilYanXo]

literally. I am looking and thinking, wth? The whole point of MQ was its simplicity of use, even replacing a MQ engine is not such a big deal that it reauires this massive MassTransit abstraction on top of it.

[@Astral100]

@@MilYanXo Yeah, the problems you run into using MassTransit for a while are not worth it in my opinion.

[@888emce]

There is no need to configure the consumer side. Doing so only makes this example confusing (as people are already asking if it uses a database instead of a queue). The consumer can be also configured to use an outbox pattern (which actually combines inbox and outbox), but it's not obligatory.

[@1235663]

Interesting to hear about pettern for reverse situation. When you read from queue, and something happens with database, or the next queue sending

[@ian_hammond_cooper]

it is worth noting the trade-off here. By assuming you are on EF Core, Mass Transit can avoid you needing to inform it about the "unit of work" that the Outbox needs to participate in. For Brighter we take a different view, and support multiple approaches to your DB: Dapper, EF Core, DynamoDb etc. but you need to pass us a unit of work. We will create this for you, via config, so that you can just use DI to inject it into your code. You then use the unit of work to create the transaction for your Db and the publish to the queue. As ever, trade-offs, but it removes the hard dependency on EF Core for us.

[@mrcarlpaton]

Glad im not the only one using paint to explian things! Brother id love to see a video on the other approach you mentioned to solve for atomic operations over a database and queue!

[@danielmischiu511]

a very scientific tool...opens up paint. Great content man. Many tanks

[@Locustfiretree]

At 16:37, you use AddAsync() to prepare adding a customer to a database. Why? Normally you would only need to use the Add() method unless you have some reason to use AddAsync()? The documentation says as much (e.g. SequenceHiLo).

[@rakeshkumarreddymudda]

12:32 I am confused about how id created and is passed to outbox table which was done before save changes. Does the masstransist handles to update id and send to outbox table?

[@ealeykin]

In EF you can implement interceptor for SaveChanges operation - so this is how MassTransit, most probably, does that

[@rakeshkumarreddymudda]

@@ealeykin got it thanks

[@BrunoDPO]

Thanks once more, Nick! Very interesting pattern! I want to know about the other approach using native cloud products!

[@FSEAirboss]

I'd be interested in know what you have actually implemented and used in practice for you previous requirements.

[@darthrainbows]

How does duplicate message detection work, particularly if the subscriber has many instances and does not have access to the DB?

[@petermihalik9453]

The subscriber must have access to the database

[@chrismantonuk]

Yes I’d really like to know your preferred solution to this problem. I’m not a big fan of this approach, it smells a bit “automagic” - i.e, it isn’t obvious when looking at the service code that the queue message isn’t published until the save changes transaction. I can imagine picking up this codebase and scratching my head as to why the message isn’t in the queue when it looks like it should be.

[@jeremy3264]

In this approach is there a behind the scenes polling of the outbox? If so, that could be costly from a cloud perspective. I've seen patterns that inert into a db, then add a service bus message with a reference to the record in the db, then ack to the client. You need to have a cleanup process if there is a failure between db and service bus, but there would be much less db polling. Not quite as elegant as the Listen To Yourself pattern, but it works with large messages that might not be suitable for service bus

[@julienraillard3567]

Love you're "very scientific tool" as you mentionned :D

[@tanglesites]

Awesome video on Information Systems. I think this was a great introduction to the development side of Information Systems. I think you touched on it, but did not say anything specifically, but what about Event Sourcing. And I vote yes on a video detailing your alternative method of solving this problem.

[@Eatyaatbreakfast]

I've been seeing this pattern a lot lately. Many teams implementing it at work. What is the other approach that you suggest (besides two-phase commit, obviously)?

[@BrunoBranco-kd3ir]

Hey Nick, I would like to congratulate you on another excellent video. I have a question regarding to the need to add EF Core packages to Consumer. The way I was assuming it work is following: - publish push the event to the outbox table - a background process (on the publish side) is responsible for scanning the table, pushing de event to the queue, and the update de status of the event row as processed - the consumer still consumes messages from the queue with no changes needed What am I missing here? 😅

[@ZuvielDrama]

This is nice. I solve things like this just with a Task / Thread Process in the background and a messagequeue as a command/message store to process dependend tasks like this. It is simple and easy and only as many features in this construct that i need for my problem that i have to solve. But i really love this content, because it validates my way of doing things, if big companies walk same ways in a bigger radius of problem solving.

[@nickniebling]

Love videos like this - explain useful pattern and show a good way to implement it using OSS 🙌

[@StefanoTempesta]

Would you send the message to the queue AFTER saving the record to the database? What if you get a DB exception and the message was already propagated?

[@abdulrafeyartist5025]

how does mass transit know that the event shall be tied to customers table ? what will happen when there are multiple queue calls ? perhaps storing data into multiple tables etc.

[@foonlam7134]

We have a legacy system that implements something like the outbox system. Some action will write a message to our custom outbox and some separate background app will check that outbox and send the message. But it doesn't have retries and no idempotency.

[@az6876]

But what happens if there is no consumer? Is there a timeout and api returns error?

[@timseguine2]

7:30 When you said "Don't log like this", to me it was slightly unclear what you were referring to. Do you mean we shouldn't have log lines that change the semantic behavior of a function? Or do you mean abusing the message queue to log something that the creator of the message should have logged? It's hard to tell exactly what you meant, and I can think of multiple reasons why it might not be a great idea.

[@888emce]

He was referring to the concept called "Structured Logging"

[@NicolaiSkovvart]

He meant don't use the object .ToString() as a log message template. Why he didn't just do it "right" is beyond me though, would have taken less time.

[@DJHightower77]

I like to let you know that i am interested to learn how you solve that problem without a outbox. Is it the build-in transaction-support in some messaging frameworks (i think azure service bus supports that).

[@dbohdan]

I am so excited to read this comment

[@BrendonParker]

I think I like the simplistic of save changes the fire the message. Having a relay poll the database and then enqueue the message seems complicated and oddly familiar to using the database as a queue. Maybe my problems aren’t enterprise-enough to warrant this.

[@Adiu72]

But in enterprise scale of micro services each service has its own database and common queue. So it is not the same as DB will be pulled only by one service just to publish messages to the queue while in your solution all services would need to access single DB and pull it.

[@peterk4694]

I use outbox all the time. What's your preferred way to do it? We use masstransit, SQS, RMQ & rdb but not the built-in masstransit outbox.

[@ИванИванов-я5э9к]

7:25 what is the right way to do the logging?

[@xtazyxxx3487]

Using templating to achieve structured logging

[@georgekopadze791]

Hey Nick! Great video, thank you. I have a question though. What if I have 2 micro service with its own database and want to implement this pattern, how do we do it? In this video, you have single database and publisher/subscriber pointing to the same database. Thank you

[@RohitMoni]

Hey Nick, you mention the sponsored AWS course is free from the link in the description, but there is no link? And the course has a price tag on the website? Interested, thanks!

[@nrmagalhaes]

Idk if I miss something about DI injection magics but who guarantees that Publish will work after inserting to database? Isn't the same problem as the beginning?

[@lukegordon4734]

I believe they're both added to the changetracker which EF will ensure happens atomically (they either both are created or neither is). So it's not one then the other. It's either both or neither.

[@nickchapsas]

The Publish method is hijacked behind the scenes and it turns into a database operation that is part of the transaction

[@nrmagalhaes]

Thanks! @@nickchapsas

[@PauloWirth]

A good follow-up for this would be a video detailing the Inbox Pattern, for receiving messages.

[@binoypatel09]

So far I used Azure Functions but this looks interesting, I also wanted to know what other approach do you suggest, thanks for the great turor

[@MeMeNaMi]

Please create a video for your preference for dealing with this issue, thanks

[@RaitLaast-Laas]

Nice! Hey what's that high-tech uml tool you using?

[@mauritsvdp406]

Nick, with the pattern you ensure the message is published exactly once, but how can you make sure the message is also consumed exactly once?

[@ladrillorojo4996]

It includes a "lock" column which should be enough for queries to ignore it if it's not filled with zeroes. I didn't see it but it must also have a "processed" column or something too.

[@Revocdeb]

The message queue should support locking so a message can only be consumed by a single subscriber.

[@Great_Critic]

He didn't explain this. From what I read, one solution is to save message id's to be able to identify already handled message. If already handled - then skip.

[@MeinErsterKanal]

Please let me know how you are dealing with distributed transactions.

[@tuberklz]

actually i need less auto-magic way to implement this pattern. i have a legacy app that i cannot add dependencies, and uses oracle database. so maybe i would just manually write to manually configured tables in oracle, and on the client side, expect to get a similar result. this is about payment system that i need to be sure more than one independent services are processed the request successfully. and, i should not be a bother to writes of main system. any suggestions on that would be great. if not i will try my luck so, maybe i am late and lucky for already you have publised second video you have metioned. if not, please do consider

[@ismailm123]

Another fantastic video, definately interested in the other option.

[@ShaneMontgomery32114]

Could someone clarify why there is a service that reads from the outbox table and pushes the message to a queue to be read and dealt with as opposed to just resding from the outbox table and dealing with the messages at that point in time? IE read and process directly from the outbox table and cut out the message queue.

[@Adiu72]

Other microservices probably have its own DBs. Still if you have a single DB for all then you need to pull it even more often because each service will be pulling it. And you loose built in support for topics etc for queues.

[@alfonsdeda8912]

Hi Nick, very nice video! One question: In api methods that work with related entities(aggregates), when for example I want to update all data in related in main and related entity, is a good idea to pass the related entity id or should get directly with the main entity id all the childs and update then? Sorry, maybe the explanation is not the best... The question is if should I send directly to the api related entity's IDs or should I obtain them from the main aggregate? Thanks in advance.

[@Iingvarka]

The only issue I can see consumer should have access to publisher database or outbox configuration should be enough on publisher side only?

[@nickchapsas]

It doesn't need to have access. It depends on your setup

[@deddykosasih387]

Why does the working need to use DBContext? didn't it just listen to sqs and process?

[@jonasbarka]

I want to know your favorite way of solving the problem!

[@Nouriishly]

But does the worker has to connects to the same database?

[@nickchapsas]

The worker doesn't need to be aware of the outbox. There are multiple types of setup you can have and one of them is for the worker to be agnostic of the db stuff

[@Nouriishly]

thanks for replying@@nickchapsas, I really liked it the concept and started exploring it in a demo project.

[@thiagomatu]

It seems it deleted "physically" the event after processing. Can we soft delete instead?

[@jesussantos2934]

13:52 Somebody knows what console it is?

[@Mec2005]

@Nick, is it possible to use with LiteDB?

[@GiovanniOrlandoi7]

Thanks for the video!

[@romanpelikh1862]

Awesome! Thank you 😊

[@pureevil379]

I write to a table have a status column and my service reads from it :/

[@joshuaodugbemi5864]

Instead of using entity framework in this case, I would manage the db in my code. I would start a transaction, stage the update, send message to the queue. If successful, I commit the transaction, if not, I rollback.

[@A-JesusWlkr]

Is there already a video out which is Nick's favourite approach to this ?

[@nickchapsas]

I'm working on it

[@A-JesusWlkr]

@@nickchapsas Thanks. Blessings to you in your work!

[@claucometa]

Why use await to publish to the queue?

[@ThalesAlves-o6o]

Anyone else not getting the email when registering in the dometrain course?

[@hobbit125]

TL;DR - Have your services store messages in a database table and then have separate service that sends them from there to your message broker.

[@sikor02]

Too bad that there is no easy way to stop listening for messages from the queue. Or maybe there is? What if my app received some message that starts up some heavy process and I don't want to listen on new messages like this from the queue until I'm ready?

[@levimatheri7682]

How does this compare with Sagas? Or is it an implementation of Sagas?

[@JH-qe3fu]

Saga involves taking counter measure if the message has not been delivered is not it?

[@RussellGuthrie]

I too would like to know the alternative approach, especially if it's cloud centric.

[@nanvlad]

Did I get it right that it uses db as a synchronization source instead of messaging queue? Can it be done without using db? As I know message queues have their own persistence layer, so the messages can be saved to a file and then be restored in case of failure

[@dy0mber847]

What if request modifies data in your database which other parts(microservices) of your system are interested in?

[@local9]

@@dy0mber847 Not only that, but I'd rather write to a database instead of a file... those are locks and queues I don't want to get into.

[@T___Brown]

File scales better than db. In microservices you have to create a new table for every db to make this happen or you get into dtc. And scaling reading from 1 table transactionally is very slow. I was originally doing this via triggers and the inserts were fast but i couldnt get my transactional reader to read/queue/commit fast enough. Instead, when queue write fails i write to disk with a prefix of the app name so that later i can rename it then read it and resend it to the queue. This is much more consistent during networking or rmq outages.

[@PedroPabloCalvoMorcillo]

This is a very informative video. Kudos!

[@DNI404]

I'm also curious about the other approach

[@marcobaccaro]

The outbox pattern is applicable for some use cases, in your example it is not the case though. You added extra complexity to your design and pooling in the database. A DTC with retries would simpler, atomic and efficient.

[@JKd3vLD]

How would we configure something like this if we were trying to queue multiple transactions between multiple microservices, each one independent of each other... I wonder... 🤔

[@MeMeNaMi]

Is there a way to do this in DynamoDB? Does EF support DynamoDB? As I saw there's still no support for dotnet8 in AWS Lambda! :(

[@fredericguertin3364]

Theres something really close to EF for dynamo that I used its called poco dynamo, and you could use cdc + lambdas in dynamo instead of this.

[@ian_hammond_cooper]

Brighter supports an Outbox for DynamoDb

[@tomthunderforest1681]

I want to know what is your other approach to doing this!

[@sentenza5497]

Does somebody use MassTransit with IBM MQ? Doesn't seem like it is supported :(

[@qweqwe123qewweqwe]

Why there are three tables, and what is inbox?

[@ealeykin]

What about Debbezium or CDC ?

[@markmahowald7866]

Pub sub is the best. I demo this in paint for our new hires.

[@paulkeating9259]

Defo want to hear the other solution, currently building this very thing at work and we lol'd becuase your released the vid as soon as our meeting about it was done

[@Th3MoL3]

I'm really spooked because this literally happened just after a meeting I had which spoke about the outbox pattern

[@mrplatypus0]

@Nick Chapsas - I would like to know your approach, please.

[@Michel000000001]

Aren't you just shifting the problem? Initially, you had the issue that you couldn't guarantee that both the database transaction was successful and that the message was placed on the queue. You have now solved that problem. However, there is now another, additional process that also needs to read a message from the database, place it on the queue, and then mark in the database that the message has been placed on the queue. Doesn't that process have the same problem? That it might place the message on the queue but fail to mark the message in the database? Isn't it still an issue that you want to perform both a database transaction and a queue transaction?

[@RtoipKa]

What is publishEndpoint? This was not explained. It can't be any custom object.

[@MilYanXo]

Its so funny to see the resurrection of MSMQ spinoffs now that cloud providers decided to sell (err, rent) them. I've been there 3000 years ago when MSMQ with DTS was used for all this and we didn't call it outbox pattern :D

[@thanzeeljalaldeen]

whats ur approach?

[@nothingisreal6345]

When you need to write the message to the DB - why not simply pull the DB via an IP or push some notification to a consumer of the "mesage" service that sometging int he DB has changed

[@abeljr88]

use function app with cosmos db trigger, cosmos db is your outbox

[@as10046]

This video is confusing since is not showing how to implement the design pattern, its showing how to use a black box that already has the design pattern.

[@bluecup25]

Thanks bruda

[@sebastianh7712]

Why does the Worker also need the Outbox pattern and Entity Framework as a dependency? Please watch 15min.

[@ThisGuyLoots]

The very idea that there's a line of code that says Publish and yet nothing seems get published after it's execution bothers me so much, there's so much going on that is invisible to me that it seems like I would never use it.

[@dbohdan]

Thanks for covered topic, but for me seems like overkill. Interesting how everyone deal with this problem, from my side I don't see any problems with creating transaction, save changes, publish message , complete transaction. Interesting about other opinions.

[@TheMikernet]

What happens if after you publish the message but before you complete the transaction, any of the following happens: a network outage, app pool recycles, hardware failure occurs, or web process is shut down?

[@dbohdan]

​@@TheMikernet Got it, thanks,I understand the problem. To implement all this staff you need background worker which read DB (possible bottleneck) and push messages to queue. And background works can have the same problem (network, hardware) so we have to add some logic to prevent problem with background worker and logic to prevent duplication of messages in queue. Maybe it's better avoid using the outbox pattern and just add logic to subscriber of this event where we check that operation completed. Or if it is so critical to use Service Bus transaction. Thanks why I think that this pattern is overkill.

[@davidbrinton1526]

It would be great to hear how else you deal with this.

[@EivindGussiasLkseth]

Yes, I wan't to hear your preferred way of dealing with this problem.

[@harryprince9556]

I wanna know your preferred approach very much