Seems like using a callback via a webhook after a message queue reply is picked up, would be a cool way to roll. Provided that end point is part of the message data.

Maybe I missed, but Materialized Views are another great way of having read-only tables with non-normalized data

Dealing with aggregations in caching scenarios can be difficult to get right or even do at all. If your API allows for many inputs, then caching becomes cumbersome or ineffective due to having to identify and execute on persisting the maximum level to aggregate to as possible without breaking functionality for any of the inputs, but still aggregating enough to make it worth it. I always feel like I should set up caching for these scenarios and then end up realizing it's not the best solution.

For me the hardest part is the concurrent nature of a cache, even worse if it is also distributed. Some day I'll learn TLA+ or such so I can get it... Someday... :)

I had a performance issue about a month ago. Ended up opting for memory caching. The cache expires after a couple of minutes.

Why do you put an IDE thumbnail? I think you're going to show some code to explain your idea every time.

In addition to the last part, always measure performance first! Get some metrics from your production environment so you have an objective view of where optimizations are needed so you don't waste time and effort on hypothetical scenarios. It's also very rewarding to see graphs go down after you've deployed something, as a little treat for yourself ;)

There are 2 hard problems in computer science: cache invalidation, naming things, and off-by-1 errors.

@6:17 You really need to watch the pronunciation of 'cache-hit' 😁

kzbin.info/www/bejne/fHypqYqorMl0jcU this is what I always say to my developers and customers, but most of them don't listen and want that extra work with no benefit.

What can I carve off...? And there you ave your Aggregate root

Hi, this is a great video. Can you share the code that you have been using to debug? Thanks in advance.

Through the years I've become convinced that it is not an app that owns the database, it is the opposite. The database owns the app (or multiple apps). You take care of the database and evolve the apps accordingly

Transaction does have a timeout, so for X long message processing, when X > transaction timeout, you must commit twice, and have select a condition on "processing" IS NULL

Very helpful video, imma be using it on my next project and it is very good to see a good example of all the theory I've been consuming.

Another great video Derek. There are a lot of devs out there not building new things with shiny latest tech, but really dealing with improvements to legacy systems 😊 So videos like this are really helpful.

I probably can't post a link in comments, but your video titled "Change Data Capture + Event Driven Architecture" at ~2 : 52 is a good one to pair with this! As always Derek, thank you for the great approachable content. Hope one day I can meet you at a conference or similar.

Another great video, thank you. I work in a warehouse execution system and there is no ownership of data. Inventory, putaway orders, and pick orders for example are modified in multiple services with no concern of ownership or concurrency issues. One service could be cancelling an order while another is picking it and there is no type of concurrency safety in place, I contribute that to what you mention in this video since the writing and reading of data is spread across every service. This makes everything from implementation to testing difficult because you have no idea who owns or changes what data.

Focusing on the Use Cases is better.

Stealing "6:50 Free for all in the database" :) Finishing up a 200 table ORM jungle with tons of transactions cutting across multi billion row tables with millions of users. Usually I see the database schema is in star or snowflake 3NF when all the user data changes heavily in time together and should be event sourced. I've found looking for a common "fact" table that has a sequential primary key, joining off of that to get to all the "dimension tables", then persisting + reacting to that works great. As a bonus you get exactly once delivery instead of using a persistent message broker.

Shoutout Detroit

Thanks for this very actionable advice! Everything comes down to managing coupling and encapsulation.

👍👍

Why dont you just create a comprehensive tutorial of the most effective patterns and architectures and we all pay handsomly for it.

As i here your wise advice i have failed promise of OOP before my eyes, especially missed shoot at encapsulation level - betting on objects more than modules

I'm currently working on a big spaghetti codebase where I'm trying to migrate databases. It does a lot of things including generate reports for that data. And the way I've approached it with this mindset is I'm moving out reporting to a new service. It has basically forced us to implement a way to know when data changes to be able to update the data in the new reporting database. This also means that when I start moving out other parts of the system, I will be able to still have a way to hook into these events and have the data I need to perform whatever it is I need to do. For years this project has struggled because it is too big and being supposedly impossible to refactor without a big bang rewrite but I finally see a light at the end of the tunnel thanks to this pattern. Great advice.

I work on a 20+ years old shitty laravel codebase without any test and with gigantic, 2k+ lines, files 😅 I doubt we can fix it, even more so with only 3-4 devs on the team 🤣🤣🤣

I think CRUD caught on because REST caught on, and REST caught on because HTTP won. All the HTTP verbs lend themselves to working on a certain resource, and these resources map pretty well to database objects.

I love the square peg round hole. It’s been my intuition for a while, but I haven’t heard someone say it. I love thinking about my APIs as actions, RPC style

Super great explanation, this channel is a Gem

Usually spaghetti codes are untested systems. So you'd probably write a ton of tests upfront and get to know the domain before you change anything. At least that's how I got started. Interestingly, there's a social component to the problem, too, as if the technical issue wasn't challenging enough. Most likely there's a "senior" coworker who loves ideas like DRY and doesn't understand that it introduces lots of coupling. And also the management has to be technically incompetent because they don't interfere. So you're probably dealing with a lot of incompetent people who won't understand your agenda. It's often hard to pull off a momentum shift such that people trust your judgement. Also the business side needs to be addressed to fix the trust issues between developers and managers. You need to come up with a plan that'll yield measurable results within a few weeks. It's often hard to even identify the problems and figure out how to get started. Usually there isn't any documentation either, so you'll be researching a lot. Once you identified how to tackle the project, set the expectations with management. Usually they don't understand how to work experimentally and go step by step without making big plans. So it can be hard to convey your mission. You'll probably need to do some stuff under the radar to avoid micromanagement. If you can show that you approach works, you'll probably get the ship turned around. Now you have management that trusts you and you begin to understand the system. Tests stabilize the system, so you can finally introduce some refactorings. Now things will accelerate.

Great advice! To get management buy-in, focus on other benefits as well. While important to reduce failure rate, it's also great to get better opportunities to e.g. consolidate logic, improve performance, add a cache, add monitoring, modernize tech stack, move from vertical to horizontal scaling (containers?) or whatever you might be eyeing at the same time..

Timely advice. I currently have.a system where data is duplicated in mongo, maria db, sybase and SAP where the same tables are updated via multiple apps. I am trying to convince management the idea of ownership. Then there is the reporting where they hit a mariadb directly via a third party reporting system (tables with 200+ columns) and I keep telling them to create an API because maria db is failling them and we need to migrate since scaling is an issue. I think they are sick of me in meetings because I keep asking which app should own the data . I think they understand but quite frankly there are so many apps spread out and what I am suggesting will take time just with existing functionality it is virtually impossible to get buy in.

JavaScript

Worse than the new "Kurrent" name can only be its new logo (along with the brand color theme). Something strange happened to EventStore brand.

Would that local cache be in memory only, does it still need to call other microservices in second case?

🎯 Key points for quick navigation: 00:00 *🛠️ Derek Martin reflects on 10 years of software architecture, sharing lessons from good and bad decisions made during development.* 00:13 *📈 Understanding industry trends is crucial; staying updated helps inform architectural decisions over time.* 00:27 *⚠️ Using AngularJS for a single-page application (SPA) limited evolution options due to its monolithic nature.* 01:39 *🔄 A good decision was avoiding microservices, opting instead for a well-defined monolithic architecture with clear boundaries.* 02:21 *🗂️ Defining boundaries around business capabilities allows for flexibility and adaptability in evolving the architecture.* 03:30 *📦 It's essential to apply architectural patterns like domain-driven design selectively, based on the context and specific needs.* 04:40 *⚙️ Implementing a web queue worker pattern can help scale applications and manage asynchronous tasks effectively.* 06:12 *🌐 Sharing a codebase for both API and worker processes simplifies development and maintenance.* 08:03 *📩 Event-driven architecture promotes decoupling, allowing independent processes to operate without affecting each other.* 10:24 *🚫 Not all events need to be stored in the same stream; compartmentalizing data improves clarity and utility.* 13:22 *🎯 Best practices in architecture involve defining clear boundaries to facilitate evolution and reduce complexity.* Made with HARPA AI

Thank you for your great explanation way data from different microservices can be composed by BFF component and share to client. I was looking it. I would have a other question about duplication data cross different microservice. Let's assume simply scenario for ecommerce solution: Catalog microservice gather products with price. User adds selected product to basket (basket microservice). Should basket item contain price? If let's say in checkout process there is a need to verifiy user limit is exceeded or to sum order price. I imagine basket item contains guid reference to product from catalog bounded context but there is need to keep price too? Or maybe is enough in checkes process sync request (http request) should be done to get price from product context? Regards :)

Great content, watching you for some time now, and I'm not C# developer, but your content really helps! :D How would you represent vehicle in Unhook Trailer action and Dispatch Order? I really didn't understand if those are separate entities in db, or is it just pure logical separation?

this is terrible advice. the C# type system is not sophisticated enough to stop you from forgetting to check the Error Values (unlike in a lot of Functional languages). no point in pretending that it is (this is not Haskell). it hurts the reliability of your code because you will inevitably forget and that will result in a silent error at runtime, where you would have had an exception. that's an actual "landmine". you have to adapt to the language you are working in 2:51 .NET is full of code that can throw exceptions. you haven't even considered what your code would look like .NET returned an Either type instead of exceptions. it would add unnecessary code churn. most code is not interested in all the specific possible exceptions that can be thrown. most code is only interested in whether something succeeded or failed, in which case the error is reported. Exceptions allow this kind of code (which is the majority), but your solution does not.

sounds very subtle

Thank you for the clear & simple explanation!! 👍👍

So many companies think they can go On-Prem and then end up paying top dollar for cloud and system engineers to migrate them to the cloud or struggle maintaining hybrid systems that requires complex networking and expensive integration equipment (like DirectConnect in AWS). Self-hosting is great if you are a massive corporation who has the resources to maintain all the hardware and networking components , as well as the complex software require deep-level system and networking knowledge to maintain them and it is utmost importance to have full control of your data and infrastructure (Think like Top Secret Gov stuff). Not to mention the extra amount of security personnel you will need to protect any type of software behind your own self-managed infrastructure. It is hard enough to keep applications secure on the cloud with a full cloud team , just multiply that many times when hosting everything yourself On-Prem. It's like saying why use higher level software languages that manage your memory and abstract away all the complex algorithms , networking , system drivers, UI code, etc when you could just manage it all yourself. Yes, I get it you have to pay for your cloud infrastructure, but you will also have to pay to run a data center or to lease the equipment in the data center. The more services, portals, consoles, etc the data center provides you the more "cloud-like" they become and then you wonder is it truly better than a true cloud platform?

I didn't get the first part about having SPA. Are the regrets about not having server rendered pages instead, or using Angular in particular ? I also didn't get the part about "monolith" on the front end.

I think most of the time micro services is just the wrong solution to a very real problem - you are not able to separate logic in you code and put everything in the same class and make everything dependent on everything. This both applies to angular js and backend code.

The problem with event driven architecture is not the learning curve it is that it is quite hard to debug. You can't just run your project into the debugger and step through. You need to chase items through queues and through multiple systems. Sometimes it is needed but it is not free.

Love this video!!! My best decision was to move to micro- frontend with microservices !

The best decision I made and am constantly making is to spend time outside of my regular work hours to read and learn. Countless times I was sitting there at either 23 in the evening or 5 in the morning reading a blog or book, watching a YT video or listening to a podcast and getting the moment of realization: "oh, this would have solved my problem back then", or "this is exactly what we did wrong, now I understand why it was a mistake" etc. Never stop learning.

monoliths. yuck.

Amen to this! It just creates spaghetti. My least favorite is when people extract methods and name it something like "DoXAndDoYIfCondition". SRP gets lost on these people and they might as well just not make a separate method at all. And as you say this is especially important when throwing exceptions. I should be able to see from a parent level exactly what exceptions will get thrown for certain conditions not dive 5 methods deep to find it.