Event-Driven Architectures Done Right, Apache Kafka • Tim Berglund • Devoxx Poland 2021

[@alexanderpodkopaev6691]

"If it is small program, you don't call it a monolith. Architecturally it IS one. I want to give you permission just embrace that" - BRILLIANT! Noted for further discussions with middle developers, thanks a lot Tim!

[@DevoxxPoland]

Good catch 👍

[@tarunsingh7608]

@@DevoxxPoland , a Great SALES pitch for Confluent Cloud by an architect who hardly discussed real time scenarios or let us Learn something concrete about Systems Design of Distributed systems or Event driven architecture

[@KirBirger]

Thanks for this. There's a shortage of honest content out there that doesn't simply evangelize concepts and approaches.

[@mosespeter9711]

This is one of the best talk I've ever listened to

[@DevoxxPoland]

Tim at his best

[@vanivari359]

Don't want to complain too much, but from an expert like him i kinda expected a bit more than: "only use EDA if you need it", "Use DBs if they are a good idea", "the event schema is kinda important because stuff changes" and "stuff is hard, don't implement your own scaling or state management". I was already suspicious after the claim that there are 5 ways to fail with EDA because i've seen at least 10. And the 5 in the talk are for sure valid points, but not the stuff (except scaling) projects typically struggle with. The real challenge is stuff like eventual consistency and the lack of transactions - issues which could have been discussed pretty well with that reference architecture. But hey, now i know that he wanted to buy tailored shirts once... but didn't. ;)

[@KirBirger]

I'm not sure what you want here. The guy broke down some concepts, and explained the pros and cons of some approaches. It's up to you to look critically at your use case and determine if you "need" it.

[@samanthamccarthy9765]

exactly what do you want here .

[@benisrood]

​​​@@samanthamccarthy9765 Hi Samantha, I am guessing that the OP wanted Tim to forgo the digressions-and digressions upon digressions! 😂-and associations. Instead Tim could have just methodically iterated through the various examples and cases he wanted to cover. Not Tim's style, I grant you, but it was more than a little unnecessary this time, and makes it harder to follow the points he is making and wanting to connect together. Not a bad talk, and absolutely a necessary talk, but I can also imagine that the next time Tim gives this same presentation he will probably refine the way he gives it. Maybe it was also situational, perhaps the Poles were a tough crowd 😂 and Tim was nervous!

[@darrenhorwitz1860]

this guy is brilliant, I love his sense of humor and what a great talk !!!

[@marekchodak6705]

I usually watch on 1.5x speed but he is so good to listen i kinda dont want to

[@DevoxxPoland]

Wow. One of the best compliment that I saw. Thanks a lot!!!

[@arto00-g2n]

Or you miss most of it at that speed 😅

[@3enny3oy]

The last 15 minutes of 2001 a Space Odyssey 😂 I’d say that’s where all the meaning of the movie is. And yeah, don’t be afraid of a monolith if you’re building something small and isolated. If you can get something up and running to do the job with little up front effort or admin overhead, it’s probably fine to spend the extra effort to migrate to an event driven architecture in the future once you’re actually generating value. If you over engineer up front it will be far longer before you start to generate business value and it will probably cost more in the long run. Build fast, fail fast and iterate. Just avoid decisions which will lock you down and make iteration difficult. Be pragmatic and agile.

[@atkinpaul]

Thanks! I hope you do engage more with Data Mesh. Zhamak seems to de-prioritize operational data flows but there is a massive opportunity to converge operational and analytic pipelines when you use an event driven architecture combined with polyglot persistence.

[@smaug9833]

Another pitfall of event driven architecture is that one service derives the whole context from the message it consumes. But sometimes, there is a business need for a service to maintain its own context in addition to what it reads from the topics. I know this is not the way to go for purely event driven services, but sometimes you are forced into a situation where you have to do this.

[@alexanderpodkopaev6691]

Yeah, something like this one: A user payed for an order, then decided to change shipping address and placed another order. So, where 1st order will be shipped in this system? Because shipping service maintains its user DB , I assume shipping address is not included into order or payment event. With such design, I guess chances that address change event will come and would be processed BEFORE order payment for order#1 event would be high for any reasonably loaded site causing shipping to wrong address. During promo or BlackFriday - it's almost guaranteed.

[@cxmais]

do things right, don’t do things wrong

[@mehrdadk.6816]

The reason for developers think when using event-driven is a database replacement can be , there is no example that uses the combination in Confluent materials. Maybe this can be getting done

[@gerritlikestohike]

A nice talk but I have to admit the takeaways are very little here. Real pitfalls which you need to takle when writing a real QRS or DDD event drive architecture have not really been adressed. So if any1 is looking for answers to those you don`t have to watch this.

[@DevoxxPoland]

Thanks for your feedback.

[@sarwan.surchi]

the real pitfalls as he mentioned here are I think scaling and state management which is hard to figure out on your own and hence he recommends tools to be uses.

[@joshzwicker]

Asked event-driven-architecture into your heart 😄👍nice metaphor

[@DevoxxPoland]

😁

[@RaviChandraEnaganti]

Isn't storing/maintaining user data within the Shipping service an anti pattern? Each micro service is supposed to handle one domain?

[@immrsv]

IMHO, the shipping service isn't 'maintaining' the data so much as it's just keeping a cached view of the data. All the rules around the handling of User data are still kept within the User service.

[@kitkarson4226]

It is called materialized view . It is not an anti pattern

[@MikeZadik]

The materialized view thing is obviously bullshit. I think the point it to avoid coupling between shipping service and user service. If the user service is down for example, you could not ship orders because you cannot get hold of the addresses. Or if the interface of the user service changes, you might have to change the shipping service in tandem. And as @immrsv writes the shipping service still doesn't handle the user domain, he just saves the data that is relevant for it's own domain. I think it's also reasonable to allow those cross-service calls. You just need to consider the pros and cons of either. A EDA purist would probably put it all on the event bus.

[@AlexanderSergeenko]

Thank you for the great talk!

[@DevoxxPoland]

Our pleasure!

[@tanertasim3637]

This presentation is just gold!! Very clean and precise explanations! Thanks!

[@RoarkeRandall-q4x]

Given the event log is the system of record, and lets say we want the customer's address updated in the user service. Does that mean we have to fetch the current version of the user from the event log? Seems challenging and requires specific technologies. Alternatively you'll use the DB in the user service. By doing that, the DB has effectively become the system of record, right?

[@ibgib]

I've enjoyed the first 30 min, and I always jump the gun on these comments (am I the only one? )...but at a 29:50 you mention more evolutionary architecture. Well I've been developing a web3 protocol since before bitcoin that came from the event sourcing side of things, with the idea that you can apply transforms as data ("event" analog) that live on the same Merkle DAG timelines to produce more "living" data. Think git applying semantic diffs (but not text-centric) to create/recreate branches per use case, as opposed to applying events strictly to rehydrate aggregate state. I see that the speaker is a git guru, and I actually reimplemented much of their internal DAG storage mechanism (even completely coincidentally using the ^ for a delimiter). Anyway it's hard to express to computer people because most are already enamored with bitcoin or existing event sourcing paradigms, but both share the same difficulty when it comes to scale: that a "single source of truth" is a very funny thing, and that we can produce more biological (evolutionary) code which even unifies microservices with the future of ANNs/RNNs/Transformers/IoT, and allows for sovereign node implementations (like the speaker's continual reminder that RDbs are good). It's just really good at unifying/simplifying what kubernetes and other orchestration systems do, with the ability to have consensus structure/specs data "on chain" and able to be shared alongside the data itself. Anyway, it's hard to find people who know ES and git internals extensively, and if you're interested you can look at ibgib on Github (prototype) & GitLab (low level DAG graphing lib). Also I have a few (crappy!) YT videos. Perhaps I could explain the evolution from ES design more in depth if you'd care to connect. Thanks for the video! (And getting this far in this comment hah)

[@isorry123_]

i also around ~22m was thinking why not just replace database and notification system with a blockchain ledger. write and read directly to it.

[@ibgib]

@@isorry123_ yes exactly. But the devil's in the details of how you decide to allow for performance/trust/availability/etc tradeoffs. For example, a blockchain with a hard-coded consensus algorithm like bitcoin has only one "shape" that it can take WRT how it performs & where it centralizes. Ethereum, both pre- and post-Merge, will also be a one size fits all, and this will always generate lock-in with e.g. requiring solidity to enforce their beliefs on how the math guarantees their level of trust. The same goes for every blockchain-based approach that I've looked at, though if you take the entire blockchain ecosystem as a whole, you get a wide range of dynamics. My approach literally uses the biological paradigm by enabling version control-like dynamics - diffing + DAG like in git, but with the diff transforms themselves as metadata as well. These transforms are not text-diffs but rather semantic diffs, each transform itself being a node in the DAG. So in the event sourcing world, this is like having a redundancy of storing both the events and the hydrated state, which sounds like a waste. But if you use them in context of version control, where you apply the same transforms to **different** branches, you come up with different results (something not wanted in event sourcing). So it ends up looking like biological organisms, having the transform dna "genotypes" and the expressed "phenotypes" of the state when actually being consumed in an app. Interestingly, this has many positive after effects like idempotent CRDT behavior. So when I go to "synchronize" two spaces (merging multiple timeline branches), I can simply look at which transforms have been applied...though only when order is not important. (Ordering implies that you should have a localized space or the standard barrier/critical section/etc parallel constructs. But I digress... My point is that this enables consensus specification to be "on chain" in a consistent manner, whereas with all approaches I've seen host their code on github with no plans to move. (Mine also is on git because I don't have time to implement the version control cli while also doing the entire rest of the shiny UX stuff that I suck at) But the point is that git itself is a blockchain (DAG) under the hood, so right off you're duplicating code, increasing surface area, and decreasing flexibility and agility because you have to edit the code in git as opposed to the code already existing in a granular fashion in its own "version control". This flexibility is absolutely necessary to actually implement the unification that you're talking about. There are a couple projects that are somewhat close (IPFS/IPLD/etc and w3c's rdf triples addressing protocols which have changed over the years), but I don't see them getting the whole picture.

[@shashanksharma8254]

This is how to make complex things Simple enough.

[@yangyun6221]

Same in Mobile App Developing and small companies. Every people say use MVVM, but your code has only 10 views. And small companies pay lots of money buying managing SAAS tools while they only have 5 people.

[@DevoxxPoland]

Thanks.

[@ChandraShekhar-by3cd]

Thanks for the Amazing talk.

[@vernetto]

I was really interested in Kafka, but I will look for some other video where the speaker goes straight to the point without a lot of blabla

[@DevoxxPoland]

Sometimes introduction is necessary for better understanding and explanation

[@BosonCollider]

Just a note that Something can be logically event-driven while being a monolith from a devops perspective. For example, many programs in Go, Elixir, or using Streams in Python anyio/trio are event/message queue driven. They just happen all to live in the same runtime and don't require Kafka to act as a message bus. Imho using language native multitasking in this way is a good way to build a system that can be refactored into distributed systems. And especially for the case of Elixir, the BEAM is just so reliable that you will probably not need Kubernetes or anything fancy to keep it going. Just put it in a freebsd or Rocky/RHEL VM with a managed database and you will have very good uptime.

[@debugin1227]

How do you handle data security in this scenario where a one business unit should not be able to see / consume the data of another business unit in a company? Some users might be able to see qty/price in their plant, other users maybe qty only.. how does this work in these events?

[@dinoscheidt]

You can use Crypto Shredding (field level encryption). Can / is also be used for personal identifiable data. Meaning: Users that do not have access to a key due to their role, don’t get the data. Thats what key vaults are useful for.

[@joelkorpela2706]

@@dinoscheidt That's a fascinating idea for data access control when multiple services consume the same event. Thanks!

[@rdean150]

If the universe of consumers/permission permutations is finite and manageable, the approach I've seen taken is for the owners of the original topic to create consumers of their own topic that then publish filtered subsets of the messages to new topics. The downstream consumers would then listen to the filtered topic instead of the original topic. This can potentially be configured dynamically. This is surely not the best solution, and may be difficult to scale, but for relatively limited use-cases, it gets the job done.

[@TheRedbeardster]

T-shirt really rocks!

[@sergeykichuk2586]

What you described it is not Event Driven Architecture! Correct me if I'm wrong but what you described here as reference architecture with event logs and database as views is more like Event sourcing and CQRS where source of true is your events!

[@sarwan.surchi]

@sergeykichuk2586 this is the bigger picture described as Event-Driven Architecture, ES and CQRS are definitely in play here but he simply avoids explaining them. ES here is when materializing/projecting user data into shipping and CQRS is obvious, the requests.

[@hmaina0]

This should've been a 15 mins talk max

[@flynnblu6992]

You had me at persnickety.

[@nullah100]

Nice and Informative

[@DevoxxPoland]

Thanks

[@VasuNori1]

this talk can be condensed to 10 min. too much random stuff.. isn't the audience mostly engineers? or a bunch of know-nothing newbies?

[@diegorosadossantos8493]

Thanks for sharing your knowledge 😃

[@DevoxxPoland]

Our pleasure...

[@thepracticaldev]

Nice intro!

[@DevoxxPoland]

Glad you liked it

[@petersuvara]

Otherwise known as GOTO hell :)

[@youtux2]

Today's new hot microsetvice architecture is tomorro'w legacy systems to bear with.

[@mjj0829]

I think he knows a lot and the agenda was appropriate but man he talks a lot. What he covered in this 50m can be said in 20 or less. I wish he actually put more depth into it rather than small sideway talks. I watched it at 1.75x and skipped a few times.

[@mugishaalainchristian2613]

This man is hilarious haha :)

[@juilipanse-kanade9583]

thanks for sharing!

[@DevoxxPoland]

My pleasure!

[@sergey5551]

if we will skip "water" in the speech, video will be way shorter......

[@lhxperimental]

Good talk for some reason I was reminded of Louis CK as I watched.

[@samlee4490]

tough crowd

[@flesz_]

500 euro for an entry ticket to a conference , what a joke

[@LarsRyeJeppesen]

Lol I knew he would say ,"React"

[@bernaridho]

The word ARCHITECTURE is overused. Too many things are named with Architecture.

[@rdean150]

Perhaps, but in this instance it is definitely appropriate. This topic is about a complex arrangement of interactions between independent processes running on separate hardware but working collectively in a concerted fashion to achieve a task. The roles of the various components, the manner in which they interact and contracts they hold between each other, and the spectrum of implications that the particular arrangement results in - this can aptly be described as the system's architecture.

[@DevoxxPoland]

Good point

[@jopsuey]

Ultra bored

[@madpuri]

Do an interesting lecture and put it in KZbin

[@a_guy_called_Jerry]

hard audience it is

[@miguelfernandez4051]

You go too off topic too often.

[@ftnsco]

Omg 🥱🥱🥱 this video has many annoying points

[@DevoxxPoland]

What do you mean by annoying?

[@calebvear7381]

@@DevoxxPoland I think zero is referring to some of the tangents that came up. Eg he is about to tell us something and then decided to explain his tshirt instead.

[@xamax4]

@@calebvear7381 breaking the line of thoughts

[@JosiahWarren]

I was so tired throughout the presentattion and learned nothing new. Congrats🫤