Building event-driven (Micro)Services with Apache Kafka by Guido Schmutz

Please subscribe to our KZbin channel @ bit.ly/devoxx-youtube
Like us on Facebook @ / devoxxcom
Follow us on Twitter @ / devoxx
This session will begin with a short recap of how we created systems over the past 20 years, up to the current idea of building systems, using a Microservices architecture. What is a Microservices architecture and how does it differ from a Service-Oriented Architecture? Should you use traditional REST APIs to integrate services with each each other in a Microservices Architecture? Or is it better to use a more loosely-coupled protocol? Answers to these and many other questions are provided. The talk will show how a distributed log (event hub) can help to create a central, persistent history of events and what benefits we achieve from doing so. Apache Kafka is a perfect match for building such an asynchronous, loosely-coupled event-driven backbone. Events trigger processing logic, which can be implemented in a more traditional as well as in a stream processing fashion. The talk shows the difference between a request-driven and event-driven communication and answers when to use which. It highlights how a modern stream processing system can be used to hold state both internally as well as in a database and how this state can be used to further increase independence of other services, the primary goal of a Microservices architecture.
Guido Schmutz works for the Oracle Platinum Partner Trivadis. At Trivadis he is responsible for the innovation in the area of SOA, BPM and Application Integration solutions and leads the Trivadis Architecture Board. He has long-time experience as developer, coach, trainer, and architect in the area of building complex Java EE and SOA-based solutions. Currently, he is focusing on the design and implementation of SOA and BPM projects using the Oracle SOA stack. Another area of interest are Big Data and Fast Data solutions, and how to combine these emerging technologies in a modern information and software architecture. Guido is an Oracle ACE director for Fusion Middleware and SOA and a regular speaker at international conferences.

[@charliehubbard5436]

From listening to this topic you begin to see how we can't really decide on our choice of architecture. We've done request-response, we've done events, and yet we keep changing vacillating back and forth between the two. Coupled but simple, Decoupled but complex. Kafka's main feature that is different is replaying event streams and it's scale. That is what is driving all of this re-evaluation of architecture design. However, microservices have produced a similar consternation over system design about how these services should interact with each other and how much each service should be responsible for its own destiny (ie a service unto itself, or a collaborative member of a larger system). And, now we're suggesting duplication of storage within the microservices. Each service will now store a copy of all of the data its dependent upon is the advice du jour. At best this is mere duplication of work at worst this has the potential to just recreate many monoliths especially for services that depend upon many things which always happens as product cycles demand. Certainly this can't be good advice for storing user information which has PII concerns and security implications. It doesn't make sense to spread data across many storage tiers that requires extract careful logic and governance unless you're just trying to make your CISO's job harder. Reusing APIs isn't just about data; it's also logic encapsulation! Which if you follow this advice of storing other microservice's means you now have to follow the logic that goes with that data too. Just having the data doesn't absolve you from the responsibilities of using and managing that data. The utopian dream of events everywhere and little to zero request response will be limited by these inconvenient facts. Then there is the issue of microservices need reuse to keep them small. If I can push off logic that isn't central to my problem domain then I can simply reuse someone else's work, and that keeps me light and small. Yes it creates a dependency, but I'm trading off my independence for convenience and a smaller footprint. There is lots to like about Kafka and event streams, and persistence of streams really is what's new here, but I'd be careful about how much this "changes everything" mentality you take away from this because we've seen this play before.

[@Pjblabla2]

Very true Micro services might have worked for Netflix which is largely a read heavy use case But one size does not fit all

[@splashinventor]

This is best presentation and tech talk I heard in recent days. Surely the presenter is highly experienced and precise, concise in explaining the whole picture.

[@boubeniamohamed236]

I found that this presentation is definitly one of the best synthesis. Great work Mr Guido Schmutz.

[@imten5518]

Excellent session. Running through multiple architectures was simply great

[@Oswee]

Really great talk!!! This is something im playing with. I mean last CQRS stuff.

[@igrai]

nice architectural overview + good diagrams, thanks!

[@hovgrig]

Thank you for an ​extremely interesting talk!

[@Pjblabla2]

Excellent presentation that talks about different types of application architecture patterns leading to micro services and then different patterns and concerns for micro services architecture But everyone should make their own assessment on whether microservices is the right architecture for what they are trying to solve

[@ujvalsanghavi4788]

Excellent Talk, nice way to demonstrate how you can evolve legacy systems to use event driven model to make more scalable & available across wide variety of consumer applications.

[@einstein6751]

Great presentation with deep diving into EDA.

[@medmed1714]

Thanks for the presentation. It is really helpful.

[@stack.1]

Very informative and insightful talk, thank you!

[@OutboxThinker1]

It was really interesting to watch, well explained the problems and the approaches as well.

[@dineshkhetarpal4767]

Must watch! Covers all the questions you may have moving a traditional monolithic application to microservices and scale it.

[@tyapka]

Wow, such an informative speech.

[@andrewshiner1606]

This was a very nice talk. Thank you.

[@arkaprovobhattacharjee4858]

one of my favourite

[@pawegraczyk6050]

This was very good.

[@nihontravel]

Good discussion and clear explanation of architectural aspect

[@USONOFAV]

As a full stack java developer working on Angular+Redux application. I somehow get the whole event-driven architecture thing for microservices. But here you are storing the event instead of the the state.

[@johanbester4484]

Well presented.

[@vjnt1star]

excellent presentation full of very useful diagrams !

[@jilljanqueiro884]

Enlightened

[@oyinlolaolasunkanmi]

Thank you

[@AshutoshKumar-zl8hk]

Awesome....

[@easyappsmarketingestudio2408]

Best Regards From Mexico City. Manuel Silva

[@AjayKumar-fd9mv]

Thanks

[@smogstate]

nice presentation, tnx a lot. this is close how we do write our microservices. If you use eventsourcing you can simply expose you events with CDC (debezium), so you dont have any mechanics in you software for event sending.

[@seNick7]

CDC gives you technical events, not business events. EDA won't work on it. CDC is just for copying stuff around.

[@smogstate]

​@@seNick7business event is inside a technical one :) only inserts easy to interpret.

[@connykuehne]

The consumer does not necessarily have to know the offset though (at 33m30s). Kafka stores the offset after commit.

[@guidoschmutz]

Yes, that's correct, but it's still not really "known" to the Kafka broker. It's just another topic which is used to store the offset. Conceptually I would still argue that the consumer "knows" the offset and is responsible for it, i.e. he is in control of how often the offset is commited (into the _offset topic)

[@BryanChance]

"Bus" and "Driver" topics.. I thought words "Bus" an "Driver" are part of Kafka like a message bus or Kafka stream driver, etc. Got me so confused. LOL Great talk.. ..

[@1testrad]

Thanks ...

[@hansenz7033]

This is a great talk, one question that I have is: People treat order Pizza as a typical async scenario and a good example for event driven architecture, like I put an order event, then get back the order finished event etc. Most cases, the client-side just want to get the realtime data, like I try to get my messages, get the latest emails, use credit card to purchase something. Does this mean we can't use event driven architecture here for the online process? Only for the offline BI/Search purpose?

[@HimanshuGupta-nk9yf]

Client side is sending and receiving events as well. Email client is sending emails but is also receiving emails from others to be displayed on its UI.

[@vivienh.missamou208]

hi Guido, Interessing slides and work!! But i'd like to seek attention on the part about microsevices database. you say "microservice might share a central database to exchange states at low level with other (Micro) seevices". I think that's against the microservices principles, patterns and best practices, namely independency and non-centralization. As of a statefull case (vs stateless) and self-boundary , a microservice must handle data (input) contained in its scope (domain/context/subcontext). Thefore, Payment can't share state at low level with Order. But paiment can only do so with a microservice part of its domain context/subcontext boundary. Any external-domain microservice must exchange through endpoint, topic, etc.Because a microservice is and must be responsible for its own inner logic. So when presenting SOA along side microsevices we need to be carefull, as soa is used to implement component-based services:) I really appreciate your job

[@ayaskanta100]

serde --->producer to consumer in-between a schema store known an schema registry

[@rachelleyu1774]

Oracle monolithic computing - RDBMS Kafka distributed computing and data stores Long live the king !!

[@jorgwende6314]

JMS ... 100-200 Events per Second ? You should check this again, sorry!

[@kennethcarvalho3684]

With kafka should be possible right??

[@guidoschmutz]

That's what we got, of course it was a couple of years ago. I'm talking about persistent messages, send within a transaction to a clustered JMS message broker. The "within transaction" is a bit an unfair comparison, I agree, as Kafka does not participate in transactions. But the sending of persistent messages is important. Sending JMS message in a "non durable" way will of course allow for a much higher throughput.

[@Alperic27]

Confused explanations

[@lusiek21]

SOA and (Micro)services that is original. I guess (Micro)services are not real microservices. Yesterday's technologies with Kafka :)

[@guidoschmutz]

No, the idea was to say that Micorservices could be seen as SOA done right. I'm not promoting "old SOA style services" communicating with Kafka. But it also does not have to be Microservices to get a benefit with decoupling through Kafka. That's why the title is (Micro)services. At the end you can profit as soon as you start creating smaller application modules and Kafka can help to decouple them.

[@shagging8727]

of of of of of of of of of....

[@lusiek21]

Nothing useful here.

[@abdulkaderjeelani]

RedPanda