Martin Kleppmann - Event Sourcing and Stream Processing at Scale

Domain-Driven Design Europe 2016 - Brussel, Jan 26-29 2016
dddeurope.com - / ddd_eu
Organised by Aardling (aardling.eu/)
If an idea is good, different communities will independently come up with it, but give it different names. For example, the ideas of Event Sourcing and CQRS emerged from the DDD community, while similar ideas appeared under the title of Stream Processing in internet companies such as LinkedIn, Twitter and Google.
This talk attempts to bridge those communities, and works out the commonalities and differences between Event Sourcing and Stream Processing, so that we can all learn from each other.
We will discuss lessons learnt from applying event-based architectures at large scale (over 10 million messages per second) at LinkedIn, and how such systems are implemented using the open source distributed messaging projects Apache Kafka and Apache Samza. We'll also discuss some of the architectural choices that affect scalability (both scalability in terms of data throughput, as well as organisational scalability).

[@chrise202]

Excellent talk! As a feedback, please when people ask questions at the end, repeat them so the audience has a clear understanding what is being discussed.

[@jay31415]

I recently read his book, "Designing Data‑Intensive Applications"-highly recommended. You will not find a more qualified speaker for this topic.

[@herturluhalikilimtravel]

Excellent speaking, thank you.

[@Oswee]

Such a MASSIVE topic and only 300 views... :(

[@exitiummagnum]

28:17 Depending on the downstream, the receive order might not matter but eventually (e.g. ORDER BY in a SQL OLAP batch job), and then you want to have an sequence number generated in the most upstream source (ie. session)

[@CostisPan]

Very exciting topic, masterfully presented.

[@praskatti]

thank you Martin. what kind of data base is used for storing the events?