Scaling Event Sourcing for Netflix Downloads

InfoQ Dev Summit Boston, a two-day conference of actionable advice from senior software developers hosted by InfoQ, will take place on June 24-25, 2024 Boston, Massachusetts.
Deep-dive into 20+ talks from senior software developers over 2 days with parallel breakout sessions. Clarify your immediate dev priorities and get practical advice to make development decisions easier and less risky.
Register now: bit.ly/47tNEWv
-------------------------------------------------------------------------------------------
Download the slides & audio at InfoQ: bit.ly/2vR9LFE
Phillipa Avery and Robert Reta describe how Netflix successfully launched their Download feature with the use of a Cassandra-backed event sourcing architecture. They describe their event store implementation and cover what they learned along the way, and what they could have done better. Finally, they review some improvements and extensions that they are planning to address going forward.

[@cupofkoa]

This is probably the most convoluted implementation of event sourcing I've seen.

[@Ianl6311]

Totally agree with Miggleness's comment. It's strange to see that CQRS and Domain Driven Design were inspirations for the team - as you can clearly tell from terms like Aggregate and Command - yet they took a number of liberties with the DDD/CQRS/ES "canon". For instance, naming an Aggregate "Downloaded" is confusing because by convention only Events have past tense names. And normally, you don't introduce a new aggregate "because you have a new query and can't support it efficiently". That's the job of a Read model. You add an aggregate when a new business domain concept arises, or when you have a new *constraint* (a.k.a. invariant) on an existing aggregate and can't support it efficiently. But you always do so along the lines of the language domain experts use.

[@nitrovent]

Yes, that sounds more of an event-driven approach where the "Downloaded aggregate" is rather a microservice that handles the downloads (and has some logic?)

[@harrylee27]

Thanks or sharing this, and it was a great presentation, a good level of details, and passionate speech. Event sourcing can be really complicated, especially this design mixed write and read model together. I thought the consistency requirement was not so high for such use cases, who would download same episode so many times in a minute? So I would probably store "snapshot" in a separate database/table, say simply a counter, and let the service read the counter directly. That would be less coupled, more flexible, and no need to add the repository layer, or make it thinner.

[@xprt642]

It is funny that you started with ES without CQRS and added it later, knowing the scalability requirement of Netflix and what great scalability on the Read side the CQRS gives you (almost liniar scalability).

[@manishshukla5890]

Hey Robert, Great talk man. Some speech i couldn't hear properly but overall it was educational for me. Thanks

[@jssaggu]

Great presentation. Explained a real difficult subject with great real life examples.

[@runesun752]

Regarding the event versioning question that came up at the end, Greg Young is currently writing a book on the topic: leanpub.com/esversioning

[@Miggleness]

Interesting how they placed the business logic in the service rather than in the aggregate class. The aggregate doesnt “do” anything but it “knows”. The service’s role is to ask. The only part that confuses me is the command handler being in the repository. Ive the impression both the service and repository both contain business logic.

[@infoq]

If you ask the speakers on InfoQ they will get back to you.

[@nitrovent]

It's not the first time I see the devs of Netflix implement something in their own way and not the "normal" way that is usually tought. I figure that's not really an issue if it works but it may be confusing when almost everywhere else it's done another way.

[@jonathanhwong2483]

I've been confusing about cqrs for a long time, your business logic determined how you aggregate events, once the business logic changed, means you lose the ability of debugging, you got different result based on the same events, so should I versioning the aggregate service?

[@cristianpallares3847]

Aggregates are meant to be business logic gatekeepers. What they allowed in the past, is already done. But it depends on what you want to achieve...

[@danielpapukchiev3754]

As what I understand when doing CQRS there are the commands that produce events saved in the event store and a query model which is asynchronously gets updated as new events are appended to the event store. How does the command service validate if a certain command is allowed if it has to check with the projections (query model) which is updated asynchronously? From the example everything looked synchronous and there was no query model as all the required aggregates to make a decision on a command were directly taken from the event store.

[@Miggleness]

Daniel Papukchiev as explained in the video, the commands are performed against aggregates which in turn are built from the event store, to the last event. In the worst case, due to a bug or unhandled concurrency scenario, a new event is added while a command is running, the save event operation will fail thru optimistic concurrency control. This will fail the operation back to the service, which in turn is given a chance to retry with the latest event applied to the aggregate

[@infoq]

If you ask the speakers on InfoQ they will get back to you.

[@williamhastings6675]

It doesn't. The command side re-hydrates the model from the event store. It doesn't depend on the projection. Worst case the UI has a stale view of the current state that it got from the read side and when the user attempts to perform an action that will result in a command it gets rejected because of the state mismatch.

[@ameyjain3462]

event source is not equal to logs. I really don't get the use of old events other than debugging which a good logging solution can achieve.

[@VictorMartinez-zf6dt]

The events aren't used for debugging, they're replayed every time you load up the aggregate to get it to the current state, run a command on it which would guard the invariants, and then produce a new event.

[@epicgameryt4052]

You indeed have a history of all changes, which is a small gain. A real benefit is that you guarantee all state changes are initiated by event objects. Imagine if you had two or three controllers updating your account balance, now a developer executes a command manually to lower your account balance because you didn't pay your monthly fees, what would you do? Using event sourcing it doesn't matter, you can rerun literally all events for your account from the beginning to get it back in the right state! Another benefit is that it allows you to determine the app state at any point in time. You can also rerun events to a point in time.

[@emadabushofa2379]

I loved Greg Young's approach with event sourcing and cqrs, but this example used in this video feels ugly

[@epicgameryt4052]

Why does it feel ugly? You are not giving any arguments nor feedback except that it 'feels' ugly.

[@YandryPozo]

now I don't want to use Event Sourcing

[@Budulai89]

Slightly hard to understand the topic.

[@andreacappuccio58]

Very clear explanation of a very ugly implementation. But of course industrial-level code ain't gotta be pretty to be efficient... They've built part of Netflix' backbone after all, so kudos to them.

[@epicgameryt4052]

What makes it ugly?