How Netflix Handles Data Streams Up to 8M Events/sec

UPDATE: Checkout the latest developments from the Netflix data team at DataEngConf SF April 17-18, 2018. Follow the link for more details dataeng.co/2u1yElw
Peter Bakas, Dir Eng @ Netflix discusses Keystone, their new data pipeline. Hear in detail how they deploy, operate, and scale Kafka, Samza, Docker, and Apache Mesos in AWS to manage 8 million events & 17 GB per second during peak!
ABOUT DATA COUNCIL:
Data Council (www.datacouncil.ai/) is a community and conference series that provides data professionals with the learning and networking opportunities they need to grow their careers. Make sure to subscribe to our channel for more videos, including DC_THURS, our series of live online interviews with leading data professionals from top open source projects and startups.
FOLLOW DATA COUNCIL:
Twitter: / datacouncilai
LinkedIn: / datacouncil-ai
Facebook: / datacouncilai
Eventbrite: www.eventbrite.com/o/data-cou...

[@malware_creations2606]

Also I've read the Kafka has an issue with consumer lag. How do you handle those ?

[@venar303]

Thanks Peter! What is the organizational overhead of maintaining such a large kafka cluster? Do you have multiple engineers on it full-time?

[@francopedro4910]

Not sure if anyone gives a shit but if you are bored like me atm then you can watch all the latest movies and series on instaflixxer. I've been watching with my gf these days =)

[@adrienjairo6750]

@Franco Pedro Definitely, I've been using instaflixxer for years myself =)

[@arunsinghk1]

Nice talk, very informative.