How Netflix Uses Kinesis Streams to Monitor Applications and Analyze Billions of Traffic Flows

Thousands of services work in concert to deliver millions of hours of video streams to Netflix customers every day. These applications vary in size, function, and technology, but they all make use of the Netflix network to communicate. Understanding the interactions between these services is a daunting challenge both because of the sheer volume of traffic and the dynamic nature of deployments. In this talk, we’ll first discuss why Netflix chose Amazon Kinesis Streams over other data streaming solutions like Kafka to address these challenges at scale. We’ll then dive deep into how Netflix uses Amazon Kinesis Streams to enrich network traffic logs and identify usage patterns in real time. Lastly, we will cover how Netflix uses this system to build comprehensive dependency maps, increase network efficiency, and improve failure resiliency. From this talk, you’ll take away techniques and processes that you can apply to your large-scale networks and derive real-time, actionable insights.

[@staj]

The Netflix Use Case bit is Brilliant! Very insightful Video! Thanks.

[@chrispaul7890]

At about 25:00 he you start talking about the issues with using cache and cache invalidation. You mention that if you set the TTL too long you will have stale (unreliable) data. Aren't you just pulling in events effectively as historical transactions? How does this get invalidated with time? What am I missing?

[@muthukumarkusalavan3261]

i also had the same doubt.

[@MrMylonz]

@@muthukumarkusalavan3261 I think he is indicating that during the enrichment process of resolving the historical transaction's IP address into something more meaningful (which may change over time) by a lookup to a cached value it may be a stale value if that cache not updated quickly enough. Hence a transaction gets attributed to the wrong resource.

[@Eskimoz]

Nice.

[@AbHIShEkUPAdhYaYshekup]

The talk is good but I didn't get towards end if Netflix is using Kinesis or Kafka

[@solderbuff]

29:08 - Both.

[@belisario93]

not nearly enough people laughed at 28:48

[@TouchedAlot]

don't cross the streams!

[@xmsmdmx]

Apparently, nothing strange ever happened in their neighborhoods.