High Performance Data Streaming with Amazon Kinesis: Best Practices and Common Pitfalls

Amazon Kinesis makes it easy to collect, process, and analyze real-time, streaming data so you can get timely insights and react quickly to new information. With Amazon Kinesis, you can ingest real-time data such as video, audio, application logs, website click streams, and IoT telemetry data for machine learning, analytics, and other applications. In this session, we will discuss and demonstrate best practices for streaming architectures, highlight services such as Amazon Kinesis, Amazon Kinesis Analytics, and Amazon Kinesis Firehose and dive deep into a number of exciting features such as automatically transforming data as it is ingested by Amazon Kinesis Firehose and integrating AWS Lambda into your workflow to create a serverless streaming architecture.

[@gsminfy]

Index Kinesis Data Streams intro = 7:00 Lambda Poison messages = 10:18 Kinesis Enhanced Fan out = 15:09 Kinesis Firehose ITL = 19:25 Firehose with Glue catalog for schema/format conversion = 22:55 Glue catalog demo = 27:55 Using custom prefixes demo = 32:05

[@Art-kz6zf]

Best introductory presentation of Kinesis that I could find. Thank you!

[@najah555]

Great session Randy!!! As always!

[@sbvuyyuru]

It's one of best tech talk i have seen so far, it very clear and amazing presentation

[@Rk23able]

Thank you perfectly explained, learned something new and re-enforced existing concepts.

[@anuyajoshi4360]

Excellent video, entire session had multiple aspects learn. Technical information overview no-doubt was good but below points added benefits - a) conceptual thinking b) how to reduce redundancy with EFO-Pipe c) composed delivery of concept from start till end.

[@nowtify3227]

do you have any idea about camera alert system to sms using kinesis

[@dharam35]

Very neat ! Thank you for a great knowledge sharing !

[@GeorgeZoto]

Thank you for this informative session even if it was a while ago :)

[@progermv]

Great session!

[@YEO19901]

Super !!

[@sakthivel6189]

Good session! Thanks Randy...

[@user-jv4qf7mf9d]

what i should use in case if i need to fetch all data in stream, convert , transform all data and send result by API ? is KDS support converting/transformation in real time or i have to use Firehose?

[@gsminfy]

if you are talking about format conversions or data transformations/manipulations/cleansing etc, I would read the streaming data through a Lambda function, do the transformation and push it to DynamoDB or an S3 bucket or other integrations downstream. if you can do with near-real time, I would go with firehose.

[@jaysonDfernandez]

Hi sir what's the difference between iot core and kinesis. Thank you

[@m_nouman_shahzad]

10:20 Poison Messages

[@sanooosai]

GOOD BETA

[@sukulmahadik0303]

*Notes Part 2* *Benefits of Kinesis (managed service) for Streaming* 1) Kinesis is a managed service and so no infrastructure provisioning and no management. 2) Automatically scales during re-shard operations. 3) No stream consumption costs when no new records to process. 4) High Available and secure (allows encryption) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ *Producers/Stream Ingestion:* On producers side there are number of tools and utilities for ingesting data into kinesis. Producers can be broken down into 3 categories. 1) AWS toolkit/libraries - AWS SDK, AWS mobile SDK, Kinesis producer library)KPL) , Kinesis Agent (we install this on our systems to be able to aggregate the data at the source and then persist out in batches) 2) AWS Service integrations - AWS IoT, AWS Cloudwatch Events and Logs, Amazon DMS 3) 3rd Party Offering - tools like Log4j, Flume or FluentD can have integrations with Kinesis. These normally help us pull logs from applications and push them into kinesis. A producer is an application that writes data to Amazon Kinesis Data Streams. There are several methods for writing data to your data stream: • You can develop producers using the Kinesis Agent. It is a standalone Java software application that offers an easy way to collect and send data to Kinesis Data Streams. • You can develop producers using the Amazon Kinesis Data Streams API with the AWS SDK for Java. There are two different operations in the Kinesis Data Streams API that add data to a stream: PutRecords and PutRecord. • You can develop producers using the Kinesis Producer Library (KPL). KPL is an easy-to-use, highly configurable library that helps you write to a Kinesis data stream. It acts as an intermediary between your producer application code and the Kinesis Data Streams API actions. You can monitor the KPL with Amazon CloudWatch. ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ *Consumers/Stream Processing:* Consumers also can be broken into 3 categories: 1) Kinesis - Kinesis Data Analytics (SQL and Flink) , Kinesis Client Library + Connector Library that will allow us to create our own applications and scale those out using EC2 instances + Auto Scaling. 2) AWS Services - AWS Lambda to build completely serverless applications, Amazon EMR. 3) 3rd Party offerings - Spark, Databricks, Splunk , Apache Storm etc A consumer is an application that reads and processes all data from a Kinesis data stream. There are several methods for reading data from your data stream: - You can use an AWS Lambda function to process records in your data stream. - You can use an Amazon Kinesis Data Analytics application to process and analyze data in your data stream using SQL or Java. - You can use a Kinesis Data Firehose to read and process records from your data stream. - You can develop a consumer to read and process records from your data stream using the Kinesis Client Library (KCL). The KCL handles many complex tasks associated with distributed computing, such as load balancing across multiple instances, responding to instance failures, checkpointing processed records, and reacting to resharding. The KCL enables you to focus on writing record-processing logic for your data stream.