Batch Data Processing with AWS Kinesis Firehose and S3 | Overview

In this video, I go over AWS Kinesis Firehose and how it is useful to batch data and deliver it to other destinations.
Looking to get hands on experience building on AWS with a REAL project? Check out my course - The AWS Learning Accelerator! courses.beabetterdev.com/cour...
📚 MY RECOMMENDED READING LIST FOR SOFTWARE DEVELOPERS📚
Clean Code - amzn.to/37T7xdP
Clean Architecture - amzn.to/3sCEGCe
Head First Design Patterns - amzn.to/37WXAMy
Domain Driver Design - amzn.to/3aWSW2W
Code Complete - amzn.to/3ksQDrB
The Pragmatic Programmer - amzn.to/3uH4kaQ
Algorithms - amzn.to/3syvyP5
Working Effectively with Legacy Code - amzn.to/3kvMza7
Refactoring - amzn.to/3r6FQ8U
🎙 MY RECORDING EQUIPMENT 🎙
Shure SM58 Microphone - amzn.to/3r5Hrf9
Behringer UM2 Audio Interface - amzn.to/2MuEllM
XLR Cable - amzn.to/3uGyZFx
Acoustic Sound Absorbing Foam Panels - amzn.to/3ktIrY6
Desk Microphone Mount - amzn.to/3qXMVIO
Logitech C920s Webcam - amzn.to/303zGu9
Fujilm XS10 Camera - amzn.to/3uGa30E
Fujifilm XF 35mm F2 Lens - amzn.to/3rentPe
Neewer 2 Piece Studio Lights - amzn.to/3uyoa8p
💻 MY DESKTOP EQUIPMENT 💻
Dell 34 inch Ultrawide Monitor - amzn.to/2NJwph6
Autonomous ErgoChair 2 - bit.ly/2YzomEm
Autonomous SmartDesk 2 Standing Desk - bit.ly/2YzomEm
MX Master 3 Productivity Mouse - amzn.to/3aYwKVZ
Das Keyboard Prime 13 MX Brown Mechanical- amzn.to/3uH6VBF
Veikk A15 Drawing Tablet - amzn.to/3uBRWsN
📚 References:
PART 2 - Kinesis Firehose to S3 Console Walkthrough - • AWS Kinesis Firehose t...
S3 PUT to Lambda Trigger - • AWS S3 File Upload + L...
Getting started with AWS: • Introduction to AWS | ...
☁Topics covered include:
- Kinesis Firehose
- Kinesis Firehose Example
- Buffer Size
- Buffer Interval
- Best Practices
- Gotchas
🌎 Find me here:
Twitter - / beabetterdevv
Instagram - / beabetterdevv
Patreon - Donations help fund additional content - / beabetterdev
#AWS
#Serverless
#Kinesis
#Lambda

[@DarkApplesTasteBest]

Now, I finally understand why it is called Firehose, if you want to kill a fire and you walk up and down with a bucket(single event), it's not efficient and will cost you extra energy. With a firehose you direct large quantities of water (events) to the fire(database).

[@henryeleonu6237]

Great! This is straight to the point, and I like that you explain these AWS services with use cases. thanks for your content.

[@kino-xw4xg]

thank you so much for your videos, your channel has become my go-to channel for anything aws related!

[@raphael-okere]

Thank you so much for all the content you provide. You really help people become better devs

[@BeABetterDev]

Thank you so much fro the kind words Raphael, really appreciate it.

[@TomerBenDavid]

Best practice section is a great added value! Thanks!!

[@BeABetterDev]

Thank you tomer!

[@hasanbiyik01]

Huge thanks for your efforts!

[@SimoneIovane]

Clear explanation of a business case. Thank you

[@BeABetterDev]

You're very welcome Simone!

[@TivoKenevil]

You're awesome!! Keep up the great content

[@BeABetterDev]

Thanks so much Tivo!

[@jamesmiranda5152]

great explanation !

[@BeABetterDev]

Thanks James!

[@a354681]

Thank you sir

[@abhishekkempanna8280]

thanks! One question: How can we send data with source as Azure Function(From Microsoft Azure) to AWS Kinesis firehose?

[@warrenb7450]

can we setup to invoke Lambda from the Transactions directly without going through SNS? and can we setup to load the Transactions data directly into Firehose? Thanks

[@pk_90]

Thank You for this very good overview 🤝👍 Can you please tell me, how good the job opportunities for AWS kinesis?

[@raghukumar6959]

Hi, can you let me know whether it is possible to move data from S3 bucket -> Kinesis -> Athena. for example Whenever data is added in bucket, Kinesis service should pick the data and move to Athena.

[@BeABetterDev]

Hi Raghu, Data that is in S3 can automatically be searched for using Athena. Why do you need Kinesis in this case?

[@yahbiamal3075]

@@BeABetterDev hi ,if we combine Athena with kinesis we get a quickly responce and the queries will be synchronous .is it right ?

[@krishnasanagavarapu4858]

Hello, love your content . Can you please create hands-on for the 2nd case where data is transformed using lambda before sending it to s3

[@BeABetterDev]

Hi Krishna, Good idea for a vid. I'll add this to my TODOs!

[@krishnasanagavarapu4858]

@@BeABetterDev Thank You 😀

[@sexyrexy6358]

Thank you for the content. In the specific transaction example, I didn't really get why we introduced Lambda into the equation just to put to Firehose, which in turns does a put to S3. Lambda can write to S3 directly. If the use case is for large streams of data that Lambda cant scale to as efficiently as Firehose, then once again, why can't Firehose read the data from the stream directly without Lambda as a middle man? Is it because Firehose cannot subscribe to a topic?

[@galeop]

so, as a Lambda function is needed to forward the message from SNS to Kinesis Firehose, what is the point of the latter ? Just to aggregate messages based on time/size criterion?

[@ralfrolfen5504]

Thinking the same, did you find an answer to your question?

[@galeop]

@@ralfrolfen5504 , the answer is "yes". Firehose is a kind of "ETL for streams" : it will "buffer" the stream (on disk, not in-memory), and cut it into chunks. The messages composing each chunk will be aggregated into a single file (JSON, CSV, Parquet, or other), and you may transform their data, or compute aggregations from those messages (eg: compute the average temperature for 1min, rather than list the temperature emitted by your IoT sensor every second). The goal is to : - aggregate your stream into chunks, that only contain data that is relevant to your analytics. Hence the idea of only sending to S3 a 1min average of the temperature, if you don't need a granularity smaller than 1 minute in your analytics. - transform the chunks of stream; for instance to standardise the way fields and values are structured (eg date in MM/DD/YYY format) across all your streams (as you may want to compare streams against each other during your analytics). - organise the chunks of stream in the storage destination, according to a key of your choice (eg date). The goal here is to enable "table partitioning" for better query performance. For instance you may organise in S3 the chunks of stream by date, so that they each file representing a chunk is stored in a "folder structure" organised by date in S3. Then when querying your stream with AWS Athena, if your query contains " SELECT * from MyStreamTable WHERE date > 03/03/2023", to execute this query, Athena will not have to load all the objects in S3 representing your stream: it will only read in S3 the objects that have a date greater than this date (and this is possible thanks to the way the "folder structure" was organized in S3).

[@ralfrolfen5504]

@@galeop Thank you! This is the longest answer I've ever got on youtube since... it launched. Thank you very much for the effort you put into writing all of it! Also: Very helpful example! This should be upvoted to land in the top!

[@galeop]

@@ralfrolfen5504 😂 thank you! 🥰

[@asdfasdfasdfasdf219]

Good video. Please put the AWS Service name at the same time as its logo. Must of us don't remember the name each each logo from AWS Services put the name at the same time

[@BeABetterDev]

Hey Ronald, great suggestion. I'll try to incorporate this in upcoming videos. Thanks for watching.

[@TheLostBijou]

Is it streaming processing or Batch Data processing?

[@professional6635]

Can we connect Transactions directly to Firehose or do we need a lambda in between ?

[@BeABetterDev]

Hi there, You can write directly to firehose using I believe putEvent API (or something similar) - Lambda isn't required.

[@ralfrolfen5504]

Question: From the explanation it sounds like Kinesis is doing nothing else but grouping message together. Is this really everything there is to Kinesis? It also seems like these kind of use cases could be accomplished with other services as well. E.g. store every transaction in a dynamo DB and read from there with a lambda function. I don't see the "wow, finally we have this service"-Effect.

[@reagan4417]

What if this processing will take longer than 15min, is it possible to use AWS Batch in the middle instead of a Lambda?

[@BeABetterDev]

Hi Naheed, I'm not too familiar with Batch. But I do know if your lambda function exceeds 15 minutes it will time out and abort the execution. Hope this helps.

[@ray811030]

It's possible to use kinesis as real-time computing framework like flink

[@BeABetterDev]

Hi yunrui, Kinesis actually supports flink as a managed service. You can read about it here: docs.aws.amazon.com/kinesisanalytics/latest/java/how-it-works.html

[@tacticalgaryvrgamer8913]

Stay tactical

[@BeABetterDev]

Copy that, Gary.

[@john-danson3113]

So 1gb [me] @ $0.50 on bare metal versus $3.00 per gb for kinesis. 6 times less expensive. Does anyone need a secure in-transit feed into s3 over direct connect at 1/4 times the cost of Kinesis?

[@leonstorey]

Huh? Kinesis Firehose is priced at (for first 500TB) $0.029 per GB + S3 storage at (for first 50TB) $0.023 per GB-Month. So that's a max of $0.52 per GB (1 month of storage), where's does the $3 come in?

[@krishnaramisetti7435]

damn sexy short explanation of apllication..

[@BeABetterDev]

glad you think so krishna :)

[@bhanuchirutha]

didnt understand any thing