Build a Real Time Data Streaming System with AWS Kinesis, Lambda Functions and a S3 Bucket

Support the channel plz 😊: www.buymeacoff...
Send emails in lambda: • Automate Sending Email...
Save data to database: • Build a CRUD Serverles...

[@FelixYu]

Video on send emails in lambda: kzbin.info/www/bejne/o32QZYObqqaojcU Video on save data to a database: kzbin.info/www/bejne/i6WYdJ6Jr5uBiJI

[@IRFANSAMS]

Can you make a video on Credit card fraud detection and entire transformation from capturing streaming data , Historic data to uploading in s3 and for prediction insagemaker

[@nandyalasurparaju2834]

I learned a lot of real-time use cases of AWS services from your channel. Thanks! for your efforts and please keep doing Felix.

[@FelixYu]

Glad that u found my tutorials helpful!!

[@saltdomeguy]

Glad I found you. I've been looking for some walk-thru projects. Keep em coming.

[@totachava6235]

thank you for writing the code step by step and explaining it, its really helpful for beginners :)

[@FelixYu]

Glad that u found it helpful :)

[@rahuldeo5813]

This is a great tutorial, previously, I just have basic idea about AWS stuffs, now with this tutorial, I learned the Kinesis data streaming part very easily. which can help me with my upcoming Interviews. Thanks a lot :) Felix.

[@FelixYu]

You are welcome!! Best of luck with ur upcoming interview :)

[@sentinelese6717]

Great stuff and awesome explanation. Throughly enjoyed.

[@FelixYu]

Glad that it helped 👍

[@shubhayanai8981]

Code for producer : const AWS=require('aws-sdk') AWS.config.update({ region:'us-east-1' }) const s3=new AWS.S3() const kinesis= new AWS.Kinesis() exports.handler = async (event) => { console.log(JSON.stringify(event)); const bucketName = event.Records[0].s3.bucket.name; const keyName = event.Records[0].s3.object.key; const params = { Bucket: bucketName, Key: keyName } await s3.getObject(params).promise().then(async (data) => { const dataString = data.Body.toString(); const payload = { data: dataString } await sendToKinesis(payload, keyName); }, error => { console.error(error); }) }; async function sendToKinesis(payload, partitionKey) { const params = { Data: JSON.stringify(payload), PartitionKey: partitionKey, StreamName: 'mydatastream' } await kinesis.putRecord(params).promise().then(response => { console.log(response); }, error => { console.error(error) }) } Code for Consumer: exports.handler = async (event) => { console.log(JSON.stringify(event)); for (const record of event.Records) { const data = JSON.parse(Buffer.from(record.kinesis.data, 'base64')); //send emails clients, publish the data social media console.log('cosumer #2', data); } };

[@riteshbehal1019]

perfect men :)

[@boypotus]

One thing that will improve content is explaining maybe each concept as if hovering over it in vs code // where it explains things ie: say stringing then give a small summary for it because everyone won’t know these things

[@reeturajchatterjee]

You saved my Day! Thanks a lot Felix.

[@FelixYu]

np..Glad that it helped :)

[@techdesk915]

Nice content Felix : ) love to see more of these projects. it helps learn a lot : )

[@FelixYu]

Thank you thank you :)

[@bindureddy6148]

Hi Felix Yu, You are videos are clean, precise, and very informative. Can you make a video on Kafka?

[@Namaryop]

good quality video, keep up with the good work!

[@FelixYu]

Thank you!! :)

[@BlackSk8ter100]

Good stuff Felix. Keep going.

[@FelixYu]

Thank you thank you :)

[@mehmetkaya4330]

Thanks a lot for another great tutorial!!

[@FelixYu]

Glad you liked it!

[@AS-zw4lk]

Thanks for your content. It is really helpful.

[@rashigupta5995]

Thank you Felix !

[@kristofferhaukaasberg6258]

This is great! Thanks so much, Felix! I was looking for a simple tutorial that scratch the basics of all these fragments in AWS, and also in Javascript - so this was spot on. I'm building a service that posts messages to a websocket handler (Lambda function) and I want to save that information coming from the websocket (basically health data from an iot-device). So from there, AWS Kinesis seems like a good idea to collect all incoming data from these devices, and then consume the data into a ie. DynamoDB. BTW: Are you open for freelance work with small AWS-based projects?

[@FelixYu]

Thanks man!! I appreciate the offer but I don’t think I have time for freelance work atm!!

[@anandchiluka7611]

Nice video, thanks for the detailed explanation

[@FelixYu]

Glad that it’s helpful!!

[@Karthik_I_Am_]

Im glad that i found you and your videos really help me and many like me. So my request would be AwS Pyspark with Big Data pipelines and Aws Dynamo DB. Please consider this and thanks in advance. Much appreciation!

[@sagar1689]

Thank you for this video. Very helpful 👍

[@FelixYu]

Glad that it’s helpful!!

[@abelcarvajalgil6705]

great demo!!

[@FelixYu]

Glad that u found it helpful 👍

[@rajibmahato2608]

awesome sir

[@muditmishra9908]

Hi Felix, does in real time system, is it common to have get data first in to s3 and then produce to kinesis data stream using lambda ? Can you please make a video on different type of producers and which one to choose ? I want to know how to decide the producer if we are designing real time system say with latency of milliseconds and also for near real time like having latency of say in minutes to hour. Thanks

[@lavakumar5181]

very clear explanation...please try to do lambda functions on python too

[@FelixYu]

Thank you!!

[@pohboonxin4137]

Hi Felix , It is a nice and clear video. One question, may I know if there can be applied on other types of sources data such as pdf, jpg? Can I add a aws textract in between to read the text from document or image rather than just text file ?

[@FelixYu]

yes..in 5:17 of the video, if you leave the Suffix field blank, u can upload any file types and it will trigger the producer lambda

[@pohboonxin4137]

@@FelixYu Yes, it will trigger the producer lambda. Could you show something like using textract and convert other formats of source data into csv ?

[@_Documentation]

💯

[@monuramadutta454]

Good work

[@nainaarabha9186]

Can we directly send the data from lambda functions to consumers? If yes, tell me the cons

[@kpashupati]

I am great fans of your aws videos.Kindly help me to understand how one lambda is calling another lambda and also updating 2 diffrent databases..Like product is purchased by customer and also updated in product inventory. In node js please😊

[@FelixYu]

u have to like all my videos before i answer this question......just joking lol there are 3 ways u can enable one lambda to call another lambda: 1. use a "middle man" like kinesis or sqs. lambda #1 construct a data object send it to kinesis/sqs and then it triggers lambda #2. lambda #2 then extracts the data object and do whatever u want with it 2. attach an alb (application load balancer) to lambda #2 so it acts as an API and then lambda #1 calls the alb to invoke lambda #2 3. lambda #1 calling lambda #2 directly using the invoke method. here is a reference to that (www.sqlshack.com/calling-an-aws-lambda-function-from-another-lambda-function) i personally like the first approach the most but all 3 would work!! for ur second question, u can just perform 2 dynamo requests (one per each table). just change the dynamoTableName in the request params (e.g., github.com/felixyu9/serverless-api-tutorial-code/blob/main/index.js#L80-L83)

[@kpashupati]

@@FelixYu thanks lot...I have successfully implemented ..Actually I was trying to implement ticket reservation system..While booking the ticket it will also minus the seat from the main table also keep booked seat no in users table... Thanks again..I am just having 1 month of exp in AWS.

[@FelixYu]

Nicee..glad that u figured it out and good luck with the aws exploration 👍

[@rashigupta5995]

Can you share the difference between using apache kafka and kinesis?

[@frankyan8135]

thank you

[@FelixYu]

Glad that u found it helpful

[@igordatsenko5283]

thanks!

[@FelixYu]

glad that u found it helpful 👍

[@Overbound]

thanks

[@FelixYu]

👍

[@ganeshnayak4179]

Can you please make a video with Python instead of Node JS. Though the content was quite explanatory. Thanks Felix for awesome content

[@FelixYu]

I can write the code in python and then push it to GitHub when I get a chance. All other setups will be the same!!

[@evanserickson]

Could you also use step functions and put it all in one?

[@2mahender]

Can you do one video on Athena,Glue,and EMR

[@RedCloudServices]

Is there a NodeJS function that connects to a remote JMS ConnectionFactory endpoint for use inside the vpc?

[@samyaktjain698]

Hi Felix, I need producer for CLICKSTREAM then what is the changes we need in the lambda code? Please correct me If I am wrong.

[@dotan2211]

Hi Felix, I have question that can we modify the data received from S3 bucket or API gateway, then save it to DynamoDB by using Lambda Function

[@FelixYu]

yes u certainly could..for example, in consumer 1 lambda, u can read in the data as json/strings/numbers and then u make changes to it to make it a different (e.g., numeric calculation, string concatenation, etc.) and then u can save this new data to dynamodb

[@weilichen2961]

Hi Felix, love your video, if it is possible to be Python code in lambda? Thanks

[@FelixYu]

Yea u can write the lambda in python as well

[@2mahender]

nice, hi sir, can these lambda functions usefull for Video,audio files also?

[@FelixYu]

i think so. u just needa find a lib that handles video files and add the logic in the lambda code

[@khandoor7228]

Felix loving your channel, great content! Can we do more with Lambda functions? They seem to be an important link to many other services in AWS. For these 2 consumers maybe send an email with 1 and save to dynamo with the other?

[@FelixYu]

tyty 😄 and yessir, we can certainly do more with lambda functions!! lambda has gained a lot of popularities in recent years becuz it is light weight, easy to set up and great for horizontal scaling. u just reminded me that i do already have videos for send emails in lambda and saving data to dynamo database: send emails in lambda: kzbin.info/www/bejne/o32QZYObqqaojcU save data to database: kzbin.info/www/bejne/i6WYdJ6Jr5uBiJI

[@darshithaindupally1515]

I am getting errors like "Cannot read property '0' of undefined", and "event. Records is not iterable", can you rectify the error

[@nishantshah4718]

where does cloud front comes into picture?

[@pradeepsharma9035]

Hi Felix what if I have s3 in one aws and kinesis on another . And if I added cross account policy on lambda as well. What should I do in lambda to send the data to kinesis on another aws account.

[@FelixYu]

if u have the producer lambda in account A and then wanna write data to a kinesis stream in account B, u will needa configure 2 roles. 1. in account B, create a role that has write access to the kinesis stream (e.g., putRecord, PutRecordBatch, etc.). let's call it kinesis-role-in-account-B. 2. in account A, create a role (let's call it lambda-role-in-account-A) and use this role to assume the kinesis-role-in-account-B that was created in step #1. attached this role to the lambda and it will be able to send data to kinesis in account B!!

[@pranayshukla9980]

My use case is Amazon connect data -> kinesis data stream -> delivery stream lambda -> se can you help me with lambda heree

[@parikshithshivaprakash5523]

this is same as s3 streaming files right

[@kavyap3184]

Hello Felix, Thanks for the content. Is it possible to have the Json file (Source with JSON format) template from s3 to Kinesis?.

[@FelixYu]

yes, just change txt to json in the s3 trigger suffix 5:18 of the video and u can upload json files as the data source (e.g., test_file.json) (or just leave that field blank so it will take all file types). let me know if that works

[@kavyap3184]

@@FelixYu Hi Felix, Thanks for the quick response. I have tried with changing the suffix of the file in S3. But this does not work either .

[@FelixYu]

@@kavyap3184 gotcha....lets try a few things here. 1. when u have the txt suffix set up and upload a txt file (the exact setup i have in the video), does everything work fine? 2. after u change txt to json and upload a json file, does it trigger the producer lambda? if not, it might be a permission issue. 3. if it triggers the producer lambda, lets console.log out dataString and see how it looks like (9:29 of the video)

[@kavyap3184]

@@FelixYu Thanks for the suggestion. It is just that my file is too big to process. Also do we see the records in the AWS data stream ?. Because specific data stream monitor shows no data even though i see the data in console.

[@FelixYu]

Yea if ur file is big, u needa increase the producer lambda timeout and memory....and I don’t think u can look at the data from aws console. Under monitoring u can see metrics like incoming data in bytes, get record counts, etc

[@sormagupta9170]

I am getting error "stream/kinesis-stream because no identity-based policy allows the kinesis:PutRecord action" while porsting data to kinesis stream

[@FelixYu]

Make sure u have the kinesis policy for ur iam role (1:55 of the video)

[@Roop-tk2rb]

Please advise where can we find the source code for this example

[@FelixYu]

sorry i dont think i keep the code after i uploaded the video :(( try to pause the video and copy the code that way. let me know if u run into any problems!!

[@victoriabressan4557]

I want to stream but to a event bus. I couldn't find a way yet...

[@austinchettiar6784]

Can i get the same code in python pls?

[@subhamchakraborty6822]

Hi, is it possible to generate a real-time drill rig data stream by AWS Kinesis?

[@FelixYu]

Yes, u would have to build a data producer that process the drill rig data and then sends it to kinesis thou

[@subhamchakraborty6822]

@@FelixYu Thanks! Is it possible to build a data producer via AWS?

[@FelixYu]

@@subhamchakraborty6822 i guess a better question is - are u able to connect aws with ur drill rig system cuz idk how u have ur drill rig system set up and where u store the data

[@subhamchakraborty6822]

@@FelixYu I am planning to build the drill rig via MATLAB Simulink. But I don't know it's possible to connect the system with AWS or not.

[@FelixYu]

@@subhamchakraborty6822 the easiest way i can think of is that..after matlab saves teh data to ur machine, u can write a script/cron job/program to upload the data to aws s3 and have that trigger a lambda function to send the data to kinesis

[@priyaavhad3688]

Hi need the producer code can you pls help

[@mohammedshihab8577]

Video was very good to understand, would you mention your mail id I had some queries related to iot service

[@saswanthikaschannel7708]

I am getting an error like cannot read property of 0 of undefined

[@FelixYu]

its prob caused by a typo....make sure it is event.Records[0]

[@suresh1088]

Link for the code ?

[@asifhossain1874]

Please upload the code in Python here Sir...

[@abstractionGod]

Oh nvm lol

[@ALDARPRATHAMESH]

there's error that says "The specific log group: /aws/lambda/consumer1 does not exist in this account or region."

[@randomvideos6117]

Consumer1 - send to elastic search

[@stevebalu]

got The specific log group: /aws/lambda/Producer does not exist in this account or region. errors. can't see the triggered logs

[@FelixYu]

two things to check here: 1. make sure the cloudwatch console u are viewing in the same region as the lambda function 2. make sure the iam role the lambda is using has the CloudWatchLogsFullAccess policy attached to it