Back to Basics: Building an Event Driven Serverless ETL Pipeline on AWS

To gain business insight, you and your data engineering team build and manage data pipelines to extract, transform, and load data. ETL provides the foundation for your data analytics and machine learning applications. You are able to apply business rules in ETL to clean and organize data to address business intelligence use cases such as monthly reporting. Join Lewis as he discusses modernizing ETL processes with an event-driven serverless data pipeline you can build on AWS.
Check out more resources for architecting in the #AWS cloud:
amzn.to/3qXIsWN
#AWS #AmazonWebServices #CloudComputing #BacktoBasics #Serverless #ETL

[@kalidsherefuddin]

The great work

[@ahsin.shabbir]

Really like this pattern. Events created by s3 added to sqs followed by downstream processing by any/all subscribers. Lambda is great for push based processing of the sqs events, and can invoke large data processing jobs using the message metadata. For example the lambda can invoke an ecs fargate task using a custom docker image containing all run time dependencies needed

[@Revanth2344]

Hey have you implements above event drive with ecs instead of glue?! Please let me know if so

[@ahsin.shabbir]

@@Revanth2344 yes...

[@patrykbindacz4333]

Great vid

[@amazonwebservices]

👍

[@liiilliil3297]

Why would we use s3 over a rds?

[@awssupport]

Hi. 🙂If you'd like, you can learn more about Amazon S3 and Amazon RDS using these links: go.aws/46yjBNn and go.aws/3LJEtJA. 📚 This is also an excellent question to share with our community of experts over on re:Post: go.aws/aws-repost. 📬 ^SA