How to Build an End-to-End ETL Pipeline with Airflow, Snowflake, S3, and Salesforce!

In this video, we'll go through how you can build your own ETL pipeline using Airflow! You can swap out any of these components for your specific use case, but the general concepts will still hold true!
Code:
registry.astro...

[@AlexZdanov]

Hey George, great stuff! I run a couple of websites that use a lot of data from a bunch of different APIs / CSVs / excel sheets that are published online on a regular basis, and it's generally not very "clean", in that it requires a lot of transforming, testing, combining, etc (so big emphasis on the T in ETL). I'm looking for a tool that would be able to handle all of that transformation, while also making it very easy to visualize and document where everything is coming from, how it's being manipulated and where it's going. For the latter part, everything is pretty much being stored in a single postgres DB, and I doubt there's any need for something bigger. Would you recommend Airflow for this? Are there any other tools you think would be appropriate?

[@thedataguygeorge]

I would highly recommend Airflow for this! Especially with the new Triggerer, it's really good for async consumption/transformation of that kind of data, and you can build DAG generators for adding new resources.

[@harizizzudin]

hi, can i do this project using aws free tier?

[@thedataguygeorge]

100p!

[@saket8431]

Good stuff🤌

[@thedataguygeorge]

Thanks so much! Let me know if there's anything else you'd like to see!