Installing and Running Airflow on a small AWS EC2 Instance

[@Orionalpha48]

Fantastic video. Got it up and running in no time. Simple yet comprehensive video that came out just in time to help me with my project.

[@MarcLamberti]

Happy to help

[@BD_warriors]

Great work Marc! Please keep it up

[@siddhisail7105]

This video is a blessing for me. Thank you so much for making this.

[@MarcLamberti]

Happy to help 🙌

[@noahfisher1955]

This video was such a tremendous help, thank you so much Marc :)

[@LoveisHell85]

Airflow on Fargate next please! Thank you so much for your work.

[@RAVIKIRANKANUR]

Simple and great explanation!!! Thanks a lot

[@AlexHitchon]

Hey great vid Marc was able to follow along just surprised we need postgres. I found this useful to get me started on my personal project of taking some data and writing to a SQL server db daily. Thanks

[@anirudhnegi4523]

Great work Marc. Helpful content!

[@kaushalkrishna5249]

Will webserver run even if we close terminal ? Also how will we run both webserver and scheduler in background even after closing terminal ?

[@ccc_ccc789]

Fantastic! Thanks Marc!

[@MarcLamberti]

Thank you!

[@AlDamara-x8j]

Hi Marc, again, thanks for your videos. After running airflow on aws ec2 instance, can I still follow your "Apache Airflow: The Hands-On Guide" Course on Udemy without any problems? What happens when I stop the ec2 instance for later resume? Do I need to recreate everything again from scratch? How can I avoid that?

[@hananenadi3744]

Hey great vid thank you very much, just i have a question The t2.small instance in EC2 is free ? Or should i pay sometimes at the end of usage ?

[@HarishMahale-y8n]

In cases where you are installing certain "latest versions" (e.g. airflow 2.5.0, python 3.10, etc.), can you mention if we should continue with those versions in the video or find out the actual latest version while installing. Will installing another highest latest version cause any issues with rest of the steps?

[@harishmahale1]

How do we keep the scheduler always running?

[@parashualamel6104]

Iam facing while mapping sql alchemy connection with postgres when i tried with what you said its not reflecting to correspong variable

[@ChinmaySathe-bx4ck]

You are a life save

[@harishmahale1]

How do we disable the example dags and reduce the clutter in web UI?

[@thghtfl]

Thanks for the video, Marc. It was unclear though how it differs from running on 4GB or 8GB instances because I didn't see any specific points on memory allocation. Also, I tried to run Airflow in a Docker container on a 2GB ec2 instance, but at some point, the instance stopped responding for some reason. I wasn't able to find out whether Airflow or Docker caused the issue, but for me, it didn't work.

[@alexanderpotts8425]

Thanks for this. I once got Airflow to start on a Micro instance with an external database 😂but once I installed the Databricks operators it was game over from a disk space POV

[@OmerNadler]

can you make it with docker ? and kub8 ?

[@javiercavieres2843]

Why when i hit the command airflow db init that don't create de airflow folder?

[@xmagcx1]

excellent example

[@GuidoPerez-z8f]

Hello Marc, i setup Airflow on ec2, using your video. I was able to successfully install it but I keep getting this error "The scheduler does not appear to be running. Last heartbeat was received 20 hours ago." How can i fix it. thank you

[@ronakshah6336]

Can you make video how should we do airflow for windows rather than ubantu

[@hjop010]

Hi Marc. Can you give an idea of montlhy costs for this setup? Thanks!

[@JeanBzh]

$0.023 per hour, so $201 per year ($16.80 per month) Prices vary depending on the region you select

[@GMedchess]

The tutorial works but I have a problem, The scheduler does not appear to be running. ... This happened after I close the shh connection, but the webserver is still active. I need use this airflow to schedule a periodic task what should I do ? ... How do I keep the scheduler active

[@dipeshg8581]

Sir please can you tell how to stop all of airflow components and restart them in one single command. so that changes can be made easily

[@findsarfaraz]

hey do you teach sqllite db in airflow in your udemy course. ?

[@0kazaki]

If I want to deploy airflow in a production environment with multiple workers for scalability, how should I do it ? I assume with kubernetes?

[@danielmawdsley157]

Great video! Have you considered making a terraform script so all of these steps could be deployed together?

[@MarcLamberti]

Nop, but if you wanna partner on that I would be happy 🫶

[@findsarfaraz]

in your udemy course have covered sparksubmit operator?

[@MarcLamberti]

not yet

[@IvanRmz27]

Do you have an azure equivalent video? Thanks!

[@cesarofuchi]

python3 -m venv venv is missing from the notion page

[@ML-bu7tu]

Hello ! Your video helped me a lot! What should I do if I want to modify the port number from 8080 to 8081 ?

[@guruprasadbs]

Hi, where are those dag files stored

[@jeanmarieabengzoa2600]

I have question. It is important to install postgresql before?

[@MarcLamberti]

yes

[@user-c3p0-zk1mr]

Awesome video. Is it possible to install airflow via cloud9 (it runs on ec2 and can be stopped anytime)?

[@MarcLamberti]

absolutely

[@jimaustin3608]

Anyone notice that the written command sequence is different from the tutorial verbal sequence? For example (4:25), ‘python3 -m venv venv’ is left out of the written seq; necessary to create the venv directory with the virtual environment. Then the lib install (sudo apt-get install libpq-dev) comes before the virtual environment activation (source venv/bin/activate). (don’t know if this is a problem) Since the EC2 instance works in the tutorial, might be a good idea to use it to ‘fix’ the written sequence … or just follow.

[@abdelhakemabdelhak4624]

merci

[@kikecastor]

Beautiful video, thank you! Two questions: If I shut down my computer, will the dags still run in the cloud? Can anyone enter with the public ipv4 dns 12:42 or can only I enter?

[@MarcLamberti]

correct

[@kikecastor]

@@MarcLamberti Awesome, thank you!

[@MrMadmaggot]

@@MarcLamberti Correct what?

[@UsrU-y8c]

@@MrMadmaggot shutdown pc - resources still run in the cloud. Can anyone enter... -Security is not my best friend, but this does not look secure at all to me. Anyone with the link can enter for sure.

[@MrMadmaggot]

How is it possible? How the dags can run if the Instance is shutdown?@@UsrU-y8c

[@EliaMartali]

Tried this with azure VMs and got some errors... I had to give up lol

[@duynghiavo7915]

Need many dollar to create airflow on aws

[@MarcLamberti]

Good thing you can have a free trial on Astronomer

[@nathanielsolomon917]

Encountered and issue with 10:16 airflow init db. Error Error : psycopg2.errors.InsufficientPrivilege: permission denied for schema public Solution that worked for me: ALTER DATABASE airflow_db OWNER TO airflow_user; Note I am using the latest airflow 2.9.1 as of this point

[@alpacajin6868]

tysm! ran into the same issue here

[@ember2658]

In version 16, I had to add: GRANT ALL ON SCHEMA public TO airflow; ALTER DATABASE airflow OWNER TO airflow; GRANT CONNECT ON DATABASE airflow TO airflow; GRANT USAGE ON SCHEMA public TO airflow; GRANT CREATE ON SCHEMA public TO airflow; GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO airflow; GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO airflow; to the SQL he created, its overkill, but its just a test