Install Airflow, Docker (Mac only) and run a Postgres DAG in 5 mins

[@bijosn]

If anyone gets stuck with trying to install the awaa docker image then remove the old clone from local then install ubuntu wsl and start the clone again and proceed in ubuntu wsl

[@souravtechie]

I think your comment applies to Windows users. This video was recorded on a Mac so I did not face any issues. Thanks for sharing!

[@MohsenFa-o2x]

where did you store the dag description file in a way that airflow can load it automatically?

[@souravtechie]

Hey! It's all in the dags folder. Any .py file in the dags folder will be automatically picked up by Airflow to render the DAGs

[@tacilio5046]

how to enable test connection button in airflow connections ? Which file do I change? I have already changed airflow.cfg and the change does not reflect in the UI

[@souravtechie]

Hi @tacilio5046, thanks for your question! You seem to be on the right track on this one. Adding `test_connection = True` to the `[core]` section of airflow.cfg file should do it! Please make sure you prune all old images in Docker Desktop (run `docker builder prune --all` from terminal). And rebuild the container from scratch (run `./mwaa-local-env build-image` from terminal). And then run it. So that the new configuration change is actually applied to the Airflow image. Test connection is disabled by default in the latest MWAA versions. It used to be enabled by default at the time I made this video. That's the reason why I did not have to make this config change at the time of making the video. If you still need help, please feel free to message me on IG @techtalksourav Good luck!

[@bijosn]

Hi, great video. may i ask which IDE you use and recommend?

[@souravtechie]

Thank you @bijosn! I use Pycharm. And recommend the same

[@bijosn]

@@souravtechie thank you Sourav

[@kofitelli6923]

what are you using to view the log

[@souravtechie]

The task logs can be viewed from the UI. The web server logs are in the terminal console in PyCharm

[@datalearningsihan]

How do you run two postgres server at the same time on the same port? AIrflow comes with a postgres and also you have already another postgres with the table you are working on. You didnt discuss those.

[@souravtechie]

Hi @datalearningsihan Nice observation and thanks for the question. I'm not running 2 Postgres servers on the same port and machine. The Postgres DB that I ran queries against is hosted in AWS RDS. You are correct, I should have mentioned that in the video. If you pause here kzbin.info/www/bejne/fZ6xYpijespngNE, you can notice that the hostname ends in rds.amazonaws.com

[@dheeraj_israni]

usename: admin password: test

[@souravtechie]

Spot on

[@datalearningsihan]

It does not really work. I tried to use it, nothing works.

[@souravtechie]

Hi! Sorry to hear that. What exactly did not work for you? Please elaborate and I would be more than happy to help you resolve the issue you're facing here

[@luismisanmartin98]

How can you have such a massive microphone yet the sound is so bad? The background music isn't helpful either

[@souravtechie]

I stopped using that "massive microphone" now. It is a actually bad. I'm very new to content creation, and still learning. Thank you for your patience!