Airflow DockerOperator: The Basics (and more 🤫)

[@dr_flunks]

it's actually super helpful that you display the folder/file structure of everything you're using as you go along. very well thought out!

[@anthonyloganhall]

This is how we have our environment setup and it works very well.

[@saritkumarsi4166]

Thanks Marc for the video on one of the operators I use extensively :)

[@joshuabodyfelt1239]

Wonderful job Marc! If I could amend this - would be awesome to have a followup video discussing the variety of different Docker registries, and how to connect to them with the Docker Connection.

[@aerobot6571]

Merci Marc, premiere video de toi que je regarde : c'est clair, utile et complet. Ben je vais voir les autres ;D

[@MarcLamberti]

Ben merci :)

[@PedroFernandes-n9v]

Docker Operator doesn't seem to work when I run airflow in docker containers (using a docker-compose) how can I fix this ? edit: the only solution i found was adding to volumes: - /var/run/docker.sock:/var/run/docker.sock and setting user: root instead of user: "${AIRFLOW_UID:-50000}:0" but apparently this isn't the safest way??? Anyone got a cleaner way to fix this ?

[@БогданБыховец-к4е]

I am trying to use DockerOperator but ger en error: PermissionError(13, 'Permission denied') I shoul set chmod 666 /var/run/docker.sock to avoid it. Can i use DockerOperator with chmod 660 /var/run/docker.sock?

[@ВикторБычков-д2з]

Please tell me, have you solved this problem? I have the same problem

[@chadoulimanel8527]

did u solved it ?

[@mbkhan1000]

I understand we can use the templating to pass variables/xcoms/connections to env variables in the docket container, but is there anyway to push values to xcomms from within a docket container? I understand that the process running in the container is isolated from airflow (unless it connects through the rest api?)

[@gregh6586]

Why can't you use `retrieve_output_path`? What exactly are you trying to do?

[@mbkhan1000]

@@gregh6586 trying to know if we can push to xcoms within a Docker operator task

[@yuricastro522]

If I'm using an airflow container to call another container, how can I mount volumes generated inside the airflow container to the other ? I'm getting errors trying this with this source parameter

[@lokeshkumar1365]

Could you make video on best practices for kubernetesexecuter on k8s deployment and different tasks can run parallel?

[@data-freelancer]

Hi sir, can this work on production like cloud composer?

[@PedroFernandes-n9v]

when running a DockerOperator on Airflow running in a Docker Container, the mounts have to be between the DockerOperator and the actual Host machine.. is there any way to avoid this? Can we create mounts betweeen the Airflow Container and the DockerOperator Container ?

[@Ayush_1908]

Hi Marc, is it possible to use dockeroperator for running java code on airflow? Or any better option?

[@abhishekacharya5069]

Hi Marc, Thanks for the video and it really helped me in understanding airflow. Actually I'm trying to pull the docker image by using dockeroperator on Apache Airflow. But wheneever I trigger the dag again and again, it pulls the image from docker hub. Can I save the docker image somewhere , so that whenever the dag is triggered it should not pull from docker hub. It will be very helpful, if you help me with this. Thanks

[@vaib5917]

Hi, I really need to know if we put the python script into a container and run it using DockerOperator, how can we pass the values of Variables from AirFlow Admin UI t the container ?? Please help.

[@mbkhan1000]

Templating into docker container environment variables

[@736939]

How to send data via XCom from DockerOperator? It there any better way than just print the values? how to run PythonOperator like script from DockerOperator let's say I want to run not the whole file but the function inside the file, how to do it via DockerOperator?

[@emanuelgiannattasio3366]

Marc, in your opinion, in which cases would it be convenient to use DockerOperator over PythonVirtualenvOperator?

[@MarcLamberti]

IMHO the dockeroperator is great as your run a docker image so you can encapsulate your task in it. That can help for testing and versioning. Otherwise, go with the python virtual env operator

[@ReenanOFC]

Is it possible to set different schedules based on tasks?

[@MarcLamberti]

Nop

[@danielpapukchiev3754]

split in multiple DAGs

[@trench6118]

With a branchpythonoperator you can - for example, I have some tasks which have changed from daily to hourly within a client DAG (all sources for that client are in the same DAG). What I did was add a function to check the execution_date.hour and if it was a certain time, I would return all extract task IDs. Otherwise, I would return only the hourly task IDs. The result is that my daily tasks are skipped each hour unless it is say 13:00 UTC, and my hourly tasks run each hour. It makes the DAG a bit messy though because of so many skipped tasks

[@kimted3272]

hello Marc, thinking of listening yout lectures on udemy. are there any lectures that covers kuberenetesPodOperator? think operartor reference is the closest, but asking just in case if u already have a video. thanks :)

[@eduardocarrerah3704]

is this a replacement for k8soperator?

[@MarcLamberti]

Nop

[@jagadishlucky1793]

Hi Marc, thanks for the videos it really helped me to understand airflow effectively. Actually, Iam trying to generate dynamicness in tasks creation. Based on the config parameter from UI the dag has to run the tasks. For Ex: If conf parameter has t1,t2 as true, there should be two tasks running. And if I increase the tasks in config params(t1,t2,t3,t4) it should run that many tasks. I tried multiple approached using operators,its not happening. Can you please suggest an approach? ---> Thank you

[@DhirajAvasthi]

getting this error requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused'))

[@vladdank9158]

Anyone figure out how to get this to work with Airflow itself running on Docker? Kind of lost. It's mentioned in the video around 8:03. I'm on Windows so it's kind of horrible LOL

[@PedroFernandes-n9v]

the only solution i found was adding to volumes: - /var/run/docker.sock:/var/run/docker.sock and setting user: root instead of user: "${AIRFLOW_UID:-50000}:0" but apparently this isn't the safest way??? Anyone got a cleaner way to fix this ?

[@PrakashReddyK]

Hi 👋

[@MarcLamberti]

it's been a while

[@slimebwoy]

I know this is 6+ months after this has been posted but I think your teach would be a bit more effective if you took the time with your hand writing. If I saw the diagram at 3:25 after listening to you talk about it, I'd have a hard time understanding what it means. Just wanted to provide a helpful tip!

[@MarcLamberti]

thank you for the helpful feedback :)

[@Klayhamn]

space at the end of the string as actual functionality determiner? who the hell designed that bullshit?