Airflow XCom for Beginners - All you have to know in 10 mins

[@JavierBeneitoBarquero]

Quick and clear. With your Udemy courses and your KZbin videos, I'm learning a lot about Apache Airflow. From Zero to hero in a few weeks. Thx.

[@quanminfeng8375]

Best xcom video I watched

[@MarcLamberti]

Thank you 🙏

[@waodezalmawati460]

Whats a clearly explanation! Thankyou so much! Love from Indonesia!❤‍🔥

[@guolunli1908]

Thank you so much! By following your videos closely I not only avoided bugs but also learned in detail how to set up an airflow pipeline!

[@ХеллБой-я1щ]

thank you! as it gets very often, some detail is missed by me in order to use new functionality and that detail was ti thanks for explaining!

[@pyrojoke2]

Thank you! I spent a few hours trying to figure out how it works, but your tutorial did it in 10 minutes

[@anatolys9203]

great tutorial and explanantion! thank you!

[@luuquanghuy8193]

Thank u for your sharing, keep it up because u doing that right clearly and easy to understand. Love from Viet Nam

[@victortelles9670]

Amazing tutorial as always. Love yout channel.

[@guyfridman4426]

Thank you marc, I did the course in udemy and it's very recommended

[@MarcLamberti]

Thank you!

[@shantanubugadi3188]

Very well explained...! Very awesome...! I am just enjoying the series

[@sodiqafolayan4921]

Thanks Marc as always

[@MarcLamberti]

My pleasure!

[@АндрейРезниченко-п9ъ]

Hi all! Please suggest how to implement the following scenario: Have a dag with sshoperator that executes shell script. Shell script returns different return codes. How to analyze which return code is received??

[@dmdv-h1t]

thank you for clear explanation

[@yunusemrahulucay4953]

Thank you so much. It was really quick and clear.

[@kyuukev]

Nice video but is there a way to do it between 2 DockerOperator? And if not with the DockerOperator creating some kind of CustomDockerOperator to make it able to do it?

[@GustavoCosta-jr1mh]

Thanks for all your videos, Marc. Brilliant content, very straightfoward!

[@saritkumarsi4166]

Thanks Marc. Great info for a beginner like me :)

[@MarcLamberti]

Great pleasure, let me know what you would like to see next :)

[@saritkumarsi4166]

@@MarcLamberti It would be awesome if you can share some knowledge by taking this tutorial further by adding (if possible) a new service and assigning it a task via DAGs within the Airflow container. Just to give you some background on what I am currently trying to do --> Similar to your compose file, I have Postgres, Airflow Webserver and scheduler services defined in docker compose file and running perfectly. However, I have an extra service (say an Ubuntu server) defined in the same compose file which basically takes an .sh command and runs it. I am trying to send an sh command via DAG DockerOperator, to this 4th container. I took help from your video on DockerOperator as well :). If I understood correctly, the Airflow DAG in that video is running on the host instead from within a docker container. I searched for help on the internet, but haven't come across any. Do let me know if such use case is possible with airflow. Appreciate you help :)

[@padegalsaigiriraj3459]

Thanks, Marc. Great info Can You Please Confirm This I have Two Tasks in DAG1 Task1 - Normal Function Which does some process Task2 - TriggerDagRunOperator (Trigger DAG2) Can I Use the return value/XCom of task1 in Task2 and use this value in DAG2? Thanks

[@Jmignet]

Pretty enlightening! Thank you very much!

[@Jamdat33]

Excellent thank you

[@bigkongenergy6054]

Commander, the aliens continue to make progress on the Avatar Project. If we're going to slow them, we need to move fast.

[@marcuscheng9413]

ok. we're raiding the blacksite.

[@prashantjoshi4691]

add provide_context = True in task which uses ti in older version of airflow

[@thati2792]

can we pass xcom as params in redshiftsqloperator?

[@kipodiha]

Very useful video! after reading airflow documentation and example for me this XCOM was still not clear :( Now I m ok! Thanks a lot!!!

[@drakdragon]

Sir, I tried to do the needful to do good and nice Airflow in the XCom, but my hit chance was still saying 99% chance of to becoming the hit sir, but it still missed most of the time. Sir I do believe you should do the needful by improving the Airflow by using the Minecraft Mushroom Soup Windtunnel.

[@ramjan135]

Hi Marc, how to cross communication data (data sharing) between master_dag and child_dag(this is triggered by master dag)?

[@BULLSHXTYT]

How to access xcom in subdag from parent dag?

[@djalan84]

what about sharing data between dags?

[@MarcLamberti]

You can. Just specify the dag_id in the xcom_pull method.

[@wll9085]

useful!

[@antoniosaldivar4163]

I have the same xcom key from same task and it does not get replace or overwrite, instead it creates other xcom record with same key and task id

[@MarcLamberti]

look at the execution date which should not be the same. That's why you see that. XCOMs are created based on their DAGRun too

[@antoniosaldivar4163]

@@MarcLamberti thank you for the info!!, you are right the execution dates (YYYY-mm-dd, HH:mm:ss) are different, I see it has to be executed at the same time to overwrite that XCOM.

[@RaviPrakash-vu6hf]

Hi @@MarcLamberti I have the same issue. I do want to overwrite the value from the last run. I have a DAG that will run every 15 mins and it will push a key to XCOM. Since this key will be stored in the DB, the number of records will keep on growing. Hence, I want to overwrite the value of the key from the previous DAG run. please let me know if that is possible. Thanks!!

[@yerkhankabyl6284]

Does someone have the same issue like me? I got 3 “None” -> [None, None, None]

[@yerkhankabyl6284]

Oops, I solved this problem, the main error was the name of “Xcom” 😅 Thank u, man, u are awesome (:

[@BM-uf4pp]

The three people who disliked this video are nifi fans

[@marcuscheng9413]

or ADVENT troopers.

[@g3rzin]

Please don't read the underscore!!!! Just read it a "choose model" and not "choose underscore model"

[@MarcLamberti]

Okkkkkkkkkk!!!!!!!! Will do

[@g3rzin]

@@MarcLamberti ahahah great! And thanks for teaching me airflow!