Big data in Airflow? Discover the XCom Backends with AWS S3!

[@darsh_shukla]

Marc you always surprise me. 🙏

[@MarcLamberti]

I do my best 😉

[@MrIgorbpf]

Great video man!!! Thanks a lot! I've been struggling to deploy airflow 2 on AWS ECS. Can you make a video of it? I believe other people will enjoy because deploying airflow is tough!

[@fahadshoaib8735]

Hey man nice video but I have one query when the extract task shares the dataframe into the process task the dataframe becomes None and we are not able to perform any pandas operations on it. What's the use of sharing info between tasks if it not usable.

[@manqobadlamini2207]

Brilliant video as always! Thanks.

[@candyskullxoxo4660]

Hi, i love your vids. Can you show how to integrate minio xcom backend in Airflow running on kubernetes? Do i need to modify the Pod executer?

[@sunilpotu6028]

How can we use xcom between two SparkOperator's. In spark command I am generating value in python file so want to use in sencond SparkOperator python value?How can push value to xcom in spark submit?

[@luiztauffer8513]

thanks for the great content! wouldn't it be safer to store the secrets as ENV variables and retrieve them when needed?

[@shubham_chourasia]

How to achieve this in AWS MWAA environment?

[@AbdurrahmanKocukcu]

Awesome :)

[@chetansurwade]

Low or apparently no documentation for Azure Blob and GCS custom xcoms.

[@MrQsam]

very useful! ty!

[@pavelpetkun5269]

Merci beaucoup!

[@murugangan6817]

Xcom with mssql

[@Arnob_111]

You skip some things in your video such as while creating an IAM user when need to create a new policy you opened a new browser tab for IAM and created the policy and then came back to your user and attached the policy. These little things may look unimportant but makes it hard to follow you.

[@SoumilShah]

I sent you messages on LinkedIn through email Tried contacting you no reply sad to say all you are doing is just for views