5.2 - Airflow-Dataproc Integration | Apache Spark on Dataproc | Google Cloud Series

Apache Airflow is the tool of choice of Data Engineers for orchestrating large scale data pipelines and integrates with lot of tools such as Apache Pig, Apache Hive, Apache Pinot, Google Kubernetes Engine, Google Dataproc to name a few.
In this video we'll discuss the Airflow's integration with Dataproc and see how we can setup a simple workflow of creating a transient cluster, submitting a job and then deleting the cluster.
This video is part of the course Apache Spark on Dataproc. You can find all the videos for this course in the following playlist.
• Apache Spark on Datapr...
I regularly blog and post on my other social media channels as well, so do make sure to follow me there as well.
Sample DAG : gist.github.com/kaysush/ade06...
PySpark Code : gist.github.com/kaysush/65fdd...
Medium : / sushil_kumar
Github : github.com/kaysush
Linkedin : / sushilkumar93

[@soumyasourabh8799]

Huge props Sushil. The way you simplified the concepts, things just got slotted into my head. Great work, really appreciate it!

[@manishpathak6262]

Very good work bro!

[@limeraghu579]

Very good , you have god given talent use it well

[@swapnamohankumar6602]

@ Sushil Kumar, Thank you so much for this wonderful playlist on GCP... it will really help the beginners... Good explanation and end-to-end data pipeline flow is covered.

[@chetanhirapara7082]

Thank you so much Sushil kumar. You are doing awesome job for learner. It would be great and complete playlist if you will add installation of airflow on GCP

[@sahillohiya7658]

you are so underrated

[@sundarrajkumaresan8045]

create.a playlist for all the services and different DataFlow templates. Once again Thanks for the useful content!!!

[@user-fb1ue1fr8n]

You are doing a great job. These videos are very helpful. I had one doubt in this video, how is Airflow able to access dataproc and create cluster and fire spark jobs to it. Don't we need to add some kind of permission in dataproc to allow airflow access dataproc and also some configurations in Airflow?

[@ExploreWithArghya]

Hi sushil, I am following you videos of GCP . I have two doubts. First how do I set the job_id of dataproc jobs via airflow and second which is very important - how do I add 'additional python files' in dataproc job via airflow?

[@loke261989]

What permissions are needed for airflow to manage dataproc assets, can u pls explain

[@Amarjeet-fb3lk]

Nice video, But,how we can create dataproc cluster with gcs storage and sql metastore using this airflow operators? What if I need to read data from a gcs bucket and write to another bucket?

[@prachiagarwal9457]

How can I mark the Airflow task as failed when the spark job fails? The operator used to submit the spark job is DataprocSubmitJobOperator

[@Kondaranjith3]

air flow basics needed sir

[@etgcrog1]

where i put the archive? in bucket?

[@aldoescobar3973]

did u try spark serverless?

[@AdityaAlkhaniya_adi]

If possible can you please make a demo detailed video on cloud composer

[@kaysush]

Sure. I’ll add that as a separate video and post the link here. Thanks.

[@kaysush]

Hey Aditya, I’ve added the video on Composer. Please have a look and let me know if you have any feedback. kzbin.info/www/bejne/nWepnqWjnZ12aJI

[@etgcrog1]

@@kaysush thanks

[@etgcrog1]

How i can create the dag ?

[@kaysush]

The DAG is a python file. You put it on $AIRFLOW_HOMW/dags folder. Depending on how your Airflow instance is configured, it could either be a bucket (if you are using Cloud Composer) or a folder on filesystem. Watch my video on Cloud Composer to know more. Thanks

[@vikaskatiyar1120]

can you please share the github link for this example ?

[@kaysush]

Sample DAG : gist.github.com/kaysush/ade06ca3b4f42218f720e92e455c7b7b PySpark Code : gist.github.com/kaysush/65fdd9a5d5bb03a198d8fb1e23125bf1

[@kalyandowlagar3901]

@@kaysush Hi, In this tutorial is the airflow server outside the GCP if so how is the connection established when you switch from airflow to dataproc and from dataproc to vscode how are the connections established i know vscode can connect to remote host using SSH but how is dataproc and airflow standalone server connection is established