2.2 - Submit Dataproc job via Cloud Console | Apache Spark on Dataproc | Google Cloud Series

In this video we'll see how to submit a Spark job on Dataproc cluster via Google Cloud Console.
Spark Job Driver : gist.github.co...
This video is part of the course Apache Spark on Dataproc. You can find all the videos for this course in the following playlist.
• Apache Spark on Datapr...
I regularly blog and post on my other social media channels as well, so do make sure to follow me there as well.
Medium : / sushil_kumar
Linkedin : / sushilkumar93
Github : github.com/kay...

[@anshikayadav5718]

Great learning!!

[@naveendayyala1484]

why we adding jobs every time here we can not use this cluster as we use in databricks and what is the platform we can use to run our query like we run in databricks.

[@sharadshrivastava4285]

Hi Sushil, I am getting error while submitting the dataproc pyspark job. Job failed with error task was not acquired. On Google, I found that to this may be due out of memory or network issue. Non of my pyspark job run . Do you know how to check the network issue here? Master and worker nodes are showing up and running on gcp console

[@kaysush]

Hey Sharad, did you check in the YARN UI if all the nodes are healthy and have resources ? Is this a shared cluster or exclusive to the job you are running ? This error usually is because of YARN being unable to provide required resources. Also check your per executor configuration is not more than a single node (e.g. an executor is asking for 10GB memory but you are using smaller nodes). Let me know if this works. Thanks.

[@parasbansal6069]

How to add the python file on the mentioned path?

[@kaysush]

Its at the start of the video. We are using gsutil to put file in the bucket.

[@parasbansal6069]

@@kaysush Got it. Can I copy the spark py file from github to GCS directly?

[@kaysush]

@@parasbansal6069 I don’t think you can. You’ll have to download that file.

[@neha-fp6kp]

@@parasbansal6069 u can clone the git repo and then copy paste the git file to a bucket

[@duskbbd]

Better you should show how to see spark ui dag after job submission with help of dataproc and writing logs to gcs buckets @sushilkumar