51. Creating a Spark Job Definition and Submitting it in Azure Synapse

[@bharathkumarreddy2834]

Your videos are helping a lot.....I was GCP Data Engineer...Just started with Azure Projects

[@WafaStudies]

Thank you. All the best 🤗

[@jmendez_enersinc7168]

Great explanation, better than official doc, thank you

[@WafaStudies]

Thank you 😊

[@rameshpeddine7123]

@@WafaStudies what is the use of this spark job defeination.we can directly execute whatever file in notebook?

[@ericz7881]

Great video thanks. How did you configure access to ADLS for spark job definition? I watched your video on configure access to ADLS (lesson 45). I've assigned blob data contributor role to the managed identity of synapse workspace, but the spark job definition run still failed with error :Access unauthorized on the provided ADLS Gen2 account. The strange part is that I could use the same code in spark job definition in a notebook, then run the notebook in a pipeline, which means the synapse workspace managed identity does have proper access to ADLS. Now I wonder - whether the spark job definition uses the same managed identity to access ADLS. Look forward to your reply. Thanks again.

[@annukumari9629]

Much appreciated ! Thanks for covering this topic

[@WafaStudies]

Thank you ☺️

[@5536sandy]

been a great series well done, so helpful

[@WafaStudies]

Thank you 😊

[@bchandra912]

Good, appreciate your work. If possible kindly continue POWER BI series.

[@auhumanmedium]

Hi Maheer, Thanks for sharing your knowledge. I have a question - Why we are using abfss path -- instead of url... ? Please help me in this.

[@roshankumargupta46]

Thank you. Any plan for azure devops playlist?

[@sakshiagarwal9596]

Can you please make video on dynamic creation of external table through serverless sql pool

[@ESWARKUMAR-t2t]

Hi mahi, will it be possible to schedule the spark jobs without using the pipelines activities.

[@parvathygm1791]

Hi , My spark job definition failed due to time out error when I debugged Can you please suggest a solution

[@alexfernandodossantossilva4785]

If we have a Python file that mention other files

[@devops840]

Hi Sir , your videos are very helpful . could you please create a video for working with XML file in Cosmos db with Azure Data bricks ?

[@narendrakishore8526]

1) There are 5 activities in sequential in adf pipeline. When anyone of the activity fails it should send email from activity 6 send mail web activity. All of them are connected in sequential order? 2) There are 5 csv files in source folder in blob. You have to copy only last modified date file to output folder in blob Please let me know exact answers for these two interview questions