Synapse Espresso: Notebooks vs Apache Spark Jobs Definitions: which one should I use in Spark Pools?

Welcome to the 34th episode of our Synapse Espresso series! In this video, we will demonstrate and explore the differences between using notebooks and Apache Spark Job Definitions as authoring options in Synapse Analytics. We will cover the advantages and disadvantages of each approach and provide guidance on how to choose the right option for your use case.
We also highly recommend reviewing our blog post, which provides more in-depth coverage of all the details and decision factors when choosing between Spark Notebook and Spark Job Definition. Check this link: techcommunity....
Stijn Wynants - Fasttrack Engineer
/ stijn-wynants-ba528660
sql-stijn.com/
/ sqlstijn
Estera Kot - Program Manager
LinkedIn: / esterakot
Twitter: / estera_kot

[@avicool08]

Pls share learning video how we can use Spark Job Definition for Stream Analytics

[@EsteraKot]

great idea! thanks for sharing @Abinash Tumulu

[@owenbolton7514]

how nicely you explained all the concepts!!good job

[@shawnglover5150]

the spark job definition is so deep to understand

[@aliceross9961]

watched the video thrice but did understand the concepts...thanks

[@germanareta7267]

Great video. Thanks.

[@scottgipson3505]

thanks for introducing me to so many new topics

[@moeeljawad5361]

Thanks for this video, i am currently using the notebook activity in Fabric Pipelines. My notebook is mature now and it runs very well. I was thinking of gathering the code in the notebook into a job definition, for the sake of saving execution time in the notebook. Would replacing a notebook by a job definition makes the code execution faster? Another question would be about job descriptions themselves, if you have defined some helper functions in the notebook, can i move them to a side job definition that is being called from the main job definition? if yes then how? Thanks

[@patriciafuller721]

I never knew about the web interface till this video...

[@jaborahwade2498]

I figured out which tools to use

[@rosehudson9437]

that demo was needed to understand more precisely

[@keen8five]

Are there any any noticeable disadvantages concerning execution performance of one over the other? Does a scheduled Notebook take longer time until the actual execution of cells begins in comparison to Spark Job Definition? e.g. because Notebook need to initiate IPython interpreter?

[@EsteraKot]

If you compare the performance of the same code running under SJR or Notebook, there should not be a significant difference. But, that is right, that scheduled Notebook might take a bit more time due to pre-run cells.

[@pakkalibrandu]

John and Jane are characters of Tarzan X, she picked up those name in demo 😆

[@agammishra9674]

but there is notebook activity as well, what is the issue if we schedule the notebook activity in the pipeline, it can run automatically as well, please comment on this