Synapse Espresso: Notebooks vs Apache Spark Jobs Definitions: which one should I use in Spark Pools?

[@avicool08]

Pls share learning video how we can use Spark Job Definition for Stream Analytics

[@EsteraKot]

great idea! thanks for sharing @Abinash Tumulu

[@aliceross9961]

watched the video thrice but did understand the concepts...thanks

[@shawnglover5150]

the spark job definition is so deep to understand

[@owenbolton7514]

how nicely you explained all the concepts!!good job

[@germanareta7267]

Great video. Thanks.

[@scottgipson3505]

thanks for introducing me to so many new topics

[@moeeljawad5361]

Thanks for this video, i am currently using the notebook activity in Fabric Pipelines. My notebook is mature now and it runs very well. I was thinking of gathering the code in the notebook into a job definition, for the sake of saving execution time in the notebook. Would replacing a notebook by a job definition makes the code execution faster? Another question would be about job descriptions themselves, if you have defined some helper functions in the notebook, can i move them to a side job definition that is being called from the main job definition? if yes then how? Thanks

[@patriciafuller721]

I never knew about the web interface till this video...

[@jaborahwade2498]

I figured out which tools to use

[@rosehudson9437]

that demo was needed to understand more precisely

[@keen8five]

Are there any any noticeable disadvantages concerning execution performance of one over the other? Does a scheduled Notebook take longer time until the actual execution of cells begins in comparison to Spark Job Definition? e.g. because Notebook need to initiate IPython interpreter?

[@EsteraKot]

If you compare the performance of the same code running under SJR or Notebook, there should not be a significant difference. But, that is right, that scheduled Notebook might take a bit more time due to pre-run cells.

[@pakkalibrandu]

John and Jane are characters of Tarzan X, she picked up those name in demo 😆

[@agammishra9674]

but there is notebook activity as well, what is the issue if we schedule the notebook activity in the pipeline, it can run automatically as well, please comment on this