Get started with SPARK in Azure Synapse Analytics

[@bluedreamerssso]

need more of this, moving from AWS and Databricks to Azure Synapse Studio

[@ryanbowns1517]

Can u tell me the benefits of moving away from Databricks go Synapse ?

[@bluedreamerssso]

@@ryanbowns1517 cost, unfamiliarity with Scala/python, data flow model.

[@sakeeta6498]

@@bluedreamerssso I can understand that Scala/python issue, but is cost really a thing? I think you could make it with a reasonable cost in Azure using Data Factory pipelines that call (Azure) Databricks' notebooks and do all the ELT and data modeling you would need that way (getting data flow model) without using Synapse. -> Hard to believe that would be more expensive solution than using Synapse, but of course needs Scala/Python expertise.

[@nilsbuer]

Very good and easy explained thanks

[@natfind4724]

Great episode!

[@keen8five]

Concerning Spark Pool settings: I think they mixed up "auto scale" and "dynamic allocation"; these are two different things

[@germanareta7267]

Thanks for the video.

[@piraviperumal2544]

Loved it!

[@megapixelphotos338]

Started using this today, but noted that there wasn't so much in term of documentation. Can you recommend a good source of further reading?

[@jgowrri]

Is spark is good for DW? How this differ from sql dedicate ?

[@MsVikramaditya]

Can you give some guidance on choosing optimal sizing of spark pools and understanding DAG

[@kishlayamourya3141]

was that pyspark?

[@the_invisible__]

Python+ spark = Pyspark... The language that used in Spark cluster for data processing, transformation etc

[@ryanbowns1517]

Note sure how this is better than using Databricks in Azure. Can anyone shed some light here.

[@benjamincarter6095]

Both Synapse and Databricks use Spark. Synapse uses an open source version of Spark with built in support for .NET, while Databricks uses an optimized version of Spark that improves performance and allows for GPU-enabled clusters with higher data concurrency which improve processing performance. Synapse may be easier for a BI team to pick up, and Power BI can be used inside Synapse Studio. At this time, Databricks outperforms Synapse for ML. Microsoft is pumping R&D into Synapse, so it is worth watching.

[@sakeeta6498]

@@benjamincarter6095thanks for nice clarification. What do you think now, is it same thing in your opinion? MS putting a lot of effort on Fabric side, and on the other hand Databricks is constantly evolving...

[@baiganil5203]

Im 314 th person to view this