AZ-900 Episode 15 | Azure Big Data & Analytics Services | Synapse, HDInsight, Databricks

Microsoft Azure Fundamental full course.
Skills Learned
- Describe Azure Big Data & Analytics Services such as
- Azure Synapse Analytics
- Azure HDInsight
- Azure Databricks
Study Materials: marczak.io/az-900/#ep15
* Episode Resources
* 📚 Study cheat sheet marczak.io/az-900/episode-15/...
* 🧠 Practice Test marczak.io/az-900/episode-15/...
* Study Guide
* Microsoft Azure: Azure Synapse Analytics azure.microsoft.com/en-us/ser...
* Microsoft Azure: Azure HDInsight azure.microsoft.com/en-us/ser...
* Microsoft Azure: Azure Databricks azure.microsoft.com/en-us/ser...
* 🌐 Wikipedia: Big Data en.wikipedia.org/wiki/Big_data
* Expand your knowledge - extra resources
* Microsoft Documentation: Azure Synapse Analytics docs.microsoft.com/en-us/azur...
* Microsoft Documentation: Azure HDInsight docs.microsoft.com/en-us/azur...
* Microsoft Documentation: Azure Databricks docs.microsoft.com/en-us/azur...
* 🌐 Apache Spark Homepage: Overview spark.apache.org/
* Azure Databricks Tutorial | Data transformations at scale • Azure Databricks Tutor...
Agenda
00:00 Episode introduction
00:30 What is big data?
01:42 Azure Synapse Analytics
04:15 Azure HDInsight
05:27 Azure Databricks
09:24 Summary of Big Data services
Want to connect?
- Blog marczak.io/
- Twitter / marczakio
- Facebook / marczakio
- LinkedIn / adam-marczak
- Site azure4everyone.com

[@markcaneda7764]

The channel name says it all, very intuitive visualizations and explanation. This is much better than udemy paid courses. Thank you for creating such contents!

[@AdamMarczakYT]

Wow, thanks! Appreciate your kind words Mark :)

[@idougy18091809]

It's even better that Ucertify which a course leader was teaching from.

[@kavinyudhitia]

Let me also add... this is even better than Coursera courses (that was created by microsoft themselves).

[@koduruchandu]

@MarczakIO Hi, this is Chandra from Bangalore, India. Today i have successfully cleared AZ-900 exam by just going through the 37 youtube vedios you prepared. In spite of having prior fundamental knowledge on cloud computing, i could pass the exam with the help of your vedios on Azure. Hence, would like to thank you a lot 🙏 for making available such a great vedios which anyone can easily understand the Azure basics needed for exam.. Is a great social service..so nice of you😊

[@AdamMarczakYT]

Thank you, highly appreciated :) Glad you passed, keep it up!

[@BijouBakson]

Here I am, 2 years later and now an Admin associate, preparing for my Solutions Architect Expert AZ-305 exam... and still not entirely clear with Big Data. I am having to do the Data Engineering module to deal with it once and for all. These videos help clear things out. Thank you Adam.

[@oguzhankrcal357]

Today, I've joined to Microsoft Azure Virtual Training Day: Fundamentals meeting. It has been finished yet and I missed you. ☺️ I came to here by running. Thanks man. Your English is understandable, your blackboard is awesome. They should learn how to present AZ-900 from you.

[@AdamMarczakYT]

Wonderful! Welcome back to where all the action is! :D

[@PawanKumar-tu6ti]

Here we meet again, Adam! I passed my exam yesterday of az-900, thanks to Microsoft documentation and especially YOU, and your teachings.

[@AdamMarczakYT]

Fantastic! congrats, keep up the good work :)

[@PawanKumar-tu6ti]

@@AdamMarczakYT thanks, can you start making azure series on az-204? At least make a roadmap and prerequisites for it.

[@abhisheksoni6012]

Thank you so much Adam for making these difficult-to-grasp concepts very easy to understand and interesting to learn. There are no words for your teaching skills. You are awesome. Thanks a lot Adam !

[@sumitprakash6840]

You are simply fantastic. I really love the way you describe each topic. Thanks for support and looking ahead to see you more to take advantage of new technology.

[@ilyasontube]

all of the Adams videos are not only very clear and informative but also very inspirational, real MVP!

[@AdamMarczakYT]

Great to hear! thanks!

[@benediktemydlandsolend8515]

Thank you for your explanations. Love the way you illustrate and show us examples. Looking forward to your next video.

[@AdamMarczakYT]

More to come! thanks!

[@simonkelso3447]

Great Job Adam, this is forming the basis for my Azure learning. Keep up the good work !!

[@AdamMarczakYT]

Great to hear! Will do!

[@ngophuthanh]

Thanks a lot, Adam. As always, this episode is very well organized and presented. Diagrams and visualizations are excellent. Your deep knowledge and teaching skills are amazing.

[@AdamMarczakYT]

Many thanks! :)

[@mmohan1234]

Excellent video on Azure's Big Data and Analytics products.

[@alexluke_again]

the least i can do i like each video and recommend to friends. thank you for doing this!

[@spencerthomas9377]

Great stuff as always!

[@aurosish1]

Thanks to you Adam, now I got a quick overview of Azure Big Data & Analytics Services.

[@AdamMarczakYT]

Fantastic! Thanks!

[@SanatanYadav108]

Hi Adam, you are fantastic mentor. Your presentation is extremely interesting. Thanks alot for your hard work behind these presentations.

[@KDOERAK]

another great overview!

[@binhngo12336]

Adam. u are the king. best teacher I ever seen man. all the best for u man...

[@timmusaka9577]

The best content is on KZbin!

[@vitovito123]

Adam, thank you so much. I passed my exam today.

[@thesloable]

Thanks for creating concise and easy to understand materials. Hope to see more courses covered in this channel.

[@AdamMarczakYT]

Thanks, will do!

[@jcb3883]

I'm getting my certificate this friday. Wish me luck. Thank you for doing this series.

[@franciscorossi5633]

You are a genious, thanks mate!

[@prititiwari5183]

You explained every topic thoroughly.

[@AdamMarczakYT]

Thank you! :D

[@davidgates5189]

great as always

[@csabafeher2632]

I cannot describe how useful, professional and engaging your content is! Best of its kind! Btw, after couple of episodes I have realized you have cool T-Shirts! ;-)

[@AdamMarczakYT]

Wow, thanks! :)

[@scotmarkieboy]

Hey Adam, thanks for putting this content together and making it available for free.

[@AdamMarczakYT]

My pleasure!

[@ruchijain2584]

Thanks for amazing videos !!

[@AdamMarczakYT]

My pleasure!

[@Lordxfx]

Finally an insight to HDinsight :)

[@AdamMarczakYT]

😄

[@jvstechtips]

Very helpful, thanks

[@talentdlamini9326]

you a life saver thanks Adam

[@AdamMarczakYT]

Thank you! :D

[@shaunakv100]

Thanks a ton for making this videos. You have not only simplified all concepts for better understanding but also kept it interesting till end. Your presentations, diagrams are also so clear. I learned a lot from your videos. I will be happy if I can support you in someway Thanks Again for helping us with these videos

[@AdamMarczakYT]

Awesome, thanks!

[@farshidjamali]

Thank you, Adam!

[@AdamMarczakYT]

My pleasure!

[@sanghvian]

I am learning so much apart from just Azure !!! Thanks a lot Adam !

[@AdamMarczakYT]

My pleasure!

[@clintonchristopher6532]

Hi Adam, been following back to back on this video tutorial as i am about to take the azure fundamental exam, and i must say its been extremely helpful. Thanks so much ...

[@AdamMarczakYT]

Fantastic!

[@stephaniebuadu3474]

This was great! Thanks

[@AdamMarczakYT]

Glad it was helpful!

[@denkozlov4220]

thank you Adam

[@gauravkumar796]

Awesome.. I like the summarised comparison.

[@AdamMarczakYT]

Thank you! Appreciated :)

[@swatikandarpatel1127]

Thank you I cleared exam today.

[@vinayinfo15]

Nice cover of az 900 study tipics

[@saltnlightmalaysia]

Wonderful material!!!

[@AdamMarczakYT]

Thank you! Cheers!

[@jhosnatas]

I've been aproved in az-900 thank you

[@AdamMarczakYT]

Congrats :) Keep it up!

[@JagdishJ666]

Outstanding explanation and easy to learn

[@AdamMarczakYT]

Thanks! :D

[@thekbee5736]

The explanations in your videos are the easiest to understand of everything else out there

[@AdamMarczakYT]

Thanks! :D

[@jurijssolovjovs3575]

great work thanks mate.

[@AdamMarczakYT]

No problem 👍

[@toluagoro3232]

You are amazing. The way you explain key concepts is top notch. I wish you can teach me for my security + certification

[@AdamMarczakYT]

Fantastic, thank you, best of luck! I wish I could teach everyone, that's why I created this channel!

[@rqn2274]

great video as always

[@AdamMarczakYT]

Thanks again!

[@dhawrylkiw]

Thanks!

[@alamzaibfarooq]

The course is much better than Udemy courses. How is it possible that Adam is not charging anything for this course. He is amazing. The animation is slides are very easy to understand and eye catching.

[@luism2754]

Dude for real, I have been paying a curse that is hours of hours of a guy saying the same thing over and over and just making everything more complicated. But thanks to Adam now I am understanding everything

[@mrindian7422]

Nice explanation.. thanks

[@AdamMarczakYT]

Glad you liked it

[@tustinnrent8502]

another great video! waiting for DP-900! :-)

[@AdamMarczakYT]

You and me both!

[@tustinnrent8502]

@@AdamMarczakYT JK, no pressure!!! Started doing it the old-fashioned way this week by reading MS docs. Many of your videos have already been an excellent complement!

[@dharangsharma6863]

Hi Adam ! Just a quick question, Are your Practice Tests enough to clear the exam, or do I need to purchase some sample papers before appearing for the exam? Also want to thank you for the amazing content, will definitely recommend it to others.

[@SpeoLori]

This course is amazing. Congratulations. And there are deep-dive videos about specific topics. How did you created the drawings? They are so neat.

[@PawanKumar-tu6ti]

KZbin knows how to suggest people, here we come again, thanks brother, your help means a lot, with every video I watch, I can't resist myself from leaving a comment!

[@AdamMarczakYT]

I appreciate that, thank you!

[@debreczenytivadar1220]

Hello Adam. Now I watch all your videos and everything is very professional. Here’s a little random sound clash around 9:28.

[@AdamMarczakYT]

Thanks. Yea some overlap happened there of two audio tracks, unfortunately youtube doesn't allow to reupload the videos. Would need to delete this.

[@pankajkumar-wt2ro]

Good content Adam ,I am waiting for next video

[@AdamMarczakYT]

More to come!

[@devanandapatro7889]

Hi Adam, ur way of teaching is really awesome!! I attempted the az900 exam today and cleared with good marks.. Thanks for all your effort.. 😊😊

[@devanandapatro7889]

@@wiroee yes, I have watched all episodes and thereafter took a practice exam from Udemy..

[@AdamMarczakYT]

Great job! Thanks! :D Keep it up! :)

[@samruddhikalsekar1038]

@@devanandapatro7889 could you mention which practice exam exactly? There are multiple options on udemy

[@windmid39]

This is good for DP-900 too. The comparison make thing more clear but not just list out the services and characteristic.

[@AdamMarczakYT]

Thanks! Yea some topics are mandatory for all fundamental exams like AZ-900, DP-900, AI-900 and even MS-900.

[@shaileshsingh1445]

you are awesome :)

[@AdamMarczakYT]

Thanks! :D

[@abdulkleit]

Hello Adam, eagerly waiting for your remaining episodes from 32-39.

[@AdamMarczakYT]

Soon! :) I had some time of during x-mas and new year, back on schedule soon! :)

[@abdulkleit]

@@AdamMarczakYT Thank you Adam. Happy New Year!

[@sahajrajmalla]

I have been using Azure SQL and making API to query those databases but the query is taking very long time to execute. Now i want to add spark's distributed system and query through the database. Which azure service is better suite for me? Thank you for this wonderful free content Adam. I really appreciate it alot!

[@limsky8012]

When I should choose the Synapse and when Databricks? What type of question I should answer to make a decision?

[@AdamMarczakYT]

It's a difficult choice. Maybe I'll do a video on this in the future, there are too many factors to explain in a simple comment.

[@RandyRieger]

You need to stop being so awesome. Actually don't. Love how you explain everything

[@AdamMarczakYT]

Can't stop being awesome!

[@SiyamandRashid]

Very good videos and the most beautiful part of it the way you prepared... what kind of tablet did you use please? Regards

[@AdamMarczakYT]

It's just my surface pro laptop :)

[@SiyamandRashid]

@@AdamMarczakYT thanks a lot for the reply, i just bought wacom small size but it is kind difficult to draw... Keep it up brother

[@StephenBoesch]

Can these capabilities be scripted or configured via properties/json files? I prefer to use text editors to set things up *repeatably*, check in the config scripts to git, and be able to clone and tweak them.

[@Abecedarianme]

Very informative video. Hoping you could do for DP-203 too.

[@AdamMarczakYT]

Noted, thank you :)

[@Search4Knowledge]

Thanks for this awesome info! Just a question, is HDInsight still part of the DP200 certification? Also when to choose between HDInsight and Databricks as both are spark based?

[@AdamMarczakYT]

Well as you've seen in the video HDinsight is not spark based, Spark is one of several available engines for HDInsight. But more detailed comparisons can be made when using Spark here and in Databricks. In general Databricks is better optimized version of the spark as it was created by Spark creators.

[@sid0000009]

Hello Adam, how do we choose between Databricks and HD Insight if we need a Spark Cluster for our use case?

[@AdamMarczakYT]

Thanks a tough question! But a very good one. I guess both can achieve pretty much the same so it's just matter of organizations/skillsets/etc. In my company we decided to use Databricks for all out data transformations projects. But I have few MVP colleagues which prefer using HD Insight. Some companies also use HD Insight because it's more secure.

[@imsteven3044]

What program do you use to make the notes of your videos?, such as the minute 1:23

[@TheLangmusi]

Thank you for the summary. I wonder what if I want to save a data frame to Azure SQL DB in a Python Notebook like you demonstrate here? Moreover, can you explain which line on your Python block 1 register df as temp table in SQL? Thanks for your answer.

[@AdamMarczakYT]

Note that I registered data frame as Spark SQL view, it's not SQL DB view. Check out my azure databricks tutorial to learn a bit more on how to operate using databricks. Thanks for stopping by.

[@TheLangmusi]

@@AdamMarczakYT Thanks for the answer. Ok, I got it. I will watch your databricks tutorial for more info. Thanks agin.

[@sergiocoutinho6133]

Congratulations for the videos .. your diagrams are excellent for assimilating the content and i am now confident to do AZ-900 soon ! But I confess that I get a little confused by the terms PIPELINE SYNAPSIS, DATA FACTORY or HDINSIGHTS .. sometimes they seem the same thing to me. I even thought that SYNAPSIS corresponded to the old SQL AZURE DATA WAREHOUSE, without the pipeline option. Best regards !

[@AdamMarczakYT]

Best of luck on the exam! You can do it! Those services serve similar purposes but the details are different. Only once you dive into each one of them, then you will be able to distinct them better! :)

[@abhijitboda]

THIS PART IS REMOVED FROM EXAM SINCE MAY 2022

[@5siah530]

Is this the only topic covered in the course that has been removed from the exam?

[@rs-tarxvfz]

Hi Adam, I din't get difference between *synapse pipeline vs data factory*. Pls..

[@haiyingwang5987]

Dear Adam, is it normal for free subscription to get "cloud provider launch failure" when trying to start a cluster in databricks? Is there anyway to get it starting somehow? Thank you!

[@AdamMarczakYT]

Possibly, free subs have 4 vCore limits. try selecting single node checkbox and select smallest available cluster size, which I think was F4s.

[@rs-tarxvfz]

6:20 I think they have unified the creation of workspace within the Azure Portal itself. So no need to go to another platform

[@sidarthg2477]

I think time for an update to the course. Coz I think this course was formed before DP-900 came into existence. So all these portions are now not under AZ-900 and instead it now emphasize more on networking and less on data analytics tolls

[@cebabu]

Generally we store enriched and transformed data sql dataware house, now callled synapse. Can I also store all raw data(structure ) in synapse.

[@AdamMarczakYT]

Those are just tools, use them however you like/need :)

[@timoose3960]

Does anyone know what tools he uses to make the drawings/sketches on top of the video?

[@MrDanesorensen]

Struggling to understand the key differences between Azure Synapse Analytics & DH Insights. Anyone able to help clarify this for me?

[@jorgeregis5614]

Your are awsome , also give us some tips for learn english hahaha , your accents is understandable.

[@AdamMarczakYT]

Thank you! 😃

[@BijouBakson]

God, that was heavy! Oopf! fundamental huh!? :) I definitely didn't grasp this before I passed my exam, still don't understand it much. This course is a little rushed in my opinion. May it's because it's an entire complex field? I'm watching for the 3rd time now. Let's go. :)

[@AdamMarczakYT]

Well the services are quite similar so I understand how they can be confusing. They are competing offers for big data. But don't give up! :)

[@BijouBakson]

@@AdamMarczakYT Haha! my vocabulary is short of give-ups, luckily. I understand the overall idea better now! thank you :)

[@gocrow23]

Can't find code, can you pls put the Databrick code piece you used for illustration?

[@AdamMarczakYT]

Hey, check my databricks tutorial. It’s much better if you want to learn it and it has samples attached. In this tutorial I just showcased databricks but I didn’t expect people to follow. I’m on holidays now but I can upload samples from this episode when I get back. Thanks for watching :)

[@haivercarrillo8603]

Hi Adam, great content as usual, can I ask you a favor, if you can explain the azure data explorer. Thank you!

[@AdamMarczakYT]

Thanks. ADX is definitely on the list :)

[@Mybigsissy]

question 5 on your practice test: the question was about migration, so how does an analytics service perform the task?

[@AdamMarczakYT]

Migration is a process, not a tool :) Think of it from the perspective of a company.

[@shaunakv100]

Do you have also similar videos on DP-900 as well?

[@AdamMarczakYT]

Not yet, maybe in the future :)

[@Mauszinator666]

I don't know if i'm the only one, but the sounds got a little messed up around 9:28. Maybe its just me, or otherwise i hope it is usefull feedback

[@thegreenroom5820]

I am assuming this type of collaboration is extremely expensive in production

[@vr1908]

Hi Mark, Can i have a script for "Connect to Microsoft Opendatset for Sample Data"

[@hochun836]

Hi Adam, is there a plan to make a video about synapse tutorial ? I like your tutorial !!

[@AdamMarczakYT]

Possibly! Thank you! :)

[@BijouBakson]

You missed to mention that Azure Synapse Analytics is the new name for Azure Data Warehouse.

[@AdamMarczakYT]

I didn't miss it. It was on purpose. Majority of the people who study Azure Fundamentals are not familiar with Azure so I rather avoid using already outdated terminology/names as it might introduce confusion. Of' course I will mention this in my upcoming Azure Synapse tutorials later this year.

[@BijouBakson]

@@AdamMarczakYT I understand now, thank you

[@thprodigy]

what's the difference between SNOWFLAKE and Azure HDInsight?

[@AdamMarczakYT]

Closer comparison would be between Azure Synapse Analytics and Snowflake. They are both Massively Parallel Processing databases with unified workspace and ETL capabilities.

[@artak9439]

What about Azure SQL Data Warehouse ?

[@AdamMarczakYT]

It's been renamed to as SQL pools in Synapse.

[@artak9439]

@@AdamMarczakYT Thanks

[@massimo8672]

how do we get the slides

[@AdamMarczakYT]

Sorry Massimo but I don't share my decks, thanks for the visit! :)

[@juggergabro]

09:28

[@uskapyaar8852]

i am wondering who would've disliked this video

[@AdamMarczakYT]

Me too :)

[@uskapyaar8852]

@@AdamMarczakYT you're just too nice at explaining all these concepts that too for free.. i am gonna stick with you.. 🙏 thanks for help i am feeling confident for my exam now ❤️

[@Downstar]

9:28 I thought that i accedently opend a video in a new tab or something hahah

[@AdamMarczakYT]

Hehe, small audio sync issue :)

[@Downstar]

@@AdamMarczakYT hahah no problem, i really ejoyed your content and i passed the exam yesterday! U have learned me alot, thanks!

[@CosmeJunior]

These episode for me, the most difficult to memorize any tips from anyone ?

[@AdamMarczakYT]

Try my reference study materials and read Microsoft Learn, maybe a different perspective on the topic will help!

[@prateekbhardwaj9943]

1 lac people watched this video i am wondering how many cloud engineers are in the world. and how many companies who require these engineers ! its a competitive world isnt it.

[@user-ej3iw8lw3w]

Databricks, Synapse Analytics, HDInsight = PaaS

[@SylwiaKonieczka]

Tou can skip this topic - removed from AZ-900 scope

[@1978parvez]

Hi How are you ?? Can you please brief me which topics are excluded and which i can skip as am following this playlist