Azure Databricks Tutorial | Data transformations at scale

[@AdamMarczakYT]

Dear all. If you are playing around using "Azure Free" subscription you will encounter error that only 4 cores are allowed in your subscription. There is currently a new Cluster Mode called "Single Node" instead of "Standard" try this one, it should be good :)

[@Rahul4u28]

Hello, is Azure Databricks a relational Database? Does Azure Databricks supports incremental refresh in power bi? Does azure Databricks supports query folding? If there are Microsoft documents which answers these queries woukd of great help. Anyone please help.

[@vikash-thechangeforgood..7251]

great help Adam! :))

[@thestrappingentrepreneur2822]

i also had this error, single did not fix it for me, what fixed it for me what going through the azure dashboard and i found an area to edit and request for more cores, i then found out that the location i was subscribed to "east" had no cores but if i want to east 2 i had multiple cores for a specific VM so i had to remake my workspace for these specific cores. and it finally worked

[@praveen_me]

This guy deserves way more subscribers

[@AdamMarczakYT]

Thanks 🤩

[@vijaya83]

@@AdamMarczakYT Totally agree!!! I love every single video!! Appreciate your effort :)

[@Poori1810]

Yeah . Best videos on azure .

[@yashwantbikaner]

I just love the way Adam simplifies the concept, architecture, and real-world use cases of any Azure service. Thanks for another very informative video, really Great work, Adam.

[@AdamMarczakYT]

Glad you enjoyed it! :)

[@creativeluf]

@@AdamMarczakYT You make it seem simply simple

[@joseantoniolopezmesa6807]

MVP man, MVP!!! lol

[@shantanu69073]

Adam - For a person who has just started with Azure and its components, your videos are highly recommended. Keep doing the great work. Really liked your tutorials

[@AdamMarczakYT]

Much appreciated! Will do!

[@tejkumar8727]

Hi Adam, For a person who wants to just start with big data, Azure cloud, Data bricks and its components could you guide the sequence of your videos to follow. Sincere thanks in advance.

[@lekhakotha4244]

Adam, I found your Azure4Everyone videos yesterday. I am visual learner than reading, your videos made sense and easy to learn. Thank you for taking time to make all the videos.

[@AdamMarczakYT]

Awesome, thank you!

[@Mykimbob]

I don't usually comment on youtube. You are the best instructor in Azure. Thank you tons

[@Globetrotter0510]

This guy deserve a huge applause. This piece of course helped me in understanding data bricks in way more clear.

[@close_to_life7954]

Proper explanation, all things covered, and good way of teaching. Loved it.

[@AdamMarczakYT]

Glad you liked it!

[@artaslanian2450]

Agree with Praveen - This guy deserves way more subscribers, extremely competent and clear presentation

[@christianlira1259]

Thank you for both creating this video and taking the time in putting it together. Much appreciated.

[@AdamMarczakYT]

Thanks Christian :)

[@sachidanandgaikwad171]

Adam, You made the jargon simplified, Thanks a lot! Will always prefer to watch and learn Azure from your quick simplified videos.

[@AdamMarczakYT]

My pleasure!

[@leonkriner3744]

Just started to listen. Excellent way of teaching. Finally just teaching without ghost questions in the background :) Also appreciate moving in straight line without deviating to every little detail.

[@dtsleite]

I´ve never learn about Azure like this before. Clean explanations about concepts and pretty cool hands on.

[@AdamMarczakYT]

Glad you enjoyed it! Cheers! :)

[@RajivGuptaEverydayLearning]

Truely Azure4Everyone: You make thing easy to understand for everyone...Kudos...!

[@AdamMarczakYT]

Thanks!

[@HomeChef-DAD]

Thanks, Adam, your instructions are very clear and easy to follow 👍

[@AdamMarczakYT]

Great to hear!

[@loysikdar5754]

Fantastic presentation! One of the best (if not the best) Azure series. Great job Adam.

[@johncurran9597]

Outstanding video Adam. Truly. Thank you for this. I found Microsoft's docs tough to navigate and I was concerned about spending too much $$ money on resources trying to learn. But your video addressed all that. I will be looking at more of your content for sure.

[@AdamMarczakYT]

Awesome! Thanks John :)

[@harigovind511]

You are a legend dude.....keep up the good work..... @Viewers, let's get this man to 100k subscribers

[@AdamMarczakYT]

Thanks Hari! 100k was a dream two years ago, this year, this dream might become a reality. Let's find out together :)

[@surafeltilahun7404]

Please don't ever stop making tutorials on Azure cloud computing. Your explanation is mint. Can you please do one tutorial on how to automate ETL using Azure Logic Apps and ADF? Thank you so much. :)

[@AdamMarczakYT]

Thanks! I won't stop, at least no plans to do so for now :). I;m not sure if I will do full logic apps + ADF + databricks tutorials since I want my videos to be a building blocks and let people put them together. But maybe, I'll think about it :) Thanks for watching!

[@melvinblack8209]

Great demo. The best summary of Data Bricks that I've seen

[@AdamMarczakYT]

Thank you so much :)

[@karthikram1954]

Fantastic video Adam. You are helping so many aspirants realize their dreams. Thank you so much!

[@funwithazure1861]

Great Job Adam! Thanks a bunch...love to see more on Azure Databricks and the Delta Lake

[@AdamMarczakYT]

Thanks! Will do!

[@thayal123]

Nice work - Adam. Explained very easy.

[@AdamMarczakYT]

Thanks! 👍

[@hanumantshinde5652]

Not sure why jus 3.3 lacs views. It helped me start my databricks journey. Thanks a lot Adam. I always love your content.

[@anthonyholleran2721]

I just subscribed to your channel, Adam. These videos are excellent and informative for all IT Professionals alike or anyone wanting to learn something IT.

[@redplanet1657]

This is a masterpiece, Adam! Totally understood the concept.

[@sanjaikhola7184]

Great Video Thanks Adam, You are doing a fabulous job I almost watch all your video and I am yet to love to watch them.

[@AdamMarczakYT]

I appreciate that!

[@505509richard]

Thanks Adam. Nice to have a real world demo I can build upon, rather than marketing material.

[@ChrisInkpen]

Perfect - I was looking for an intro into Databricks and Data Factories - Thank you!

[@AdamMarczakYT]

Glad it was helpful!

[@sanjaymondal8602]

Excellent presentation and so helpful to get knowledge about AZURE-Databricks

[@tjvillanueva396]

Wow - If i would become a data scientist in the future, ill definitely recommend your channel! Thanks for helping noobs like me!

[@AdamMarczakYT]

Cool! Thanks, best of luck TJ :)

[@Sivakumarpoornima]

Your sample code was crystal clear and nice video. thank you so much Adam Marczak

[@AdamMarczakYT]

Glad it was helpful!

[@sascha1785]

helped me a lot to understand what databricks is for - thank you! Will have a look on your other videos for sure

[@AdamMarczakYT]

Awesome, thank you!

[@shivapriyakatta4885]

Thank you so much Adam!....for taking the initiative and creating a great video.

[@AdamMarczakYT]

Awesome, thanks!

[@laxmikantasahoo1036]

Thanks for enriching our knowledge by providing such beautiful video . Very helpful.

[@AdamMarczakYT]

Thank you so much :)

[@ergouzz]

Hi Adam, Great video ! You really saved a lot of my time reading the databricks documentation

[@AdamMarczakYT]

Watching during x-mas times? You make it worth it even more! Thanks and happy holidays!

[@salmaboudinar8613]

same here !! happy holidays :)

[@AdamMarczakYT]

To you too! Happy holidays!

[@arulmouzhiezhilarasan8518]

Faced some minor issues in between like start time in sas generations, sas authorizations, timezones and regions etc., so deleted RG and restarted again from ground 0, finally works well! Thanks Adam for teaching even some complex things in simple ways! your passion helps us to learn new things!

[@AdamMarczakYT]

Nice! Staying persistent is the best way to learn. Sometimes smallest mistakes are hardest to catch. It's easier to start over.

[@yogeshnikam8064]

Now Azure is simple to me :) Thanks Adam!!

[@AdamMarczakYT]

Happy to help!

[@PalaniRamu1]

Great explanation of Workfllows.

[@rajeevranjan8790]

Thank you for this great video. Looking forward for next video for more hands on.

[@AdamMarczakYT]

More to come!

[@ishwantsingh5291]

thanks adam , such elaborative and clear guidance !

[@AdamMarczakYT]

My pleasure!

[@juanalfredoblancasvelazque5553]

Thanks Adam for the valuable information about Azure Databricks. Regards from Mexico.

[@AdamMarczakYT]

Glad it was helpful!

[@ngophuthanh]

Thank you, Adam. It's another great video from you.

[@AdamMarczakYT]

My pleasure!

[@mehmetkaya4330]

So very well explained! Thanks you for the great tutorial!

[@Rafian1924]

You are the ultimate instructor

[@kalyanchatterjee8624]

Your tutorials are class apart - very very good. Thank you so much.

[@wasimakram365]

Thanks Adam!!. It helped alot. Very informative.

[@AdamMarczakYT]

Glad it was helpful!

[@rembautimes8808]

Good to have a video that is technical and hands on

[@AdamMarczakYT]

Thanks!

[@venkatkondragunta9704]

Excellent.. I really liked your explanation!! Thank you!

[@ahmedmohammed1284]

Amazing work Adam, thanks for the video

[@AdamMarczakYT]

It's my pleasure!

[@SaadAllahMARZAK]

Merci beaucoup pour cette formidable formation

[@jacekkafel-kania9620]

Cudowny tutorial, gdyby każdy wykonywał swoją robotę w ten sposób, mielibyśmy inteligentne buty od nike'a i latające samochody :)

[@AdamMarczakYT]

Ha ha! dziękuje ;) fajnie ze sie podobalo.

[@luh318]

Very instructive video. Thanks for uploading!

[@TheAl217]

I'm going to start working with Databricks today so thanks a lot for this tutorial.

[@AdamMarczakYT]

Glad you enjoy it! Thanks!

[@sravanilakshmi453]

This is very helpful. Thanks Adam.

[@saikumarvenigalla9822]

Excellent explanation. Thank you so much for the valuable content.!

[@AdamMarczakYT]

You're very welcome!

[@rockyxyzable]

Your videos are more than awesome. I am flattered :)

[@AdamMarczakYT]

Thank you!! :D

[@grzegorz8743]

nice video and great introduction to Azure Databricks :)

[@AdamMarczakYT]

Thanks! 😀

[@brpawankumariyengar4227]

Awesome Video …. Very useful … Thanks for posting

[@leefig6089]

Another great presentation

[@AdamMarczakYT]

Thanks! Appreciated!

[@mouhannadoweis7605]

Thank you very much. I really enjoy your videos.

[@AdamMarczakYT]

Glad you like them!

[@nakulagham2058]

Thanks a lot Adam for this great content !

[@MilesJoyDiary]

Very useful tutorial thank you for sharing it’s so good. 👍🤝🔔😇❤️

[@davidgodinez7146]

Great explanation Adam!

[@GG-uz8us]

Even there are so many good comments here, I still want to say thank you, indeed very good.

[@AdamMarczakYT]

I appreciate your comments, thanks for stopping by!

[@Jogukrish]

Wonderful demo! appreciate your knowledge and work. nice explanation easy to understand.

[@AdamMarczakYT]

Glad it was helpful!

[@evatate3104]

Awesome course! Ran into a few snags with the getting everything to work because the current version of Azure Databricks don't show what we see in the video. I can download the query results but I don't see anywhere, where you would create a visualization using pychart. A little frustrating but that is technology, the video is 2 years old and they have already made so many changes.

[@jacekb4057]

Świetny tutorial. Pomógł mi w pracy. Dzięki

[@vivek.padale]

Awesome content Adam, Keep Going, Best of Luck!!!

[@AdamMarczakYT]

Thanks a ton! :)

[@claudineiacezar1760]

Thank you, Adam! It is a great demonstration.

[@AdamMarczakYT]

Glad you liked it!

[@seb6302]

Would love to see a video on batch processing using adf and databricks!

[@AdamMarczakYT]

Maybe maybe ;)

[@jananitamilselvan9462]

Thank you very much.. your video helped me lot to understand the concepts:)

[@AdamMarczakYT]

Glad to hear that!

[@Rafian1924]

You are the legend Adam.

[@AdamMarczakYT]

Thanks :)

[@dileepdillu666]

Fantastic video Adam Thank you so mush

[@CarlosGutierrez-go9hq]

Hey, man, great video! Just a quick question: is it "warehousing" or "warehouseing"? Just to keep it in mind, I just read it at minute 4:14 and it got me asking.

[@bifurcate-ai]

thanks a lot adam for the simple, yet very informative video!!

[@AdamMarczakYT]

My pleasure!

[@RobertoMartinez-pz7im]

Great vídeo. Keep making videos!

[@AdamMarczakYT]

Thanks, will do!

[@AsifKhan-hi2km]

fantastic this is what i am looking forr thanks man.!!

[@Sabarishnagappan]

Great Video Adam :) I'm trying to perform data manipulation operations on datasets that range from 7 TB to 110 TB, most of the elementary operations like data.count(), distinct count etc. results in query timeout/failure. But same operations work just fine in datasets that weigh-in around 500 GB. Is ADLA a more suitable option for my purpose than Databricks? I'm trying to switch from Cosmos which has been been able to handle the huge datasets without any hassel.

[@AdamMarczakYT]

Personally I'm not sure how much I would invest in ADLA since the technology is not actively developed by Microsoft anymore. Probably would consider databricks delta tables or SQL DW (synapse) database. Databases are good at 'counts', 'sums', etc. because they have those calculated and cached upon insertion, so the queries are very fast.

[@funwithazure1861]

Hi Nagappan VR! Is your cluster or are your VMs large enough ( memory and CPU) 110 TB is still relatively small for real big data workloads....

[@Sabarishnagappan]

@@funwithazure1861 You are right, the cluster configuration was not scaled up enough to handle the workload.

[@sivakumar-ef1oy]

Awesome ; Sharp & Straight content.

[@AdamMarczakYT]

My pleasure!

[@muditgupta08]

Hi Adam good information. But is there any benchmark for Cluster configuration against the load (in terms of GBs of data)

[@AdamMarczakYT]

Unfortunately there are too many variables to consider. Most of the time it depends on the complexity of the transformation and the size of the data. Databricks team publishes some on their website, feel free to check those out! :)

[@yaki879]

Thank you, so good explanation!

[@AlfredDHull]

Great job Adam!

[@AdamMarczakYT]

Thanks!! :)

[@paulhernandezgermany]

Hi Adam, great video :). You presented a slide where Azure Data Factory is shown along with Databricks and other components. My questions is, do you already have a video or a link where the choice between data factory and Databricks is discussed? For instance, the transformations you presented can also be done with a data factory low-code approach. I guess scalability and performance can be good reasons for Databricks but would be nice to have some guidelines where to choose or even combine them.

[@AdamMarczakYT]

Thanks Paul. Great question, no video yet. Just a fun fact that data factory low code approach (data flows) still compiles into a databricks package and is deployed as if you wrote the code yourself. So performance and scalability wise they are likely the same. Primary difference is that low-code has limitations of what is available in the UI, as such I typically say for data & analytics projects go databricks because you are almost guaranteed to need complex transformations which can't be achieved using low-code. But if you got a simple project with some simple data transformations low-code is great. Of' course my words might change in future with the release of wrangling data flows or implementation of new and new features. So we will see :)

[@Alwinlcw]

Hello Adam, Do you have any videos which is related to PGP file decryption and key pairs generate in Azure Databricks ?

[@sdbhattacharya]

Thanks for making this video. It was precise and provided a lot of content.

[@AdamMarczakYT]

Thanks!

[@otroleonarbe]

Great tutorial. Thx for the info

[@AdamMarczakYT]

My pleasure! Thanks!

[@ninocrudele]

Great job thank you Adam

[@mulakalanaidu3662]

Hi Adam, Amazing video... is there any possibility to compare file snapshot using different time stamps like compare today's data vs yesterday's data in data bricks? if possible can you please help me the details that how we exactly compared? THANKS.

[@vzntoup]

Thank you! Excellent tut :)

[@SS-lv8pn]

Hi Adam, I am not able to see any other training video in Azure Analysis Services, can u please create a seperate playlist ??

[@terryliu3635]

At 4:29, also Azure SQL Database, right?

[@AdamMarczakYT]

You can use any JDBC compatible service and use java to connect but it's not 'native' per se. But you are correct SQL database is on the table too. docs.databricks.com/data/data-sources/sql-databases.html

[@nadya6368]

Great video! Thank you and please continue doing this! :)

[@AdamMarczakYT]

Thank you! Will do! :)

[@arupnaskar3818]

Thank u so much Marczak ..very helpful tutorials ..

[@AdamMarczakYT]

You are welcome! Thanks.

[@eerosiljander4622]

Great work Adam!

[@Kohimond]

Great training, thanks a lot

[@AdamMarczakYT]

You are welcome! Thanks!

[@vivekselvam8676]

Thank You for nice introduction into Azure databricks

[@AdamMarczakYT]

It's my pleasure :) thanks!

[@maheshpatkar886]

Adam why did you not require to mount in when you read the data from the parquet file ? in both case we were reading data from containers ?

[@Extream917]

Hi Adam it is an amazing video and it saved my lot of time

[@AdamMarczakYT]

Thank you :)

[@jeanphelipperamosdeoliveir711]

Great video man! I l really liked the demo session.

[@AdamMarczakYT]

Awesome, thank you!