Master Databricks and Apache Spark Step by Step: Lesson 20 - PySpark Introduction

[@Darshakramani-kr2he]

What a gentlemen you are! I have been watching your videos, you explain to all age group of people very nicely and easy to understand way, from beginner to expert. I am writing this comment at 2 o'clock midnight watching your all videos from 1st to 20th of this series, will finish the other ones this week. I am so pleased the way you explain, sir, I am 30y old Indian Canadian from Toronto. I have a small question, how relevant is still this series after 3 years, do we need to modify/update something specific - tool/area to focus on? I know it is wide question but any light would help. I have also read your other above comment of darwin as well. I genuinely wish this channel were big, but these days unfortunately people with shallow knowledge with flashy graphic with little intention to explain are spreading their wings. I wish you good luck sir. you have a grace, wisdom to explain, humbleness to explain common detail which is obvious but you still explain keeping in mind wider audience, no arrogance, again, really pleased with your videos.

[@BryanCafferky]

Thank you so much and glad my channel is so helpful to you. I'm thinking I should add a video to the Databricks series to catch up the audience on what has changed or been added to Databricks. Its been almost all added things, nothing I can think of that Databricks took away. Delta Live Tables, Delta Lake (which I have some videos on),Unity Catalog, performance enhancements like Adaptive Query Execution and Photon which I recently started a series on, and lots of enhancements to workflows (I have a video on that too), and notebooks. Also Scala is not a good place to invest time anymore. Photon is basically replacing the Spark engine making Scala less and less relevant.

[@ravitutika1204]

Great video Bryan Looking forward more videos on Pyspark

[@BryanCafferky]

Doh! Thanks for asking. It was dropped from the playlist somehow. Re-added it.

[@vaibhavvaidya8715]

Well explained, looking for more videos on pyspark deep down

[@alfredsfutterkiste7534]

well explained!

[@BryanCafferky]

Thank you!

[@darwin6984]

Hello Bryan, I am newbie to big data for five years only , among apache Spark vs pyspark vs hadoop, which one should learn first? could u give me your view pls

[@BryanCafferky]

Hi Darwin, Hadoop MapReduce is generally losing ground to Apache Spark (Databricks is built on Spark). The Python wrapper library to leverage Spark services is PySpark so you always need Spark to use PySpark. Python is very popular on Apache Spark so I would recommend 1) Learn about Apache Spark/Databricks, then Spark SQL, then Python/PySpark. Also, look into Delta Lake.

[@prateeksinha2526]

Why is lesson 20 deleted?

[@BryanCafferky]

No. Just dropped from the playlist somehow. Fixed it. Thanks.