Master Databricks and Apache Spark Step by Step: Lesson 23 - Using PySpark Dataframe Methods

In this video, you learn how to use PySpark dataframes methods on Databricks to perform data analysis and engineering at scale. This is the core of using Python on Spark and you need to learn the power but also the nuances involved.
Video demo notebook at:
github.com/bca...
Apache Spark Zeppelin Notebook link will be posted later.
For information on how to upload files to Databricks see:
• Master Databricks and ...

[@andywendycox]

Bryan - thanks so much for this series. You've made Databricks ( and Spark for that matter ) very easy to digest. These videos have been a lifesaver...

[@BryanCafferky]

YW

[@1277marina1277]

Brian, thank you for great presentation. Your gift to explain complicated things as simple concept is amazing

[@BryanCafferky]

YW and thank you for the kind words.

[@ravitutika1204]

waiting for it from last few weeks , Thanks Bryan

[@abbasstovewala2298]

Thanks Bryan for this series. very helpful.

[@arpitarora293]

Hey man thanx for the whole series i just started working on databricks and was completely oblivious to how it works but your helped me quite alot so ... thanx for that )

[@BryanCafferky]

Great! Glad to help.

[@amarnadhgunakala2901]

Looking forward more pyspark vids Thanks

[@dangustafsson8933]

Reaching the end of your series, very enlightening and friendly format. These end lectures are really interesting. Now here, I’m looking to understand how to efficiently load data, based on different data sources (rmdb:s, hdfs, mongo). And avoiding ‘shuffles’ or at least understand the cluster bottlenecks…. also on my to-do list…

[@BryanCafferky]

Actually, there's 15 more videos now and more to come. Databricks has many autuomatic optimizations so you may not have issues. Shuffles only happen when you need data to be co-located, i.e. such as in joins. There are techniques to improve performance but Databricks Adaptive Query Execution may fix any issues.

[@anandmahadevanFromTrivandrum]

Wow, that was a lot to take in, but well presented. Thanks again

[@vibhaskashyap8247]

Thanks for sharing videos, great content and you make complex topics easy to understand 👍

[@BryanCafferky]

Thank You!

[@hmishra8524]

Made my weekend thanks again brayan keep up the good work . BR, Hardik 🙏😀

[@n1njab0b]

Great stuff Bryan. Very informative!

[@BryanCafferky]

Thanks

[@BryanCafferky]

It was pointed out in a comment, that seems to have been deleted, that you should use spark context instead of sqlContext as the spark context is the newer unified way to connect to the Spark session. Where you see code like sqlContext.read.format(....), just replace sqlContext with spark and you should be all set.

[@franciscmoldovan2153]

Really nice naming convention!

[@CoopmanGreg]

Great explanations!

[@neostar3498]

Thank you very Much Sir.. You made my life easy

[@BryanCafferky]

Awesome! Glad to help.

[@Gamma3]

Excelente canal amigo! Me suscribo

[@tsri187]

Thanks Bryan for the videos :)

[@BryanCafferky]

YW

[@Raaj_ML]

Great tutorial !

[@Raaj_ML]

What is the use of caching ? If you do not do caching, anyways the data frame will remain in memory...right ?

[@BryanCafferky]

Good question. The dataframe does not necessarily stay in memory. Spark can wipe it out and reuse the memory. Caching tells Spark to hold the dataframe in memory so it can be reused.

[@RandyL86]

I'm confused on when to use "sqlContext." versus "spark." How do we know when to use what? For instance, to query you use "spark.sql" but I see from documentation you can also do, "sqlContext.sql"...is there a difference?

[@eugenezhelezniak539]

Bryan - how do we know whether the dataframe is local or lives on the cluster? Is it as simple as pandas = local, spark = distributed? And follow up to that, if you have a large local pandas df, how do you work around degraded performance?

[@Rickantonais]

Thank you for the material, yet there is a background high tone sound which makes the video horrible to listen

[@BryanCafferky]

Thanks. I am replacing some videos that have noise as it did not show when I played them back. I switched to using a noise cancelling headset in a later video. Using earbuds or headphones give the most noise so you may have better luck playing at home on a laptop with speakers. I will replace this video soon with improved sound. Yetti mics are not good for this type of recording. 😞

[@juanpabloguerra9512]

Hey Bryan this is awesome content. I'm tying to open the file after cloning your GH repo but it seems like it downloads as a DBC file that can't be open on VS code using Jupyter notebooks for example. Is there anything I'm missing? Thanks a lot for the great content

[@mohamedamineazizi3360]

thanks

[@ranjeevtiwari6976]

Thanks for this series Bryan. The notebook you shared in github is of .dbc extension, can you update your git with current class notebook?

[@BryanCafferky]

YW. dbc stands for Databricks compressed format and can be directly imported into Databricks. Are you using Databricks?

[@felixscarbrough5694]

Hi Bryan, I've been watching your series for a little while now and finding it very helpful. Unfortunately this video seems to have some really high pitched tone in lots of it and it makes it quite unpleasant to listen to. Is there any way you'd be able to remove this?

[@BryanCafferky]

Hi Felix, I don't hear it. Can you give me some time placement where you get this? Are you using earphones? I have found that some people have heard background noise, though low frequency not high when using earphones. Thanks

[@felixscarbrough5694]

@@BryanCafferky Hey thanks for responding! It comes up several times in the video but the earliest is between 0:00 and 0:54. I can hear it with headphones on but also on some speakers. It's possible that either your audio equipment can't display that tone or it might be above your range of hearing?

[@BryanCafferky]

@@felixscarbrough5694 Ok. I can hear some background noise, much quieter than my speaking, when I play it on my Kindle. I think it is the fan on my laptop. I'll need to adjust the audio input and move the mic away from the laptop in the future. Thanks

[@BryanCafferky]

@@felixscarbrough5694 Testing I found it is the Yeti Mic that is doing this. Its a condenser mic that picks up everything. Please try my new video in which I just used a headset. Seems to solve the issue kzbin.info/www/bejne/qWTXh6OMj5xqbJo

[@felixscarbrough5694]

@@BryanCafferky Thanks Bryan, appreciate it c: