PySpark Tutorial: Spark SQL & DataFrame Basics

Thank you for watching the video! Here is the code: github.com/gahogg/KZbin/blo...
Titanic Dataset: www.kaggle.com/c/titanic/data
Code For Setting Up Spark 3 in Colab: jacobcelestine.com/knowledge_...
Learn Python, SQL, & Data Science for free at mlnow.ai/ :)
Subscribe if you enjoyed the video!
Best Courses for Analytics:
---------------------------------------------------------------------------------------------------------
+ IBM Data Science (Python): bit.ly/3Rn00ZA
+ Google Analytics (R): bit.ly/3cPikLQ
+ SQL Basics: bit.ly/3Bd9nFu
Best Courses for Programming:
---------------------------------------------------------------------------------------------------------
+ Data Science in R: bit.ly/3RhvfFp
+ Python for Everybody: bit.ly/3ARQ1Ei
+ Data Structures & Algorithms: bit.ly/3CYR6wR
Best Courses for Machine Learning:
---------------------------------------------------------------------------------------------------------
+ Math Prerequisites: bit.ly/3ASUtTi
+ Machine Learning: bit.ly/3d1QATT
+ Deep Learning: bit.ly/3KPfint
+ ML Ops: bit.ly/3AWRrxE
Best Courses for Statistics:
---------------------------------------------------------------------------------------------------------
+ Introduction to Statistics: bit.ly/3QkEgvM
+ Statistics with Python: bit.ly/3BfwejF
+ Statistics with R: bit.ly/3QkicBJ
Best Courses for Big Data:
---------------------------------------------------------------------------------------------------------
+ Google Cloud Data Engineering: bit.ly/3RjHJw6
+ AWS Data Science: bit.ly/3TKnoBS
+ Big Data Specialization: bit.ly/3ANqSut
More Courses:
---------------------------------------------------------------------------------------------------------
+ Tableau: bit.ly/3q966AN
+ Excel: bit.ly/3RBxind
+ Computer Vision: bit.ly/3esxVS5
+ Natural Language Processing: bit.ly/3edXAgW
+ IBM Dev Ops: bit.ly/3RlVKt2
+ IBM Full Stack Cloud: bit.ly/3x0pOm6
+ Object Oriented Programming (Java): bit.ly/3Bfjn0K
+ TensorFlow Advanced Techniques: bit.ly/3BePQV2
+ TensorFlow Data and Deployment: bit.ly/3BbC5Xb
+ Generative Adversarial Networks / GANs (PyTorch): bit.ly/3RHQiRj

[@GregHogg]

Take my courses at mlnow.ai/!

[@coemgeincraobhach236]

Thanks so much Greg, great job! Paying thousands for a masters at university, and people like you consistently pump out tutorials of way better quality. Its madness.

[@GregHogg]

Yup that's how it goes! Haha I'm really glad to have helped 😃

[@TheALahiri]

Many thanks Greg for opening up a new frontier! I had no idea Google Colab was so generous and allowed installation and practicing of Spark. Your tutorial packs an astonishing amount of information, that too in an engaging way, in a very short timeframe. You are now my Guru for Spark.

[@GregHogg]

Yup, it does! 😃

[@jacobburt5424]

I appreciate you and your videos so much. In my data science classes we're expected to teach ourselves Pyspark, Dataframe, Pandas and a bunch of other technologies and you've made the experience much more manageable.

[@GregHogg]

Well I'm super happy to hear that Jacob, thanks for the kind words!

[@andersborum9267]

These introductory videos are pure gold; thanks for sharing.

[@GregHogg]

Thank you greatly for the kind words, and for your support! It means a lot. 😊

[@darrienjohnson9053]

Don’t know if you’ll see this but I got into data engineering thru my company. They provided me the opportunity to become a software engineer, I was previously a cable installer/field tech. Although they provided this opportunity, I’ve still had to do much of my learning on my own. Your channel is amazing. Videos like these make all the difference. I really appreciate you making content where you’re walking thru the code. Once I get this under my belt I plan on creating content as well. Thank you. 🙏🏾

[@GregHogg]

Oh that's so great to hear!! Thank you for the kind words and I wish you all the best Darrien!!

[@some90sKid]

🙌🙌

[@ashutoshsingh7529]

Thank you so much. Pretty covers everything you to get started with pyspark. I wish you had included merging as well.

[@faizalshebli9520]

Great video. Simple yet effective to comprehend.

[@GregHogg]

I'm very glad to hear that Faizal, and I greatly appreciate your kind words!

[@gauravraichandani7722]

This was really amazing. Waiting for more uploads on pyspark.

[@GregHogg]

Awesome! Did you catch the other 15 minute long one?

[@gauravraichandani7722]

Yep, I have. Followed along both the videos.

[@GregHogg]

@@gauravraichandani7722 okay awesome!

[@barmalini]

In just 17 minutes I've learnt so much. Thanks!

[@GregHogg]

Perfect, really glad to hear it :)

[@ramanantoaninaharintsoanan7752]

Thanks for sharing your knowledge. Great video.

[@GregHogg]

You're very welcome and glad to hear it!

[@saketsrivastava84]

Amazing content...please prepare more like these.. 👍🏻

[@96supersh]

Dude 😍 Thank you so much, man. this is really helpful to do my office work. Hope you will make more content like this. subscribed and shared . thanks

[@GregHogg]

Really glad to hear it 😃🥰

[@prithvib]

This videos deserves 1m views

[@GregHogg]

Haha that would definitely be preferred, thanks so much for the kind words I really appreciate it!

[@nishantbahikar5639]

Bro you have explained it so well.. keep going

[@GregHogg]

Thanks, great to hear!

[@arsheyajain7055]

I was waiting for this one!

[@GregHogg]

I've wanted to make this for a long time since the PySpark RDD video did so well. Enjoy!

[@mohamedelkhaldi1096]

Thank you so much !!! Always great contents

[@GregHogg]

You're super welcome. Really glad to hear that.

[@ronaldfungss]

This is amazing! Thanks Greg : ]

[@GregHogg]

Awesome! You're very welcome 😄

[@noushinbehboudi5694]

Awesome. Please keep up the good work. Please make more videos in spark. Thank you

[@GregHogg]

Awesome, thank you!

[@noushinbehboudi5694]

Could you please suggest any good material video tutorial for pyspark for a newbie?

[@GregHogg]

@@noushinbehboudi5694 Isn't that this one?

[@noushinbehboudi5694]

@@GregHogg I started pyspark with your videos. But I only found 2 videos in your channel. Are you going to upload more?

[@GregHogg]

@@noushinbehboudi5694 Makes sense. Not for awhile unfortunately, I would recommend doing the databricks specialization on Coursera :)

[@Buxussempervirens]

This is so amazing 😍😍

[@GregHogg]

Thanks so much!!

[@Officially_fit]

You're awesome man!

[@GregHogg]

No you!

[@AkshayKumar-vd5wn]

Thank you for the video. I have a problem - When I convert a column from string to int and then run printSchema it shows String and not the int. Is there a better way to convert string column to int in pyspark and make it a permanent change? I use thr data uploaded locally, I.e from my computer. Is this happens to only locally uploaded files? Will the conversation take place smoothly when operating on okne databases i.e through servers.

[@matattz]

I would like to hear your opinion on Ponder. Considering that you can now work with Ponder similarly to how you work with Spark, do you believe it is still necessary to learn PySpark? I'm interested in your perspective on this matter, and if you are aware of any downsides or differences between Ponder and Spark.

[@soumyadeeppattanaik526]

hey.. Hogg while i am trying to extract sum of sales by grouping the states from the dataframe, its giving an unnesessary floating values. If the sum is 150.0 its giving like 150.856743 like this.can you explain this..

[@malanshinde6814]

Awesome

[@DanielWeikert]

Are there more then the 2 videos on pyspark? Thanks and great work

[@GregHogg]

That's all I've got, sorry!

[@josephjoestar995]

Thanks

[@GregHogg]

Wow, that was very nice of you Joseph! Thank you :)

[@rjayanth]

Thanks Greg , it was clean and straight forward. like it a lot.. could you suggest me a course to learn Spark .In our company we are trying to build a data lake on hadoop using hive.. We have a lot of complex stored procedures on a rdbms. i will be migrating all the logic into Data lake.. spark would be great tool to accomplish this.I would really appreciate if you suggest some online courses.

[@GregHogg]

No problem! Check out some of the big data courses on Coursera.

[@AlexMar-r]

is this the same as Apache spark ?

[@gerardolamasrosales9777]

Hola, como creo una base de datos con pyspark?

[@limingcai1508]

I downloaded the train.csv file to my laptop's local hard drive, and tried to read it with titanic_df = spark.read.csv("c:\UserFiles\My Data\train.csv", header=True, inferSchema=True), but got an error message. Do you kbnow what I did wrong?

[@kishanbsh]

Can you expand more on the sql bit along with joins?

[@GregHogg]

Not in the comments (not properly, anyway); joins is merging two tables together by matching common rows. In PySpark, a join is essentially the exact same thing as it is in normal SQL. You'd have to learn what a join is first :)

[@theodoruswidhi8192]

bro what is the difference between .limit(3) and .show(3) ? i tried it on data brick using python on spark 3.0.1 . show command showed the csv dataframe&row , but limit command can't showed the csv dataframe&row.

[@GregHogg]

I don't know, sorry.

[@user-sq8xd8tb6z]

Small annoyance, but does anyone know why when I run something, like spark.sql('select * from Movies'), for example, it gives me the datatypes instead of displaying the actual table data?

[@GregHogg]

Empty table maybe?

[@ranasana9681]

Thank u so much, sir i have à problem in converting spark.dataframe to pandas.df, beacuse i have a large number of data... How can i do !?

[@GregHogg]

Isn't there a .topandas function?

[@shankarsr1]

If we can use spark.sql then we don't need dataframes function like filter, agg etc.?

[@GregHogg]

It's essentially a different way of doing exactly the same thing. Sometimes I mix and match depending on how comfortable I am with what I'm trying to do

[@byronexaporriton318]

how can we create a python DataFrame from an already existing table?

[@GregHogg]

You'll need to read in the file using one of Sparks read functions

[@EzraSchroeder]

i need to learn "the rest" of pyspark sql **fast** (& hardly know any sql at all). suggestions??? what are some good resources???

[@GregHogg]

Honestly, the documentation is great.

[@adeyemiadeniran2871]

I am getting an error message ' E: Unable to locate package open-jdk-8-jdk-headless'. What could be the reason plz?

[@GregHogg]

I think pip install PySpark is enough to install

[@91horse]

Awesome ! (..)

[@GregHogg]

Thank you!!

[@kunjuperath]

For installing pyspark, why didn't you just do `pip install pyspark`? I'm trying to use the pandas api that was introduced in 3.2 with this method but even if I wget and unzip the spark 3.2 tar file I can't import the module. Cool tutorial though!

[@GregHogg]

That's a great question. I actually didn't know it was this easy in Colab at the time. Thanks!

[@ajaynayak4697]

just wow.

[@GregHogg]

😄

[@KradianKrad]

what is the difference between filter and where

[@GregHogg]

Nothing, they are the same :)

[@mihirgaming716]

Can anyone give the command to replace null value in age column with average age for each gender ?

[@GregHogg]

This is a great exercise 😁

[@yashsvidixit7169]

4:54 funny voice crack LOL

[@GregHogg]

You're right, that is pretty funny 😂

[@yashsvidixit7169]

@@GregHogg the video was pretty amazing. Thanks.

[@GregHogg]

@@yashsvidixit7169 Really glad to hear that, and thanks a bunch!