PySpark Tutorial

[@stingfiretube]

This man is singlehandedly responsible for spawning data scientists in the industry.

[@MSuriyaPrakaashJL]

I am happy that I completed this video in one sitting

[@baneous18]

42:17 Here the 'Missing values' is only replacing in the 'Name' column not anywhere else. even if I am specifying the columns names as 'age' or 'experience', it's not replacing the null values in those columns

[@Star.22lofd]

Lemme know if you get the answer

[@WhoForgot2Flush]

Because they are not strings. If you cast the other columns to strings it will work as you expect, but I wouldn't do that just keep them as ints.

[@oiwelder]

0:52:44 - complementing Pyspark Groupby And Aggregate Functions df3 = df3.groupBy( "departaments" ).agg( sum("salary").alias("sum_salary"), max("salary").alias("max_salary"), min('salary').alias("min_salary") )

[@yitezeng1035]

I have to say, it is nice and clear. The pace is really good as well. There are many tutorials online that are either too fast or too slow.

[@shritishaw7510]

Sir Krish Naik is an amazing tutor, learned a lot about statistics and data science from his channel

[@candicerusser9095]

Uploaded at the right time. I was looking for this course. Thank you so much.

[@farees96]

Hvala!

[@arturo.gonzalex]

IMPORTANT NOTICE: the na.fill() method now works only on subsets with specific datatypes, e.g. if value is a string, and subset contains a non-string column, then the non-string column is simply ignored. So now it is impossible to replace all columns' NaN values with different datatypes into one. Other important question is: how come values in his csv file are treated as strings, if he has set inferSchema=True?

[@kinghezzy]

This observation is true.

[@aadilrashidnajar9468]

Indeed i also observed the same issue, now don't set inferSchema=True while reading the csv to RDD then .na.fill() will work fine

[@sathishp3180]

Yes, I found the same. Fill won't work if the data type of filling value is different from the columns we are filling. So preferable to fill 'na' in each column using dictionary as below: df_pyspark.na.fill({'Name' : 'Missing Names', 'age' : 0, 'Experience' : 0}).show()

[@aruna5472]

Correct, even if we give value using dictionary like @Sathish P, If those data type are not string, it will ignore the value, once again, we need to read csv without inferSchema=True, may be instructor missed it to say that missing values applicable only for the string action( Look 43:03 all string ;-) ) . But this is good material to follow, I appreciate the good help !

[@gunjankum]

Yes i found the same thing

[@alireza2295]

This video provides an excellent starting point for the journey-clear, concise, and incredibly efficient. Great job!

[@MiguelPerez-nv2yw]

I just love how he says “Very very simple guys” And it turns out to be simple xD

[@nagarjunp23]

You guys are literally reading everyone's mind. Just yesterday I searched for pyspark tutorial and today it's here. Thank you so much. ❤️

[@centershopgaming7655]

Same thing

[@Mathandcodingsimplified]

U phone is being tracked.... It's no coincidence.... All our online activities are recorded

[@HemaPrasathHeptatheLime]

@@Mathandcodingsimplified Recommendation engines pog!?

[@srinivasn415]

Not the channel but KZbin is.

[@ygproduction8568]

Dear Mr Beau, thank you so much for amazing courses on this channel. I am really grateful how such invaluable courses are available for free.

[@sunny10528]

Please thank Mr Krish Naik

[@mohandev7385]

I didn't expect krish.... Amazingly explained

[@anikinskywalker7127]

Why are u uploading the good stuff during my exams bro

[@awwtawnoo]

HaHa

[@settarapramod446]

Xactly

[@neillunavat]

EVEN MY EXAMS GOIN ON

[@subramanianchenniappan4059]

Can't you watch it later🤣🤣

[@antonmursid3505]

Antonmursid🙏🙏🙏🙏🙏✌🇸🇬🇸🇬🇸🇬🇸🇬🇸🇬✌💝👌🙏

[@lakshyapratapsigh3518]

VERY MUCH HAPPY IN SEEING MY FAVORITE TEACHER COLLABORATING WITH THE FREE CODE CAMP

[@dataisfun4964]

Hi krishnaik, All I can say is just beautiful, I followed from start to finish, and you were amazing, was more interested in the transformation and cleaning aspect and you did justice, I realize some line of code didn't work as yours but all thanks to Google for the rescue. This is a great resource for introduction to PySpark, keep the good work.

[@arjitsrivastav555]

Krish Naik has pretty much nailed it in this video. Loved it👏

[@IvanSedov-i7f]

Прекрасное видео и прекрасная манера подачи материала. Большое спасибо!

[@MöbiusuiböM]

15:20 - lesson 2 31:35 - lesson 3

[@vivekadithyamohankumar6134]

I ran into an issue while importing pyspark(Import Error) in my notebook even after installing it within the environment. After doing some research, I found that the kernel used by the notebook, would be the default kernel, even if the notebook resides within virtual env. We need to create a new kernel within the virtual env, and select that kernel in the notebook. Steps: 1. Activate the env by executing "source bin/activate" inside the environment directory 2. From within the environment, execute "pip install ipykernel" to install IPyKernel 3. Create a new kernel by executing "ipython kernel install --user --name=projectname" 4. Launch jupyter notebook 5. In the notebook, go to Kernel > Change kernel and pick the new kernel you created. Hope this helps! :)

[@yashdhaga7047]

Thank you so much!

[@yashbhawsar0872]

@Krish Naik Sir just to clarify at 26:33 I think the Name column min-max decided on the lexicographic order, not by index number.

[@shankiiz]

yep, you are right!

[@JackSparrow-bj5ul]

Thank you so much @Krish Naik for bringing this amazing content. tutorial has really helped me clearing few concepts and really thoughtful hands-0n explanation. Hats-off to the FCC team. Looking forward to your channel @Krish.

[@sharanphadke4954]

Biggest crossover : Krish Naik sir teaching for free code camp

[@alanhenry9850]

Atlast krish naik sir in freecodecamp😍

[@SporteeGamer]

Thank you so much to give us these type of courses for free

[@SameelJabir]

Such an amazing explanation. For a beginner: 1.50!hours really worth... You nailed it in a way with very simple examples In high professional way.... Huge Hatsoff

[@bhatt_nikhil]

Really good compilation to get started with PySpark.

[@khangnguyendac7184]

42:15 The Pyspark now have update the na.fill(). It could only fill up the "value type" matching with "column type". For example, in the video, the professor only could replace all 4 columns because all 4 "column type" is "string" as the same as "Missing value". This being explain in 43:02.

[@adekunleshittu569]

You have to loop through the columns

[@ccuny1]

Yet another excellent offering. Thank you so much.

[@carlosrobertomoralessanche3632]

You dropped this king 👑

[@tech-n-data]

42:11 As of 3/9/24 the na.fill or fillna will not fill integer colums with string. 51:31 aslo df_pyspark.filter('Salary15000')

[@TheBarkali]

Dear Krish. This is only W.O.N.D.E.R.F.U.L.L 😉. Thanks so Much and thanks to professor Hayth.... who showed me the link to your training. Cheers to both of U guys

[@lavanyaballem5085]

Such an Amazing Explanation! you Nailed it KrishNaik

[@graenathan]

Thanks

[@tradeking3078]

At 26:37 , Min and Max values from a column of string data type were not based on the index where they were placed, but it is based on their ASCII values of the words ,their order of characters that are arranged within and the order is ' 0 < 9 < "A" < "Z" < "a" < "z" '. Min will be letter comes first and Max will be which comes last of all the characters, if two similar characters found, it moves to next character and checks and so on ...

[@patrickbateman7665]

True

[@zesky6654]

42:11 - Note: The fill.na function only replaces values of the same type as the replacement. So the code on the screen will only replace the NULL values in the 'Name' column.

[@cherishpotluri957]

Krish Naik on FCC🤯🔥🔥

[@sivakumarrajabather1140]

The session is really great and awesome. Excellent presentation. Thank you.

[@RossittoS]

Great content! Thanks! Regards from Brazil!!!

[@nagarajannethi]

🥺🥺🙌🙌❣️❣️❤️❤️❤️ This is what we need

[@ludovicgardy]

Really great, complete and straight forward course. Thank you for this, amazing job

[@simileoluwaaluko7582]

Great man. Great! 👍🏼👍🏼👍🏼👍🏼

[@mariaakpoduado]

what an amazing tutorial!

[@ujjawalhanda4748]

There is an update in na.fill(), any integer value inside fill will replace nulls from columns having integer data types and so for the case of string value as well.

[@harshaleo4373]

Yeah. If we are trying to fill with a string, it is filling only the Name column nulls.

[@austinchettiar6784]

@@harshaleo4373 so whats the exact keyword to replace all null values?

[@akashk2824]

Thank you so much sir, 100 % satisfied with your tutorial. Loved it.

[@siddhantbhagat7216]

I am very happy to see krish sir on this channel.

[@Dr.indole]

This video is pretty much amazing 😂

[@barzhikevil6873]

For the filling exercise on minute 42:00 aprox, I cannot do it with integer type data, I had to use string data like you did. But them in the next exercise, the one on minute 44:00, the function won't run unless you use integer data for the columns you are trying to fill.

[@Richard-DE]

@@caferacerkid you can try to read with/without inferSchema = True and check the schema, you will see the difference. Try to read again for Imputer.

[@ammadniazi2906]

Where you are setting up the environment variables for spark and Hadoop.

[@spoorthydevineni822]

extraordinary content

[@Pg11001]

At 42:23 there was a function called 'fill' of used and it only replacing the string type datatypes with other string datatype so if you are facing the issue of only replacing the rows data one or two places you go up cell in your python notebook(.ipynb) file and at the reading time set 'inferSchema=False' so it catches the the integral type data that is NULL when they are not defined as integer. Thanks for video.

[@LifeOnTwoWheels369]

Thank you

[@simple_bihari_babua]

This feels like it started in between, was there any previous video to it. Which explained the installation and other processes

[@saiajaygundepalli]

Krish naik sir is teaching wow👍👍

[@estelle9819]

Thank you so much, this is incredibly helpful.

[@raghavsrivastava2910]

Surprised to see Krish Naik sir here ❤️

[@subhajeetchakraborty1791]

sameee me tooo 🤩

[@ronakronu]

nice to meet you krish sir😍

[@dipakkuchhadiya9333]

I like it 👌🏻 we request you to make video on blockchain programing.

[@DuongTran-zh6td]

thank you from Vietnam

[@renadhc68]

Brilliant project based tutorial

[@aliyusifov5481]

Thank you so much for an amazing tutorial session! Easy to follow

[@bhanu242629]

Excellent explanation Bro... :)

[@larsybarz]

Thanks so much man. This is awesome

[@RaviKiran_Me]

At 1:01:09, maximum salary you found is basically the maximum salary of each person in the departments he/she is working and it's not the maximum total salary of each person.

[@thecaptain2000]

in your example df_pyspark.na.fill('missing value').show() replace null values with "missing value" just in the "Name" column

[@javierpatino4142]

Good video brother.

[@arulmouzhiezhilarasan8518]

Impeccable Teaching! Thanks!

[@sukurcf]

26:34 I don't think it's based on index. I just tried changing the indices for min and max values for string. Looks like it's checking the chronological order.

[@DonnieDidi1982]

I was very much looking for this. Great work, thank you!

[@sushilkamble8379]

10:00 | Whoever is getting Exception: Java gateway process exited before sending the driver its port error, Install Java SE 8 (Oracle). The error will be solved.

[@kazekagetech988]

did you solve bro? im facing it now

[@vitazamb3375]

Me too. Did you manage to solve this problem?

[@hariharan199229]

Thanks a ton for this wonderful Masterpiece. It helped me a lot!

[@bansal02]

Really thankful for the video.

[@innovationscode9909]

Massive. This is a GREAT piece. Well done. Keep going

[@saurabhdakshprajapati1499]

Good tutorial, thanks

[@konstantingorskiy5716]

Used this video to prepare for the tech interview, hope it will help)))

[@michasikorski6671]

Is this enought to say that you know spark/databricks?

[@johanrodriguez241]

Finished!. But i still want to see the power of this tool.

[@brown_bread]

One can do slicing in PySpark not exactly the way it is done in Pandas. Eg. Syntax : df_pys.collect()[2:6] Output : [Row(Name='C', Age=42), Row(Name='A2', Age=43), Row(Name='B2', Age=15), Row(Name='C2', Age=78)]

[@programming_duck3122]

Thank you really useful

[@rajatbhatheja356]

However one thing is that take precaution while using collect. collect is an action and will execute your DAG.

[@Jschmuck8987]

Great video. Pretty much simple.

[@ChaeWookKim-vd7uy]

I love this pyspark course!

[@critiquessanscomplaisance8353]

That for free is charity, litteraly! Thanks a lot!!!

[@programming_duck3122]

Min 58:23 to show the maximum salary you should use max instead of sum? sum will work because name is unique, but i found this a bit misleading.

[@amitkumarsaha2424]

Amazing content

[@sanjaygstark]

It's quite impressive 💫✨

[@ofranceable]

Excellent Video.

[@Nari_Nizar]

At 1:09:00 when you try to add Independent feature I get the below error: Py4JJavaError Traceback (most recent call last) in 1 output = featureassembler.transform(trainning) ----> 2 output.show() C:\ProgramData\Anaconda3\lib\site-packages\pyspark\sql\dataframe.py in show(self, n, truncate, vertical) 492 493 if isinstance(truncate, bool) and truncate: --> 494 print(self._jdf.showString(n, 20, vertical)) 495 else: 496 try:

[@crazynikhil3811]

Indians are the best teachers in the world. Thank you :)

[@porvitor]

Thank you so much for an amazing tutorial session!🚀🚀🚀

[@juanviola5825]

you are the best! thanks!

[@jorge1869]

The full installation of PySpark was omitted in this course.

[@praveenkumare2157]

Atlast i found a precious one

[@HariEaswaran98]

Thanks!

[@ujirali4641]

You 5ioooppeweeetyiiop0

[@anassrtimi3015]

Thank you for this course

[@PallabM-bi5uo]

Hi, thanks for this tutorial, If my dataset has 20 columns, why describe output is not showing in a nice table like the above? It is coming all distorted. Is there a way to get a nice tabular format like above for a large dataset?

[@quanoan9823]

sorry how you could see the description of the method, do you have keyboards shortcuts, example in 40:26

[@venkatkondragunta9704]

Hey Krish, Thank you so much for your efforts.. this is really helpful..

[@soundcollective2240]

This is pretty much a very useful video ;) thanks

[@rmehta26]

How do you get the description of a function inline? eg: at timestamp 36:59, while explaining the features under drop

[@santoshturamari]

At 1:22:32, it was not running just because it was detached to any cluster?

[@doreyedahmed]

Thank you so much very nice explanation If you use pyspark, its consider we deal with Spark Apache