Tutorial 2-Pyspark With Python-Pyspark DataFrames- Part 1

github: github.com/kri...
Apache Spark is written in Scala programming language. To support Python with Spark, Apache Spark community released a tool, PySpark. Using PySpark, you can work with RDDs in Python programming language also. It is because of a library called Py4j that they are able to achieve this.
---------------------------------------------------------------------------------------------------------------------------
⭐ Kite is a free AI-powered coding assistant that will help you code faster and smarter. The Kite plugin integrates with all the top editors and IDEs to give you smart completions and documentation while you’re typing. I've been using Kite for a few months and I love it! www.kite.com/g...
Subscribe my vlogging channel
/ @krishnaikhindi
Please donate if you want to support the channel through GPay UPID,
Gpay: krishnaik06@okicici
Telegram link: t.me/joinchat/...
Please join as a member in my channel to get additional benefits like materials in Data Science, live streaming for Members and many more
/ @krishnaik06
Connect with me here:
Twitter: / krishnaik06
Facebook: / krishnaik06
instagram: / krishnaik06

[@tanishasharma3665]

I was recently told by my company to learn pyspark and here is your playlist! Thank you so much!! Btw I have been offered the role of a Data Scientist.....thanks a lot for all your playlists have learnt a lot...keep up the great work!!!

[@krishnaik06]

Congratulations Tanish

[@tanishasharma3665]

@@krishnaik06 cant thank you enough sir.......throughout my internship i watched most of your videos and learnt and it helped me to convert my internship into full time

[@sourindas322]

​@@tanishasharma3665can I get any referral, I already know ETL tools and PySpark, this videos helped me a lot, so I would be entering as a Data Engineer, can ya help me?

[@ayushikaushik1138]

I feel so lucky that I started learning pyspark yesterday and you started this series as well!! Thank you sir!!

[@shrutijain1628]

This was the most simplest and understandable tutorial for pysparks

[@VikashKumar-ty6uy]

Thank you so much for the Pyspark session..Requesting you to kindly complete the playlist as per your availability. I know you have to put lots of effort for this but it is really helpfull for we people who always thrive to learn something new to come out of the box...and you are the reason for that...

[@from-chimp-to-champ1]

Thanks to all the great teachers on youtube, one of which you are. Very helpful! Good luck and all the best!

[@utsavdatta7532]

Great series..eagerly waiting for MLib...u deserve more subscribers

[@SMHasan9]

Your way of teaching is excellent, Krish !

[@mohitupadhayay1439]

Why hasn't this guy go 10mn subscribers yet? Kudos to you Bhai!

[@swaraj2235]

Excellent explanation Krish. Thank you very much.

[@biswanandanpattanayak6083]

Nice session. Much awaited vedio

[@yogaandernostlich1007]

Big data playlist..krish make this as good as ml

[@AprajitaPandey-of2kf]

df_pyspark.describe().show() at 11:32. couldnt understand this. why is it showing null?

[@AprajitaPandey-of2kf]

Hi @krish sir, can u please tell us where all videos of pyspark are available?

[@Fazshim369]

Thanks!

[@1981Praveer]

@11:39 - it does not look like it s due to the index, I think internally it sorts the strings and then show min as lower and max as highest point in the sorted array

[@nikhileshyoutube4924]

Sir can u give the link description of u r earphones ❤️

[@Nathisri]

@Krish: I'm not able to create the spark session. I get the error 'No module named spark'

[@hardikshah9542]

what to do wehn we have large integer and it shows the values in scientific format but want it in normal froamt what to do?

[@romesupaila1864]

if getting two list, list of headers and list of body ,datatype also mismatch , in this time ,how can create CreateDataFrame() if any passable or any suggestions

[@deekshamalik8813]

While displaying a particular column, I am getting this error : cannot resolve '`'Name'`' given input columns: [Age , Experience, Name ]. How to resolve it?

[@adshakin]

Amazing, you made it so simple. thanks

[@kaxxamhinna5044]

Thank you very much for the pedagogy 🙏

[@piyushjain5852]

good Tutorial Sir, it was really helpful to clear the basics!

[@bhargavikoti4208]

Explained neatly..thank you👍👍

[@abhisheks.2553]

Sir , How to read xml file in pyspark and write it to csv. if we dont know the roottag and rowtag of it.

[@alanhenry9850]

Can u also make tutorials on kafka

[@abhishekgopinath4608]

Yes please do pyspark with kafka streaming tooo

[@jondoe501]

Yes.. that would be helpful.

[@mjchalla]

inferSchema=True does not work with JSON files and read.json("path"). What is the alternative?

[@ashwani14725]

Having this issue Java gateway process exited before sending its port number

[@lalawinter4056]

Sir, how do we display the number in describe() with 2 decimals point?

[@stephenmartin6995]

Thank you for this very clear explanation !

[@pawankumarraj6653]

In Google collab while reading file it shows an error. Is there any other way we can proceed. Pls Suggest Code: file_path = 'gps_data (1).csv' df = spark.read.csv(file_path, header=True, inferSchema=True) df.show() It shows an error: AttributeError: 'function' object has no attribute 'read'

[@XiwithHighPing]

multiple columns can be renamed using .withColumn Renamed. for example: covid4=(covid.withColumnRenamed('Country/Region','Nation') .withColumnRenamed('Province/State','State') .withColumnRenamed('Deaths','Deceased'))

[@RangaSwamyleela]

Sir can u teach all the topics of pyspark

[@manishtomar8797]

Hi, Thanks for uploading this playlist. It's quite informative. Just a question, I see tutorial #26 and tutorial #1-8. Are the tutorials missing from 9-25 ? Can you please check.

[@khushboojain3883]

What I noticed is 'Describe()' function gives the data types of Age and Experience as Strings, not Integer. But 'Describe' method gives the correct data types as Integer. In addition, df_pyspark.describe.show() does not work, but df_pyspark.describe().show() works successfully.

[@joeljoseph26]

Krish, you're a good human! :) Thank you!

[@MrinalDas3107]

I am using a mac and i am new to Python. I get "This context might be already existing" error while creating the session. Can someone help. TIA

[@gorangkhandelwal4790]

Hi everyone, how can we add a new independent column (Eg: Emp ID) in existing dataframe ?

[@reenasheoran893]

Hey Krish ... what I realized from this is that SQL knowledge will help to learn spark quickly :)

[@gokulyc]

Is JDK enough to make pyspark work ? I am facing issues even after installing JDK and adding to JAVA_HOME.

[@satishkumar-ir9wy]

Thanks for wonderful tutorials while creating a spark session in my jupyter notebook, i am getting below error "PySparkRuntimeError: [JAVA_GATEWAY_EXITED] Java gateway process exited before sending its port number." can anyone suggest solution to this, i am a newbie to spark.

[@ajaysinha1007]

What is aapName here.... ?

[@ajaysaikiranpenumareddy9809]

Java gateway process exited before sending its port number,this is the error i got sir while initiating the session

[@krishnaik06]

Create a new environment and install Pyspark

[@ajaysaikiranpenumareddy9809]

@@krishnaik06 Thank you sir

[@sanjaykale5001]

Data pipeline stored in Aws S3 bucket using pyspark ,sir, plz isme video banao

[@krishnaik06]

Sab banega don't worry

[@ahimsaram5361]

if anybody knows how to connect sql server to pyspark...i use spark sql ...but i got some error...i want to fetch the data from the database

[@shwetaraghav8055]

Sir please suggest, institute for sas training &certification at free of cost.

[@nigamaveena4211]

Sir what's the difference between pandas and pyspark? Y do we have to use pyspark. Are there any advantages of using pyspark?

[@krishnaik06]

When u have huge dataset u can use Pyspark in distributed cluster to execute your code

[@nigamaveena4211]

@@krishnaik06 tq sir

[@navejpathan]

Please make video on MITO library for data preprocessing 🙏 waiting for your reply

[@ritizsaini2106]

Great content! Thanks for this wonderful series :)

[@shravanibadadha9552]

Hai Krish,How to add new column which is of string datatype?

[@gorangkhandelwal4790]

Hi Shravani, if you found any method to add a new independent column (eg: emp_id), please share in comments

[@tomparatube6506]

Please help: When I ran "spark=SparkSession.builder.appName('Dataframe').getOrCreate()", I get "Exception: Java gateway process exited before sending its port number" Anybody can point out what am I missing or need to install / modify? Thanks a ton. Love your tutorials.

[@poornimabhola341]

Yes i also got this error how to solve this

[@deekshamalik8813]

@@poornimabhola341 You'll have to install Java, Spark, hadoop & try to keep them in following directories & then run following code in python : import os import sys os.environ["PYSPARK_PYTHON"] = "/opt/continuum/anaconda/bin/python" os.environ["JAVA_HOME"] = "C:/Java/jre" os.environ["SPARK_HOME"] = "C:/Spark/spark-3.1.2-bin-hadoop3.2" os.environ["PYLIB"] = os.environ["SPARK_HOME"] + "/python/lib" sys.path.insert(0, os.environ["PYLIB"] +"/py4j-0.9-src.zip") sys.path.insert(0, os.environ["PYLIB"] +"/pyspark.zip")

[@techplay6964]

Install java SDK and restart the system and try again

[@ayaansk99]

Can any one explain me that how to save Pyspark.sql output to csv

[@bilalpanhalkar6944]

Pretty much simple😍😅

[@hariniprabakaran5434]

waiting to complete this series. Can you give 2 real time projects at the end

[@prashanthpaul2713]

This is great! Thank you! :)

[@ratnajyotibhowmick9801]

Thanks for starting PySpark playlist! Also, could you please share your exclusive explanation on Attention and BERT not by someone else. Thanks.

[@cloudlover9186]

Great content , please let me know where i can download csv files?

[@sahityamamillapalli6735]

I think in this video you have not covered about data sets please check once sir

[@prankushsharma2588]

WHEN MACHINE LEARNING AND DEEP LEARNING CLASS WILL RESUME??😒

[@Akshay50826]

Thank you Krish !!

[@martian.07_]

Wow really helpful.

[@bodhisattadas304]

RuntimeError: Java Gateway process excited before sending its port number

[@dendi1076]

music at start of video sounds like a pokemon ghost movie theme

[@SuperShaneHD]

Data frame is basically a RDD with some schema

[@shrikantdeshmukh7951]

Your doing great job

[@jvtalks1]

Min and Max of a string column is taken based on alphabetical order I guess

[@PramodKhandalkar5]

Superb

[@yogeshrashmi]

Informative

[@tanushreenagar3116]

nice sir perfect

[@maheshpatkar886]

pretty much simple 🤣🤣🤣🤣

[@chillagundlavamshi8504]

In this code line 30 adding columns in data frame lo it's only adding numbers 'df_pyspark.withColumn('Increment of Salary',df_pyspark['Salary']+1500).show()' - excute this code df_pyspark.withColumn('Increment of Salary',df_pyspark['Salary']+10% (['Salary']).show() -- it's not excuted solve this one