Tutorial 1-Pyspark With Python-Pyspark Introduction and Installation

Apache Spark is written in Scala programming language. To support Python with Spark, Apache Spark community released a tool, PySpark. Using PySpark, you can work with RDDs in Python programming language also. It is because of a library called Py4j that they are able to achieve this.
⭐ Kite is a free AI-powered coding assistant that will help you code faster and smarter. The Kite plugin integrates with all the top editors and IDEs to give you smart completions and documentation while you’re typing. I've been using Kite for a few months and I love it! www.kite.com/get-kite/?...
Subscribe my vlogging channel
/ @krishnaikhindi
Please donate if you want to support the channel through GPay UPID,
Gpay: krishnaik06@okicici
Telegram link: t.me/joinchat/N77M7xRvYUd403D...
Please join as a member in my channel to get additional benefits like materials in Data Science, live streaming for Members and many more
/ @krishnaik06
Connect with me here:
Twitter: / krishnaik06
Facebook: / krishnaik06
instagram: / krishnaik06

[@rlmclaughlinmusic]

Everything about this series is perfect. The pace, the information, and the clarity of the descriptions are as good as it gets. I've watched about 4-5 pyspark tutorials, from various instructors, and they don't even come close to the greatness of these videos. Thank you for providing such top notch content and using a no-nonsense approach. I thoroughly enjoyed these and learned a lot.

[@lananajera1081]

I am 9 minutes into the first video and let me tell you it is already better than the last 10 I have tried. It's great for real beginners like me and challenging enough too. Thank you for posting these!!

[@AInamedMia]

We can like these videos even before we see them cause we know they are bound to be extremely useful.

[@amanmehrotra44]

Sir ek hi dil hai, kitni baar jeetenge ! Once again hats-off to your efforts in uplifting the entire data science community across the globe.

[@aryanraj768]

the kind stuff that he taught is already there on the doc which is readable by anyone in the world

[@rhevathivijay2913]

Really When i am doing search in ur encyclopedia playlist,I miss this..Thank you for uploading sir

[@Abhilash3824]

Was eagerly waiting for this playlist. Thank you so much Krish! 🙂

[@vaibhavtiwari1084]

I didn't realise when those 16 minutes ended...interactive n smooth!!

[@muhammadsalmanhassan7544]

What we can divide dataset into multiple chunks in pandas and train the model on it is this good practice or bad practice?

[@user-vb7im1jb1b]

Thanks for this video. For learning purposes on my own computer, do I need to install apache.spark (spark-3.4.1-bin-hadoop3.tgz) to be able to run spark scripts/notebooks, or just pip install pyspark on my python environment?

[@deveshkumar3504]

I desperately needed this course ! Thanks a lot !

[@prashanthpaul2713]

So glad that you started this new series, Krish! Looking forward for new videos in this series. Any idea when you would be uploading? :)

[@ankushv2642]

can you tell me how he got that jupyter screen where he is installing the pyspark

[@ganeshkalbhor3928]

Hi @krish, I am getting ' RuntimeError: Java gateway process exited before sending its port number ' this error while starting spark session. could you please help me to resolve this

[@reenasheoran893]

Hey Krish, this nullable=True means it's not a primary key as we are working with SQL dataframe?

[@pyclassy]

Hi Krish I am getting a Py4j error can you upload the reuirements.txt file along with the python version so that I can start

[@yadavanubhav005]

Hi Krish, any idea why my code same as yours is not getting executed. I installed jupyter notebook using anaconda. I wish I could have pasted the screenshot here.

[@megaranvirsingh]

Hi Krish..Thanks for starting session on pyspark Please address below issue: I am using currencies csv file and it has around 40 columns while using df_currencies.show() -> the df is displaying record ,but these records are not readable as they are conjusted as not showing in tabular form. Please read some df who has around 30-40 columns and check at your end are you getting same,if yes->please share solution of above. Thanks,hope you will help in this.

[@ananyanayak7509]

Hello Sir, I got error as :- "Exception: Java gateway process exited before sending its port number" while executing line number 5. How can I resolve it ?

[@nlokesh1986]

Sir, how are you getting the automatic suggestions in jupyter notebook.. please help me, so that i can do the same with my system. Thanks alot

[@akashchauhan8436]

How to create a timeseries in pyspark. Say for example I have a column named start_date wit the format (YYYY-MM) for some event, but its not continuous, i.e. I have 2015-01, 2015-04, 2015-07. Then how do I fill the missing dates between them and assign the values to other columns as 0 in pyspark? It was easy in pandas where I could just set this column as index and then resample the dataframe.

[@hareshmu2105]

Hi Krish, you are awesome in explaining difficult topics

[@eswaragopal335]

Most awaited video from u... Thanks for the starting this session

[@arjunsubramaniyan1675]

Much waited playlist!!

[@MukeshThakur-qp5ft]

when i am trying to create Spark Session getting this error "RuntimeError: Java gateway process exited before sending its port number". Help me in resolving this please

[@amanpatkar7009]

I wanted to start with big data... Hope this course will give us understanding... Thanks sir

[@VP_SOTWMC]

When I am adding SparkSession code, I am getting below error. Exception: Java gateway process exited before sending its port number How to fix this

[@user-qu8dk3ho7u]

Hi Krish, Thanks for your videos, I dont know why I get ("Non type ) after correcting the header for pyspark and dose not show me the Schema.

[@hardikvegad3508]

It's been ages...... I had waited for this from you krishhhhhhh😭😭😭😭😭🤩...Thank you💥

[@ManoharJishu]

facing this error while creating sparkSession: RuntimeError: Java gateway process exited before sending its port number Any suggestions on this?

[@dileepk1740]

Hi Krish, I have created new environment for pyspark !pip install pyspark import pyspark are successful but import pandas as pd give error as: No module named 'pandas' what needs to do ?

[@ashwinideshmukh2198]

sir . after checking type for df_spark i am getting nonetype error .for pandas it is giving correct answer

[@abhishekpawar1207]

Do i have to install linux with virtual box in order tp work with pyspark?

[@soumitadalal6130]

FileNotFoundError: [WinError 2] The system cannot find the file specified getting this error after having jdk.1.8 any solutions?

[@alihaiderabdi9939]

sir waiting for new playlist from a longtime and here it came!!!!

[@vallimuthaiyah5098]

Can you please let us know the advantages of using pyspark dataframe over pandas dataframe

[@suneethach4052]

Hi Krish, thank you so much for informative video 👍.

[@user-bq9tq1bc8b]

Hello sir i am not able to create Pyspark session, while i am generating session i am getting follwing error :: Py4JError: org.apache.spark.api.python.PythonUtils.getPythonAuthSocketTimeout does not exist in the JVM can you give me solution of this problem

[@amberkataria9408]

spark session command : spark = SparkSession.builder.appName('Practiceee').getOrCreate() is taking infinite time. Not able to run code further as it kept on running. What is the solution for this?

[@islamicinterestofficial]

please make a video how to install pyspark. We installed it but its not importing on jupyter notebook. On terminal, its importing fine

[@avirajankitjain256]

While using this code spark=SparkSession.builder.appName('Practice').getOrCreate() RuntimeError: Java gateway process exited before sending its port number. Can someone suggest how to fix.

[@kjayeshnaidu6012]

I did the follow the things in the video but when i am seeing my spark version it is show me version 2.4.5 can anybody please help me with that

[@swaraj2235]

Very much useful.. Thanks Krish.

[@adityamathew3398]

sir i am getting an error, for the code :- spark = SparkSession.builder.appName('Practise').getOrCreate() RuntimeError: Java gateway process exited before sending its port number

[@archanapereira1333]

Do we have real time project from iNeuron team or from you so that we can include in our CV ?

[@pavneetarora5935]

Sir,can you please provide the code for kmeans clustering using pyspark in databricks from scratch?

[@bhargavram8830]

While creating the spark session this error showed up. Exception: Java gateway process exited before sending its port number. Does anybody know how to fix this error???

[@sakthikumaranvg2668]

I am getting some errors while creating spark session PySparkruntime error[Java gate way exited] Java gateway process exited before sending it's port number

[@darshanayenkar]

i am getting this error: [WinError 10061] No connection could be made because the target machine actively refused it can you plz help me to solve?

[@raghuls9010]

i get spark output like this further unable to read the dataset

[@AakarshanJha]

Java gateway process exited before sending its port number I am getting this error while setting spark session builder. Can anyone help me out in this?

[@ankushv2642]

While creating the spark session getting this below error - Py4JJavaError Traceback (most recent call last) Cell In[12], line 1 ----> 1 spark=SparkSession.builder.appName("Pratice").getOrCreate() Can someone please help??

[@mandarbirwadkar]

Hi Krish i am getting error while accessing file from D drive i kept csv in D drive .

[@mrraju9986]

When I was creating pyspark seeion it's through an erro like this java gateway process exited before sending it's port number

[@ansonnn_]

Have been searching for good PySpark tutorials and this turned up 👍 Thanks!

[@singhjagbir1210]

I am stuck while creating Spark Session getting this error PySparkRuntimeError: [JAVA_GATEWAY_EXITED] Java gateway process exited before sending its port number.. Please help

[@sanjaybohr1058]

how to resolve this "Exception: Java gateway process exited before sending its port number"

[@akshaygane159]

Was eagerly waiting for this 😂. What's in our mind in your playlist 😂. Thanks. Dedicated playlist for pyspark or extension to ML playlist. Editing as found separately created playlist

[@deveshsharma8407]

Sir last two lines code are not working in my system it shows ---- AttributeError: 'NoneType' object has no attribute 'printSchema' everything is all right even i restarted kernel

[@bryandiaz__]

Hello, I keep getting this error and can't move past it could you kindly suggest please File ~\AppData\Local\Programs\Python\Python311\Lib\site-packages\pyspark\java_gateway.py:106, in launch_gateway(conf, popen_kwargs) 103 time.sleep(0.1) 105 if not os.path.isfile(conn_info_file): --> 106 raise RuntimeError("Java gateway process exited before sending its port number") 108 with open(conn_info_file, "rb") as info: 109 gateway_port = read_int(info) RuntimeError: Java gateway process exited before sending its port number

[@varunanguralia252]

I am getting following error: Exception: Java gateway process exited before sending its port number when i am running this code: spark=SparkSession.builder.appName("Practice").getOrCreate()

[@pavankonakalla4668]

Hi,im getting this error can any one help.Like after reading the csv file i'm trying to print executing show command i'm getting something weird data in the c0,c1 columns.Any one facing same issue

[@bhaskararya5901]

my pyspark session is still running for last 2 hours. what to do, i tried other method like update my pip,etc. Did anyone face the same problem? any solution is appreciated.

[@AbhishekTiwari-xw7ux]

AnalysisException: Path does not exist: file:/C:/Users/abhi/test.csv How to solve this issue ....even i keep my file in the same location

[@annikakumar]

type(df_pyspark) is always showing nonetype for me. kindly help me how to rectify the error

[@farhaanarshad5924]

Amazing Playlist. Thanks so much! Was looking for a good tutorial for Introduction into PySpark :)

[@yogeshpathak5777]

Trying to run code in jupyter ,but always getting errors.Dont know how to access file from local in jupyter

[@akhilverma9773]

when I run spark = SparkSession.builder.appName('Practise').getOrCreate(). I am getting "Java gateway process exited before sending its port number" this error

[@sandeepnelwade]

Hi Krish I got error when creating sparksession, how I connect with you

[@batista1228mohd]

facing "Install Java and set JAVA_HOME to point to the Java installation directory" error in virtual env

[@ViratKohli-gh6ic]

Intro soundtrack jabardast hai bhai..also content bhi

[@alihaiderabdi9939]

sir, which laptop you are using with 64 gb ram?

[@mullaibharathi8255]

Java gateway process exited before sending its port number. I am getting this error

[@moderncollectionkalyan3280]

For 8 GB ram is it suitable or require more RAM

[@optimistic_guy313]

I am having some problems with thinking. Can you share how you tackle thinking and do fast thinking?

[@jatinsharma9101]

Hi Krish, RuntimeError: Java gateway process exited before sending its port number I was getting this error in SparkSession.builder.appName('practice').getOrCreate() Please help me

[@lokeshkambam9416]

sparksession is taking lot of time to create. Is there any solution for this?

[@harshvardhansingh3862]

Getting "PySparkRuntimeError: [JAVA_GATEWAY_EXITED] Java gateway process exited before sending its port number." error while creating spark session. tried many ways to fix but still getting the same problem

[@sachinkapoor2424]

Sir ek hi toh dil hai kitni baar jitoge🙏

[@SynonAnon-vi1ql]

Hi Krish! Great tutorial! Thanks for this! One (probably stupid) question and I'm a novice here. How did you enable the auto-suggest functionality in your jupyter notebook? Mine doesn't work. Could you please help? Thank you!

[@ajaysaikiranpenumareddy9809]

Most awaited playlist

[@suhass6628]

Most awaited!!!!!!! it was music to my years when he said Mlib 0:40

[@ankitbhatia3387]

Yes, more Videos on this please.

[@strangersthings7772]

Java gateway process exited before sending its port number -getting this error

[@areebakhtar9841]

Hi I am getting following error while executing spark = SparkSession.builder.appName('learning').getOrCreate() RuntimeError: Java gateway process exited before sending its port number

[@ujjwalgoel6359]

after wasting 2 hours on youtube at last found someone telling from scratch and what i was looking for

[@aaryanrawat9930]

I am running on jupyter in browser, files are importing but it gives error while creating session.. ERROR NAME - py4jjavaerror. Anyone know how to solve this ?

[@rashmikadre8900]

Omg!! I have been literally been waiting for this!! Krish u r the man!!!

[@Nishanthts]

Thanks for this .. kindly provide complete playlist

[@payelpanja7125]

will wait for more videos :-)

[@jakirajam]

Can anyone tell, what we learn to implement DWH with pyspark.

[@nishadseeraj7034]

Has anyone gotten this error trying to create a Spark session : "Java gateway process exited before sending its port number." I'm using windows btw

[@shahidnawazkhan9806]

Hi sir, Love your videos. i have a question. While you running the spark session, have you installed Hadoop already and set its path or you using any standalone cluster? Can we run this code by just installing pyspark in our python? or we also need cluster connectivity?

[@bhavanasharma3044]

Spark doesn’t compulsorily require hadoop. It can work without it as well. But if u are looking for multinode processing then hadoop is required with a resource manager like YARN and HDFS .

[@guillermoalcantaragonzalez6532]

Krish es el "Julio profe" de mi vida profesional.

[@deepaktamhane8373]

great sir ...informative video series...how to add specific value to specific cell one by one in column

[@subhajitdey4483]

When I am trying it on Jupyter it's showing: " RuntimeError: Java gateway process exited before sending its port number". What should I do now? I have tried it on Python idle also, but same error. Help if anyone has any solution.

[@SaiTeja-ob6zg]

Same problem.. did u get any solution?

[@abhijitpaul0212]

Its 2024 Sir, and still your video contents are unmatchable. My bad luck is that the moment I joined your iNeuron course, you separated away from it, but my only reason joining the course was to learn from only you! SAD :(

[@dheerendrasinghbhadauria9798]

I am getting an error " Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext. : java.lang.NoClassDefFoundError: Could not initialize class org.apache.spark.storage.StorageUtils$ "

[@KARANKUMAR-qr9nj]

Great work. You are awesome :)

[@ShahnawazKhan-xl6ij]

Awesome, 👌👍

[@kalpeshghadigaonkar3388]

Waiting for this for so long!

[@salmansiddiqui8893]

Getting below error after running spark=SparkSession.builder.appName('Practise').getOrCreate(), > Py4JError: org.apache.spark.api.python.PythonUtils.isEncryptionEnabled does not exist in the JVM