Tutorial 1-Pyspark With Python-Pyspark Introduction and Installation

[@RedShipsofSpainAgain]

Timestamps: 6:45 Create new environment and install spark via pip install 7:13 importing pyspark 9:34 Import SparkSession 9:47 Create SparkSession ...

[@parammani4717]

Hi first time looking this video, where he is creating new environment. Is this any cloud platform

[@rlmclaughlinmusic]

Everything about this series is perfect. The pace, the information, and the clarity of the descriptions are as good as it gets. I've watched about 4-5 pyspark tutorials, from various instructors, and they don't even come close to the greatness of these videos. Thank you for providing such top notch content and using a no-nonsense approach. I thoroughly enjoyed these and learned a lot.

[@lananajera1081]

I am 9 minutes into the first video and let me tell you it is already better than the last 10 I have tried. It's great for real beginners like me and challenging enough too. Thank you for posting these!!

[@arjunsubramaniyan1675]

Much waited playlist!!

[@suhass6628]

Most awaited!!!!!!! it was music to my years when he said Mlib 0:40

[@ujjwalgoel6359]

after wasting 2 hours on youtube at last found someone telling from scratch and what i was looking for

[@AInamedMia]

We can like these videos even before we see them cause we know they are bound to be extremely useful.

[@vaibhavtiwari1084]

I didn't realise when those 16 minutes ended...interactive n smooth!!

[@arjunsai08]

Krish I am a big fan of yours. You are an amazing teacher and have taught me numerous concepts in Data Science. Thanks a lot for the social service you do!!

[@sachinkapoor2424]

Sir ek hi toh dil hai kitni baar jitoge🙏

[@amanmehrotra44]

Sir ek hi dil hai, kitni baar jeetenge ! Once again hats-off to your efforts in uplifting the entire data science community across the globe.

[@aryanraj768]

the kind stuff that he taught is already there on the doc which is readable by anyone in the world

[@Nishanthts]

Thanks for this .. kindly provide complete playlist

[@alihaiderabdi9939]

sir waiting for new playlist from a longtime and here it came!!!!

[@damodharratnamthappeta2022]

much waited playlist

[@rhevathivijay2913]

Really When i am doing search in ur encyclopedia playlist,I miss this..Thank you for uploading sir

[@mbmathematicsacademic7038]

Amazing😂one thing about your channel is that I get confused whenever I get here,I wanted to learn Feature Engineering for the day here I am enjoying pyspark

[@deveshkumar3504]

I desperately needed this course ! Thanks a lot !

[@hardikvegad3508]

It's been ages...... I had waited for this from you krishhhhhhh😭😭😭😭😭🤩...Thank you💥

[@ajaysaikiran2196]

Most awaited playlist

[@namanvyas9433]

Thanks man, just wanted to start with pyspark.

[@ViratKohli-gh6ic]

Intro soundtrack jabardast hai bhai..also content bhi

[@eswaragopal335]

Most awaited video from u... Thanks for the starting this session

[@rajeshkumarmandal8422]

Thanks for this, but i am getting error while running the spark and the error is "Exception: Java gateway process exited before sending its port number". Can you tell me how to resolve it.

[@sahilshetty8640]

Hi, I faced the same issue and found the solution...all u got to do is download JDK version 8 and set it to path and make sure you uninstall any other versions of Java from your system. Let me know if u need any further help. Good luck!

[@anuvratshukla7061]

@@sahilshetty8640 How to set path after downloading JDK?

[@pankajdhut46]

​@@sahilshetty8640I do set the path still showing the same error

[@chinmayagokhale6341]

How to resolve this error..

[@chinmayagokhale6341]

@sahilshetty8640 how to resolve this error

[@Abhilash3824]

Was eagerly waiting for this playlist. Thank you so much Krish! 🙂

[@rashmikadre8900]

Omg!! I have been literally been waiting for this!! Krish u r the man!!!

[@prashanthpaul2713]

So glad that you started this new series, Krish! Looking forward for new videos in this series. Any idea when you would be uploading? :)

[@ankushv2642]

can you tell me how he got that jupyter screen where he is installing the pyspark

[@shashikantchikhle9128]

Please advise RuntimeError: Java gateway process exited before sending its port number

[@shashanktiwari133]

can you share the resolution for this error, i am facing the same issue

[@sanketsingh6881]

@@shashanktiwari133 Any luck on this issue?

[@awaizmansoor3127]

You should have the latest java jdk and python installed on your pc first.

[@sushmagoel7854]

The command "!pip install pyspark" got successfully run I got the following error after the command import pyspark "ModuleNotFoundError: No module named 'pyspark'" I had created a new environment in Anaconda and installed pyspark in it. The above error got resolved by running "pip install pyspark" command

[@manuelmeekattukulam]

This worked for me. Thanks!

[@ektaaggarwal3471]

Thanks Sushma! I was encountering the same error since last 2 days and was about to give up learning PySpark. Your comment has saved my learning :)

[@AbhijitPaulYT]

Its 2024 Sir, and still your video contents are unmatchable. My bad luck is that the moment I joined your iNeuron course, you separated away from it, but my only reason joining the course was to learn from only you! SAD :(

[@marathig0795]

Many thanks bro...keep posting such type of videos

[@swaraj2235]

Very much useful.. Thanks Krish.

[@guillermoalcantaragonzalez6532]

Krish es el "Julio profe" de mi vida profesional.

[@wellpaidmasonnothingisfree1085]

Next video please...🤩

[@akshaygane159]

Was eagerly waiting for this 😂. What's in our mind in your playlist 😂. Thanks. Dedicated playlist for pyspark or extension to ML playlist. Editing as found separately created playlist

[@sreekanthn1023]

Hi Sir, When I am trying to import sparksession and sparkcontext it is throwing an error. The error is module Java.base doesnot support sun.nio.ch to unnamed module. Could you please resolve this Thank you

[@ryandraanditto3665]

same with me, can anybody help us?

[@ansonnn_]

Have been searching for good PySpark tutorials and this turned up 👍 Thanks!

[@AprajitaPandey-of2kf]

Hi @krish sir, can u please tell us where all videos of pyspark are available?

[@ankitbhatia3387]

Yes, more Videos on this please.

[@maigan007]

Bro thank you! I swear other videos made it so complicated!

[@ShahnawazKhan-xl6ij]

Awesome, 👌👍

[@neerajkhadilkar2329]

if possible can you make video on the theoretical concept of spark such as architecture of spark and so on

[@sanroymuruh6583]

7

[@biswanandanpattanayak6083]

It's very important playlist. One querry about clustering. Which I faced in interview. How can you know which cluster is good??

[@lucianomilo358]

Dont know if anyone gives a shit but if you're stoned like me during the covid times you can watch all of the latest movies on InstaFlixxer. I've been watching with my brother for the last couple of days =)

[@bryankristian1428]

@Luciano Milo Yea, been using Instaflixxer for years myself :D

[@m2editz816]

I really appreciate your videos. One thing which is missing is that your tutorial starts with python implementation only. If you create a video on how to configure spark in a system and connect with python, that would be a great help

[@awaizmansoor3127]

Can't agree more

[@sanjeevkumarsingh4939]

Hi Krish, Thanks for these amazing videos. I am getting error "RuntimeError: Java gateway process exited before sending its port number" during creation of session in jupyter.

[@girishreddyedula2667]

was this resolved? If yes please tell me how

[@suneethach4052]

Hi Krish, thank you so much for informative video 👍.

[@amanpatkar7009]

I wanted to start with big data... Hope this course will give us understanding... Thanks sir

[@hareshmu2105]

Hi Krish, you are awesome in explaining difficult topics

[@asawanted]

Sir I am having issue when calling SparkSession.builder on local machine. The cell runs forever and nothing happens. I created a new environment and repeated the process. Still the cell gets stuck and doesn't proceed. Sir please reply

[@balachandar3587]

you need to install jdk 8(Uninstall if any other is being used). after that restart your laptop. this should fix the problem.

[@asawanted]

How is jdk related to Python and jupyter?

[@balachandar3587]

You need java do execute spark

[@wellget6]

22/11/13 20:53:00 ERROR Utils: Process WrappedArray(getconf, PAGESIZE) exited with code 1: 22/11/13 20:53:00 WARN ProcfsMetricsGetter: Exception when trying to compute pagesize, as a result reporting of ProcessTree metrics is stopped WARN NativeCodeLoader: Unable to load native-hadoop library for your platform

[@farhaanarshad5924]

Amazing Playlist. Thanks so much! Was looking for a good tutorial for Introduction into PySpark :)

[@yogaandernostlich1007]

Yes.. Full playlist

[@fluffybinibunny]

Hello Krish I am getting the following error - "Java gateway process exited before sending its port number" when performing this code "spark = SparkSession.builder.appName("Practice").getOrCreate()" I have installed jdk as well and still it gives the same error please let me know how to resolve it as I have searched in net and I am not getting much of help. Thanks.

[@karan9671]

Install JDK 8 and define JAVA_HOME under user variables.

[@abhimanyukspillai6572]

Please help! While creating the spark session using the 'getOrCreate()' I am getting the below mentioned error "Exception: Java gateway process exited before sending its port number" Please help to solve this

[@AnkitGupta-rj4yy]

same error with me

[@yassienmasoud3822]

I get the same error, how can fix it please

[@jaiyeshchahar]

Same error

[@abhimanyukspillai6572]

@@jaiyeshchahar Seems I have got rid of that error somehow. Instead of creating the session on a different environment I tried to do it on my base environment and the error didn't come (don't know why!). Anyways Please try it out.

[@abhimanyukspillai6572]

@@yassienmasoud3822 Seems I have got rid of that error somehow. Instead of creating the session on a different environment I tried to do it on my base environment and the error didn't come (don't know why!). Anyways Please try it out.

[@vallimuthaiyah5098]

Can you please let us know the advantages of using pyspark dataframe over pandas dataframe

[@Naina_Sharma_NS_Official]

Hi @krish, I am getting ' RuntimeError: Java gateway process exited before sending its port number ' this error while starting spark session. could you please help me to resolve this

[@islamicinterestofficial]

please make a video how to install pyspark. We installed it but its not importing on jupyter notebook. On terminal, its importing fine

[@MBayat-l4e]

Hi Krish, Thanks for your videos, I dont know why I get ("Non type ) after correcting the header for pyspark and dose not show me the Schema.

[@MukeshThakur-qp5ft]

when i am trying to create Spark Session getting this error "RuntimeError: Java gateway process exited before sending its port number". Help me in resolving this please

[@VP_SOTWMC]

When I am adding SparkSession code, I am getting below error. Exception: Java gateway process exited before sending its port number How to fix this

[@awaizmansoor3127]

You should have the latest version of the java jdk installed in your pc.

[@PritiModi-o8o]

Hello sir i am not able to create Pyspark session, while i am generating session i am getting follwing error :: Py4JError: org.apache.spark.api.python.PythonUtils.getPythonAuthSocketTimeout does not exist in the JVM can you give me solution of this problem

[@pyclassy]

Hi Krish I am getting a Py4j error can you upload the reuirements.txt file along with the python version so that I can start

[@annikakumar]

type(df_pyspark) is always showing nonetype for me. kindly help me how to rectify the error

[@deveshsharma8407]

Sir last two lines code are not working in my system it shows ---- AttributeError: 'NoneType' object has no attribute 'printSchema' everything is all right even i restarted kernel

[@salmansiddiqui8893]

Getting below error after running spark=SparkSession.builder.appName('Practise').getOrCreate(), > Py4JError: org.apache.spark.api.python.PythonUtils.isEncryptionEnabled does not exist in the JVM

[@singhjagbir1210]

I am stuck while creating Spark Session getting this error PySparkRuntimeError: [JAVA_GATEWAY_EXITED] Java gateway process exited before sending its port number.. Please help

[@premsaikarampudi3944]

Hi @krish Naik, When i import pyspark, i get an error "Kernal died" can you suggest what to do ?

[@rohansrivastwa827]

for me it is not working also...not able to install pyspark using the command -> !pip install pyspark

[@premsaikarampudi3944]

@@rohansrivastwa827 Hey, try re-installing anaconda. It worked for me

[@saisankar25]

installing pyspark and when I am running the code , I am getting an error-" Java gateway process exited before sending its port number". I have set the path of java in environment variable, still getting the same error. If you could assist that would be great , so that we can start testing other videos

[@krishnaik06]

Install findspark

[@aviranawat]

@Shubhangi Sakarkar # install jdk 8 and do this. import os os.environ["JAVA_HOME"] = "C:\Java\jdk1.8.0_291" os.environ["PATH"] = os.environ["JAVA_HOME"] + "/bin:" + os.environ["PATH"]

[@UttkarshJainmeb]

@@aviranawat will it work for mac also these arguments values

[@sklshappy9806]

Hi sir, Love your videos. i have a question. While you running the spark session, have you installed Hadoop already and set its path or you using any standalone cluster? Can we run this code by just installing pyspark in our python? or we also need cluster connectivity?

[@bhavanasharma3044]

Spark doesn’t compulsorily require hadoop. It can work without it as well. But if u are looking for multinode processing then hadoop is required with a resource manager like YARN and HDFS .

[@yogeshpathak5777]

Trying to run code in jupyter ,but always getting errors.Dont know how to access file from local in jupyter

[@kapilbisht1119]

Hi Krish, After installing Spark When I run spark session, I m getting below error. RuntimeError Traceback (most recent call last) Cell In [11], line 1 ----> 1 spark = SparkSession.builder.appName('Practice').getOrCreate() File ~\AppData\Local\Programs\Python\Python310\lib\site-packages\pyspark\sql\session.py:269, in SparkSession.Builder.getOrCreate(self) 267 sparkConf.set(key, value) 268 # This SparkContext may be an existing one. --> 269 sc = SparkContext.getOrCreate(sparkConf) 270 # Do not update `SparkConf` for existing `SparkContext`, as it's shared 271 # by all sessions. 272 session = SparkSession(sc, options=self._options) File ~\AppData\Local\Programs\Python\Python310\lib\site-packages\pyspark\context.py:483, in SparkContext.getOrCreate(cls, conf) 481 with SparkContext._lock: 482 if SparkContext._active_spark_context is None: --> 483 SparkContext(conf=conf or SparkConf()) 484 assert SparkContext._active_spark_context is not None 485 return SparkContext._active_spark_context File ~\AppData\Local\Programs\Python\Python310\lib\site-packages\pyspark\context.py:195, in SparkContext.__init__(self, master, appName, sparkHome, pyFiles, environment, batchSize, serializer, conf, gateway, jsc, profiler_cls, udf_profiler_cls) 189 if gateway is not None and gateway.gateway_parameters.auth_token is None: 190 raise ValueError( 191 "You are trying to pass an insecure Py4j gateway to Spark. This" 192 " is not allowed as it is a security risk." 193 ) --> 195 SparkContext._ensure_initialized(self, gateway=gateway, conf=conf) 196 try: 197 self._do_init( 198 master, 199 appName, (...) 208 udf_profiler_cls, 209 ) File ~\AppData\Local\Programs\Python\Python310\lib\site-packages\pyspark\context.py:417, in SparkContext._ensure_initialized(cls, instance, gateway, conf) 415 with SparkContext._lock: 416 if not SparkContext._gateway: --> 417 SparkContext._gateway = gateway or launch_gateway(conf) 418 SparkContext._jvm = SparkContext._gateway.jvm 420 if instance: File ~\AppData\Local\Programs\Python\Python310\lib\site-packages\pyspark\java_gateway.py:106, in launch_gateway(conf, popen_kwargs) 103 time.sleep(0.1) 105 if not os.path.isfile(conn_info_file): --> 106 raise RuntimeError("Java gateway process exited before sending its port number") 108 with open(conn_info_file, "rb") as info: 109 gateway_port = read_int(info) RuntimeError: Java gateway process exited before sending its port number

[@amanahmed6057]

bro don't use this use google collab

[@mrraju9986]

When I was creating pyspark seeion it's through an erro like this java gateway process exited before sending it's port number

[@harendrajadon1980]

hadoop is not installed on my windows machine...what to do

[@chakhil8000]

Much awaited

[@AbhishekTiwari-xw7ux]

AnalysisException: Path does not exist: file:/C:/Users/abhi/test.csv How to solve this issue ....even i keep my file in the same location

[@bryandiaz__]

Hello, I keep getting this error and can't move past it could you kindly suggest please File ~\AppData\Local\Programs\Python\Python311\Lib\site-packages\pyspark\java_gateway.py:106, in launch_gateway(conf, popen_kwargs) 103 time.sleep(0.1) 105 if not os.path.isfile(conn_info_file): --> 106 raise RuntimeError("Java gateway process exited before sending its port number") 108 with open(conn_info_file, "rb") as info: 109 gateway_port = read_int(info) RuntimeError: Java gateway process exited before sending its port number

[@kalpeshghadigaonkar3388]

Waiting for this for so long!

[@payelpanja7125]

will wait for more videos :-)

[@balramthakur9951]

Sir, after installing pyspark and when I am running the code , I am getting an error-" Java gateway process exited before sending its port number". I have set the path of java in environment variable, still getting the same error

[@anandjha6863]

im getting same error ...if you find any solution pl reply

[@balramthakur9951]

@@anandjha6863 not yet

[@avinashkar2260]

@yadav k Hi, Even after installing jdk 8 I am facing same error. Pls suggest

[@avinashkar2260]

solved with a restart after jdk 8 installation. Thanks

[@i_amanrajput]

after installing jdk8, set path where your java installed import os os.environ["JAVA_HOME"] = "C:\Program Files\Java\jdk1.8.0_321" os.environ["PATH"] = os.environ["JAVA_HOME"] + "/bin:" + os.environ["PATH"]

[@optimistic_guy313]

I am having some problems with thinking. Can you share how you tackle thinking and do fast thinking?

[@AlDamara-x8j]

Thanks for this video. For learning purposes on my own computer, do I need to install apache.spark (spark-3.4.1-bin-hadoop3.tgz) to be able to run spark scripts/notebooks, or just pip install pyspark on my python environment?

[@amberkataria9408]

spark session command : spark = SparkSession.builder.appName('Practiceee').getOrCreate() is taking infinite time. Not able to run code further as it kept on running. What is the solution for this?

[@ananyanayak7509]

Hello Sir, I got error as :- "Exception: Java gateway process exited before sending its port number" while executing line number 5. How can I resolve it ?

[@nlokesh1986]

Sir, how are you getting the automatic suggestions in jupyter notebook.. please help me, so that i can do the same with my system. Thanks alot

[@dileepk1740]

Hi Krish, I have created new environment for pyspark !pip install pyspark import pyspark are successful but import pandas as pd give error as: No module named 'pandas' what needs to do ?

[@bhaskararya5901]

my pyspark session is still running for last 2 hours. what to do, i tried other method like update my pip,etc. Did anyone face the same problem? any solution is appreciated.

[@sandeepnelwade]

Hi Krish I got error when creating sparksession, how I connect with you

[@rhevathivijay2913]

Sir Can you please give exercise at each end of your video in future?

[@yadavanubhav005]

Hi Krish, any idea why my code same as yours is not getting executed. I installed jupyter notebook using anaconda. I wish I could have pasted the screenshot here.

[@SynonAnon-vi1ql]

Hi Krish! Great tutorial! Thanks for this! One (probably stupid) question and I'm a novice here. How did you enable the auto-suggest functionality in your jupyter notebook? Mine doesn't work. Could you please help? Thank you!

[@KARANKUMAR-qr9nj]

Great work. You are awesome :)

[@dheerendrasinghbhadauria9798]

I am getting an error " Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext. : java.lang.NoClassDefFoundError: Could not initialize class org.apache.spark.storage.StorageUtils$ "

[@jatinsharma9101]

Hi Krish, RuntimeError: Java gateway process exited before sending its port number I was getting this error in SparkSession.builder.appName('practice').getOrCreate() Please help me

[@amitgupta-ty8xd]

sir please make videos regarding jons in data science for freshers and entry levels which u have started earlier it's a request

[@akashchauhan8436]

How to create a timeseries in pyspark. Say for example I have a column named start_date wit the format (YYYY-MM) for some event, but its not continuous, i.e. I have 2015-01, 2015-04, 2015-07. Then how do I fill the missing dates between them and assign the values to other columns as 0 in pyspark? It was easy in pandas where I could just set this column as index and then resample the dataframe.

[@raghuls9010]

i get spark output like this further unable to read the dataset

[@areebakhtar9841]

Hi I am getting following error while executing spark = SparkSession.builder.appName('learning').getOrCreate() RuntimeError: Java gateway process exited before sending its port number

[@ruthvikrajam.v4303]

pyspark works only with java 8 version and not the latest java software i.e java 17

[@Paragparashar03]

SparkSession.build.appName command is either taking forever with no result or I am getting an error about context already using etc. on the screen. What to do?

[@Paragparashar03]

Got the solution: Tried everything (Various python versions, Java versions, conda installation, manual installation, env variable set up--- finally JAVA 11 worked for me. Follow the link : kzbin.info/www/bejne/iKeXf4J3o5egY8k

[@muhammadsalmanhassan7544]

What we can divide dataset into multiple chunks in pandas and train the model on it is this good practice or bad practice?

[@ankitsaxena565]

Hi Sir,this playlist is enough for learning pyspark