How To Install Spark Pyspark in Windows 11 ,10 Locally

Hi All ,
In this video I have covered step by step instructions for installing Apache Spark in Local System.
I am providing All the required URL and Details for Environment Variables:
#java
www.oracle.com...
#python :
www.python.org...
#spark :
spark.apache.o...
#WinUtils File
github.com/cda...
#vscode :
code.visualstu...
#Environments Variables value for path
%JAVA_HOME%\bin
%SPARK_HOME%\bin
%HADOOP_HOME%\bin
Also I have solved error for WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Please add SPARK_LOCAL_HOSTNAME and localhost to Env Variables:

[@pradeepsudarshan6117]

Thanks! I tried following many different tutorials but this one finally worked.

[@thecloudbox]

Thnx for watching brother and glad my video helped

[@tao3964]

Gracias

[@emiel89]

Unfortunarely I can't use this with jupyter notebook. I wanted to use jupyter notebook with spark without the java error. With this I can use spark in the command line but not in jupyter notebook.

[@thecloudbox]

Hi you can use this in Jupyter notebook for sure you need to install Anaconda and try to open jupyter notebook from there , do let me know if you face any problem

[@Abhishekshetty1230]

Bro I'm having an error at the last part vs code. The code is showing an error please help me to fix it. Thank u

[@thecloudbox]

Sure bro can you send me the error to my telegram I'd axtzitzanup

[@akashpandit5464]

Hi I’m facing error python worker exited unexpectedly (crashed). Please help me

[@thecloudbox]

Hi can you please share more log details, if not can you uninstall your python and reinstall the Python 3.11 or 3.12 version and set path while installing

[@akashpandit5464]

@@thecloudbox I have reinstalled python and installed new version but still facing same issue Rdd =sc.parallelize([1,2,3]) Rdd.first() Error : Exception in task 0.0 in stage 0.0(TID 0/1)] Org.apache.spark.SpaekExecption:Python worker exited unexpectedly (crashed)

[@thecloudbox]

Can you please check with dataframe like you are using RDD also please import pyspark,

[@akashpandit5464]

@@thecloudbox when I use data frame it’s print data frame schema correct but when I execute df.show(),then same python worker crashed error .

[@abc_cba]

try installing python with any version that is a year old in its version. and uninstall the correct version (remove its registry keys as well)

[@pogoclub8495]

bro can you please upload java 11 zip file in google drive and share the link please, I am getting bad gateway error when I try to download. I have already create the oracle account and sign in.

[@thecloudbox]

Bro you can download it from here choose your os in case windows choose windows. www.oracle.com/in/java/technologies/javase/jdk11-archive-downloads.html

[@historyofislam7509]

too difficult

[@laurentiucornateanu620]

u are over the place if everthing is the same in your head have to be more organize ..............ok?

[@GaganTyagi2000]

spark-shell Python was not found; run without arguments to install from the Microsoft Store, or disable this shortcut from Settings > Manage App Execution Aliases. The system cannot find the path specified.

[@saiabhilash2445]

Did you resolve it

[@dineshh_ad]

@@saiabhilash2445 Just remove the %JAVA_HOME%/bin from path variable setting & SPARK_HOME should be c:spark/**your subfolder**

[@mukkalasaicharan8405]

@@saiabhilash2445hi did you resolve it

[@mukkalasaicharan8405]

@GaganTyagi2000 did u resolve it

[@frennardenddy8763]

THANK YOU SO MUCH FOR HAVE MADE THIS VIDEO!!!!!!!!!!!!!!!!! I tried to install it in many ways, but I got "The system cannot find the path specified". But this video gave me the solution that I needed, thank you very much!!! :D:D:D:D

[@thecloudbox]

Thankyou so much for your kind word , I am glad I was able to help you , keep watching keep learning

[@logsofzany]

spark-shell always says path not found. I have specified the variable with the bin path many times. I tried deleting every path and variable old ones and created again but still the same error. Even restarting PC didn't fix. Help me

[@thecloudbox]

I hope you have installed lower version of win-utuils files as compared to spark version and made all paths and variable same as I have mentioned in video

[@sahiltikkal-ln5kr]

when you create environment variable for SPARK_HOME set the path to C:\Spark\spark-3.4.2-bin-hadoop3 or the folder you have extracted spark files. This solved issue for me. Hope it helps.

[@logsofzany]

I really appreciate both of you guys for responding to me. 🫂. I fixed it now. What happened was so silly, my spark, hadoop, python everything and their path, variables were fine. When I checked my Java --version in cmd it was also fine. But, I included \bin in my JAVA_HOME variable and just mentioned %JAVA_HOME% as the path. I casually removed \bin in variable and then mentioned %JAVA_HOME%\bin in the path. My spark-shell worked 🙂🎉😒. Computers are so weird. Thanks again. 🤌🏼

[@abc_cba]

Hi, Anup, can you do tutorials on projects using Spark, Kafka, Flume, Storm? These are not available on KZbin, so yours would be a hit in the future, thanks.

[@thecloudbox]

Hey thanks for your suggestion buddy sure I will do it , All the topics which have mentioned it is great hit

[@cristobalquirozvillanueva6511]

Hi friend, you saved my life! before viewing this tutorial i saw many videos but none of them helped me, your tutorial helped me! thanks a lot!

[@thecloudbox]

Glad to hear that

[@vishnukesavan8116]

Can you do a installation video for mac as well will be really helpful

[@thecloudbox]

I dnt have mac bro really sorry

[@myna100]

Thank you so much! I tried following multiple other tutorials (all failed), but this one worked splendidly. Thank you thank you!

[@thecloudbox]

Hi thankyou for the kind word I just tried to help others I am glad it helped you thanks for watching my video ✌️

[@Delchursing]

I'm running pyspark based code locally! Thank you! I need to learn about high speed data analysis on my old slow laptop😂

[@thecloudbox]

You can use Google Collab or any cloud with Databricks community version

[@saiganesh-zq7qg]

Running the spark application from CMD or using PYCHARM showing error as cannot run program "PYTHON 3" create process error=2, the system cannot find the file specified. do you know how to resolve this?, please respond to this comment if you have an answer, thanks

[@usamabintahir99]

Did not work for me. Getting Py4JavaError while showing dataframe

[@AlexSilva-sp4rw]

If can't load spark-shell in the cmd, take a look in the system variables, if path %SystemRoot%\System32 is present.

[@laurentiucornateanu620]

u are in hurry? have a date or something?

[@thecloudbox]

If you find speed is more you can set your playback speed to 0.75x why are you getting angry 😂

[@joelsarpong1847]

I was able to follow all the steps but when I switched it to Pyspark am not getting what you have. Can you help me with that?

[@gropponedaficulle367]

GREAT!!!

[@vaibhavkiratkar2012]

For me it is showing spark-shell is not recognised as internal or external command

[@thecloudbox]

I was also getting same error in last part I have explained this request you to please watch the complete video you will get the solution

[@monikakorbecka7033]

@@thecloudbox I experience the same problem as @vaibhavkiratkar2012 , and maybe I am blind but I cannot find solution to that in your video (I watched a few times to the end) :) can you help with that?

[@alexTAN11]

very good video, thank you!!!!

[@thecloudbox]

Glad you liked it!

[@yogeshanand316]

I am getting this error in cmd " \Spark\bin\..\conf was unexpected at this time. " please help

[@NA-dg6um]

Iam able to check pyspark and spark-shell command in cmd but when I tried to run code in vs code it is showing error like unable to load native-hadoop library and python was not found. I followed all the steps you mentioned

[@mukkalasaicharan8405]

Hi , same issue here ..we’re you able to resolve it

[@diegofalcon5550]

I need help. When I run spark-shell in terminal, to the end appears this message: scala> 24/04/08 03:37:19 WARN GarbageCollectionMetrics: To enable non-built-in garbage collector(s) List(G1 Concurrent GC), users should configure it(them) to spark.eventLog.gcMetrics.youngGenerationGarbageCollectors or spark.eventLog.gcMetrics.oldGenerationGarbageCollectors

[@thecloudbox]

Can you please confirm your spark version and Java version

[@durishettipraneeth1244]

Thanks❤

[@aashishd2330]

when I run spark-shell I am getting "The system cannot find the path specified" . Please help me in overcoming this.

[@thecloudbox]

Hi probably you are setting up the path correctly go to environment variables again and set the path as per video it should work

[@Delchursing]

Legend!

[@sarahq6497]

TYSM

[@pythonmathui3057]

Thank you sir :)

[@thecloudbox]

Keep watching

[@pythonmathui3057]

@@thecloudbox anyway i tried to create a simple dataframe with spark but when i want to do df.show() it turns out to be error do you know what actually missing?

[@shahsn11]

thank you

[@thecloudbox]

Thankyou

[@somapradhan4572]

Hi, I installed Python, Java, Spark . But when I type python or spark- shell, nothing is coming up

[@somapradhan4572]

Ignore, Restarting helped it. Thanks for explaining steps in detail

[@thecloudbox]

thanks for watching, glad my video helped

[@akashpandit5464]

@somapradhan4572 You are able to execute pyspark queries? If yes then can you please guide me I’m getting python worker crashed error. I have tried so many times but still stuck on same issue .