Installation of Apache Spark on Windows 11 (in 5 minutes)

Prerequisites of Apache Spark Installation:
1. Download Java
2. Download Python
3. Download Apache Spark
4. Download Hadoop Winutils and hadoop.dll
5. Set Environmental variables
6. Test PySpark and Spark Shell
Important Links :
github.com/kon...
spark.apache.o...
www.oracle.com...
www.python.org...

[@dharmatejareddy3651]

Missing Python executable 'python3', defaulting to 'C:\Users\punya\AppData\Local\Programs\Python\Python312\Scripts\..' for SPARK_HOME environment variable. Please install Python or specify the correct Python executable in PYSPARK_DRIVER_PYTHON or PYSPARK_PYTHON environment variable to detect SPARK_HOME safely. The system cannot find the path specified. Facing the above Error

[@devlearning01]

i was able to setup spark-shell. But when i m doing spark-shell --master yarn it shows Exception in thread "main" org.apache.spark.SparkException: When running with master 'yarn' either HADOOP_CONF_DIR or YARN_CONF_DIR must be set in the environment.

[@ammadaqeel3776]

Thanks alot bro. All explained very well.

[@Sk19277]

THANK YOU... VERY WELL EXPLAINED

[@m.azharali8192]

Is this a stable version? I got erroe Py4jjavaerror : python worker failed to connect back...

[@UnboxingBigData]

yes its stable version

[@technicalkhurmai399]

same error

[@Jose2mil]

Mucho ojo si alguna de las rutas delas variables del sistema que intervienen contiene algún espacio, Spark no lo soporta. Os aviso porque a mi me ha consumido 1h hasta encontrar el error.

[@DevTiwari-ub6xl]

Why am I facing "The system cannot find the path specified" error?

[@UnboxingBigData]

Check environmental variable

[@yudisdwi1050]

are u have done this issue ? facing same issue can you tell me how to solve this problem

[@SaddamHussain-qt4cm]

Thanks Man---This is Accurate and Most Straight Installation---You forgot to set Java Home --do we need to set it too

[@UnboxingBigData]

yes, you can set JAVA_HOME. You can set in environmental variables

[@baseerx]

I have followed all the steps correctly but still it is showing "The filename, directory name, or volume label syntax is incorrect." in terminal.. Paths are perfectly and correctly set.

[@UnboxingBigData]

Share screenshot

[@yugendharsai]

PS C:\WINDOWS\system32> cd C:\spark\spark-3.5.0-bin-hadoop3\bin PS C:\spark\spark-3.5.0-bin-hadoop3\bin> spark-shell 'cmd' is not recognized as an internal or external command, operable program or batch file.

[@yanz01-hx1zp]

The system cannot find the path specified. what?

[@UnboxingBigData]

Check environmental variable

[@aayushsolanki3719]

Check your JAVA_HOME variable

[@yudisdwi1050]

are u have done this issue ? facing same issue can you tell me how to solve this problem

[@manojthalari7075]

spark-shell : The term 'spark-shell' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again. At line:1 char:1 + spark-shell + ~~~~~~~~~~~ + CategoryInfo : ObjectNotFound: (spark-shell:String) [], CommandNotFoundException + FullyQualifiedErrorId : CommandNotFoundException

[@UnboxingBigData]

Check environmental variable of spark

[@abc_cba]

Thanks for this video, did you skip the "hadoop" installation part? my system says : "HADOOP_HOME" and "hadoop.home.dir" are unset as well as "Did not find winutils.exe: java.io.FileNotFoundException: java.io.FileNotFoundException" i followed your steps as you asked. but yeah, the spark shell worked irrespective of these warnings

[@UnboxingBigData]

you can rename WINUTILS environment variable to HADOOP_HOME.

[@romank7944]

@@UnboxingBigData But you also have HADOOP_HOME in your variables, %HADOOP_HOME% I saw in on video !? why

[@TanLuke2003]

hi after above actions, i have an error, what should i do scala> 24/06/07 11:17:11 WARN GarbageCollectionMetrics: To enable non-built-in garbage collector(s) List(G1 Concurrent GC), users should configure it(them) to spark.eventLog.gcMetrics.youngGenerationGarbageCollectors or spark.eventLog.gcMetrics.oldGenerationGarbageCollectors

[@UnboxingBigData]

Share screenshot on mail

[@khushalraut1451]

@@UnboxingBigData is this resolved i also need help in this

[@naveenkumar9797]

I am still getting some errors ( ERROR SparkContext: Error initializing SparkContext.). can you share your mail id

[@pritnadoda4298]

*spark-shell gives(pretty long error but the message is >>>)* ERROR Main: Failed to initialize Spark session. java.lang.UnsupportedOperationException: getSubject is supported only if a security manager is allowed

[@muanomakhokha2929]

I am going through the same error. Did you find a solution?

[@pritnadoda4298]

@muanomakhokha2929 i got it from chatgpt... I don't exactly remember what was it.

[@pritnadoda4298]

@muanomakhokha2929 i got it from chatgpt... I don't exactly remember what was it.

[@juliancaste3914]

also the same error

[@muanomakhokha2929]

@pritnadoda4298 I changed my java version from 23 to 21 and it worked. Java 23 has a new implementation that causes this error