Install and setup/configure Apache PySpark 3.5.1 on Windows 11 |Install PySpark on Windows 11 | 2024

[@prabhatsrivastava8846]

You are awesome. I was trying to install Pyspark and run locally since yesterday but could not get any solution. Your video has helped me. Thanks a lot.

[@r2schools]

Glad to hear it helped! But you have not subscribed

[@eltonsouza5849]

Thank you so much my friend

[@r2schools]

Thank you too

[@bhumivedant7004]

Thank you so much i tried around 5 times after which successfully downloaded using your tutorial

[@r2schools]

Glad to hear that

[@bnrinbox]

Any suggestion, though followed as explained in this video, not showing spark context web UI.

[@crnkrgz]

Just thank you for sharing, it's been a great help for me.

[@r2schools]

Glad it helped!

[@ranjanguru007]

Thanks. Worked perfectly..

[@r2schools]

Glad it helped

[@ashish2099]

Thank you so much. It worked perfectly

[@r2schools]

You're welcome!

[@LuchoPlayer]

Thank you very much, i had an error installing it with other tutorials and with yours those errors were solved. Appreciate it :)

[@r2schools]

Glad to hear that

[@shrigandhahiremath1342]

Thank you very much sir, it helped me lot in installing this pyspark very easily!!!!

[@r2schools]

Glad it helped

[@mungamurisitaramnikhil7355]

I'm getting this error "The system cannot find the path specified." Please let me know what to do

[@ucr2256]

i have the same problem,did you solve it?

[@rosemain123]

I have same but resolve it, just declare java paths in system variable not in user one

[@isaackodera9441]

You're a life saver

[@r2schools]

Glad it was helpful

[@phantanbinh1232]

Hi, could I do the same your instruction on mac to set up pyspark?

[@prime_soldier27]

Hi, Thanks for the great video. I have followed all the steps and pysaprk is running on my windows system. However, When I am writing data to sv or any file format for that matter I am getting error. Following is the code from pyspark.sql.types import IntegerType from pyspark.sql import SparkSession spark = SparkSession.builder.appName("Python PySpark Example").getOrCreate() my_list = [1, 2, 3] df = spark.createDataFrame(my_list, IntegerType()) df.show() df.write.csv("mypath").mode("overwrite") and below is the error I get when writing to disk. line 9Py4JJavaError: An error occurred while calling o60.csv. Any solution?

[@saimmujahid2827]

Thanks a lot man

[@r2schools]

Happy to help

[@infamousprince88]

this is not working for me. I could really use some help as it has been weeks of me attempting to set up PySpark

[@yvyas9646]

Same bro in my cmd line ~ it’s say pyspark is not found as external or internal cmd even i set all the path in env variables

[@venkataramakrishnamagapu7645]

Did you find solution? I am facing the same issue.

[@infamousprince88]

@@venkataramakrishnamagapu7645 I have not. If anything, I’ve created another issue trying to configure my computer settings unfortunately

[@sudhv]

@@venkataramakrishnamagapu7645 can you please provide the solution?

[@tukemo5788]

Have you found a solution? 😢

[@akashgond5262]

I m able to complete the steps as mentioned, and in my cmd also it is showing as spark version 3.5.1 but how to run yhe code and everything, since i tired using jupyter notebook it is throwing error as df.Show() not found

[@juan.giraldo07]

have the same problem, can you resolve it?

[@akashgond5262]

@@juan.giraldo07 nope no luck with it. You can try the community version of databricks

[@venkataramakrishnamagapu7645]

Getting this error when trying to execute.. can you help??? java.io.IOException: Cannot run program "python3": CreateProcess error=2, The system cannot find the file specified

[@iAmDecemberBorn]

is this resolved ?

[@venkataramakrishnamagapu7645]

@@iAmDecemberBorn yes.. in my case spark installation is fine.. Issue was due to existing java and python installation software I cleared all variable related to python and java and uninstalled pyrhon and java Then installed new versions.. it worked for me

[@vanamalakoppala5947]

Thank you for sharing this, very helpful . Really appreciate it !

[@r2schools]

Glad it was helpful!

[@Nature_3002]

'spark-shell' is not recognized as an internal or external command, operable program or batch file.

[@chatterboxvlogs3333]

C:\Users emal>python --version Python 3.12.2 C:\Users emal>spark-shell 'spark-shell' is not recognized as an internal or external command, operable program or batch file. executed all as instructed probably the reason is selection of spark version and winutils .. guide me😒😒😒😰🥺

[@arebkhan6350]

I'm getting the same error let me know how to fix this

[@mugunthanc8660]

@@arebkhan6350 I got the same error. Resolved it by ensuring the JAVE_HOME variable doesn't contain the '\bin'. E.g JAVA_HOME = C:\java\jdk-10.0.1

[@fa7234]

vscode can't recognize it. how to solve that

[@Ben-pp2wz]

🥰🥰🥰🥰🥰🥰

[@yashgugaliya9276]

Sir I did everything that you mentioned in the video still when I am trying to write a code in pyspark it is returning an error.

[@r2schools]

Check one more time.

[@yashgugaliya9276]

@@r2schools I tried thrice I am trying to run basic readstream , writestream still getting the error.

[@anaghaabhinav8424]

if you get ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)/ 1] org.apache.spark.SparkException: Python worker exited unexpectedly (crashed) use python -->3.11.8 version

[@maxajames]

Thanks, this helped!

[@Tamilselvan-vy4vq]

Works Perfect if u guys face the issue - The system cannot find the path specified please check the ENV variables Path and version for me its ( used java version 23 it not worked so I after installed 11 ) now I changed the JAVA_HOME path from jdk-23 to jdk-11 and restarted

[@DARK-fs3rz]

java 21 works fine right ? because even after specifying all env paths , I am getting path not specified (I have java 23)