Install Apache PySpark on Windows PC | Apache Spark Installation Guide

In this lecture, we're going to setup Apache Spark (PySpark) on Windows PC where we have installed JDK, Python, Hadoop and Apache Spark. Please find the below installation links/steps:
PySpark installation steps on MAC: sparkbyexample...
Apache Spark Installation links:
1. Download JDK: www.oracle.com...
2. Download Python: www.python.org...
3. Download Spark: spark.apache.o...
Winutils repo link: github.com/ste...
Environment Variables:
HADOOP_HOME- C:\hadoop
JAVA_HOME- C:\java\jdk
SPARK_HOME- C:\spark\spark-3.3.1-bin-hadoop2
PYTHONPATH- %SPARK_HOME%\python;%SPARK_HOME%\python\lib\py4j-0.10.9-src;%PYTHONPATH%
Required Paths:
%SPARK_HOME%\bin
%HADOOP_HOME%\bin
%JAVA_HOME%\bin
Also check out our full Apache Hadoop course:
• Big Data Hadoop Full C...
----------------------------------------------------------------------------------------------------------------------
Apache Spark Installation links:
1. Download JDK: www.oracle.com...
2. Download Python: www.python.org...
3. Download Spark: spark.apache.o...
-------------------------------------------------------------------------------------------------------------
Also check out similar informative videos in the field of cloud computing:
What is Big Data: • What is Big Data? | Bi...
How Cloud Computing changed the world: • How Cloud Computing ch...
What is Cloud? • What is Cloud Computing?
Top 10 facts about Cloud Computing that will blow your mind! • Top 10 facts about Clo...
Audience
This tutorial has been prepared for professionals/students aspiring to learn deep knowledge of Big Data Analytics using Apache Spark and become a Spark Developer and Data Engineer roles. In addition, it would be useful for Analytics Professionals and ETL developers as well.
Prerequisites
Before proceeding with this full course, it is good to have prior exposure to Python programming, database concepts, and any of the Linux operating system flavors.
-----------------------------------------------------------------------------------------------------------------------
Check out our full course topic wise playlist on some of the most popular technologies:
SQL Full Course Playlist-
• SQL Full Course
PYTHON Full Course Playlist-
• Python Full Course
Data Warehouse Playlist-
• Data Warehouse Full Co...
Unix Shell Scripting Full Course Playlist-
• Unix Shell Scripting F...
-----------------------------------------------------------------------------------------------------------------------Don't forget to like and follow us on our social media accounts:
Facebook-
/ ampcode
Instagram-
/ ampcode_tutorials
Twitter-
/ ampcodetutorial
Tumblr-
ampcode.tumblr.com
-----------------------------------------------------------------------------------------------------------------------
Channel Description-
AmpCode provides you e-learning platform with a mission of making education accessible to every student. AmpCode will provide you tutorials, full courses of some of the best technologies in the world today. By subscribing to this channel, you will never miss out on high quality videos on trending topics in the areas of Big Data & Hadoop, DevOps, Machine Learning, Artificial Intelligence, Angular, Data Science, Apache Spark, Python, Selenium, Tableau, AWS , Digital Marketing and many more.
#pyspark #bigdata #datascience #dataanalytics #datascientist #spark #dataengineering #apachespark

[@eloscarc5782]

Your video helped me understand it better than other videos, now the other videos make sense. This was not as convoluted as I thought.

[@ipheiman3658]

This worked so well for me :-) The pace is great and your explanations are clear. I am so glad i came across this, thanks a million! 😄 I have subscribed to your channel!!

[@ismailcute1584]

Thank you so much for this video. Unfortunately, I couldn't complete this - getting this erros C:\Users\Ismahil>spark-shell 'cmd' is not recognized as an internal or external command, operable program or batch file. please help

[@JesusSevillanoZamarreno-cu5hk]

execute as admin

[@johnpaulmawa4808]

@@JesusSevillanoZamarreno-cu5hk You are the bestest and sweetest in the world

[@frankcastelo9987]

I was having the same issue as you, and it turn to work, simply doing what Jesus said (OMG!): "Run it as admin". Thanks everyone.. Indeed, Jesus saves us!!

[@neeleshgaikwad6387]

Very helpful video. Just by following the steps you mentioned I could run the spark on my windows laptop. Thanks a lot for making this video!!

[@ampcode]

Thank you so much!😊

[@iniyaninba489]

@@ampcode bro I followed every step you said, but in CMD when I gave "spark-shell", it displayed " 'spark-shell' is not recognized as an internal or external command, operable program or batch file." Do you know how to solve this?

[@sssssshreyas]

@@iniyaninba489 add same path in User Variables Path also, just like how u added in System Variables Path

[@arnoldochris5082]

Ok guys this is how to do it, incase you are having problems👇 1.) I used the latest version 3.5.0, (Pre-built for apache hadoop 3.3 or later) - downloaded it. 2.) Extracted the zip file just as done, the first time it gave me a file, not a folder but a .rar file which winrar could not unzip, so I used 7-zip and it finally extracted to a folder that had the bins and all the other files. 3.) In the system variables he forgot to edit the path variables and to add %SPARK_HOME%\bin. 4.) Downloaded winutils.exe for hadoop 3.0.0 form the link provided in the video. 5.) Added it the same way but c>Hadoop>bin>winutils.exe 6.) Then edit the user variables as done then do the same to the path %HADOOP_HOME%\bin Reply for any parts you might have failed to understand🙂

[@MANALROGUI]

What do you mean for the 3rd step ?

[@stay7485]

Thanks

[@ampcode]

Thank you so much 😊

[@sarahq6497]

Hello, I had to use the latest version as well, but I'm not able to make it work, I followed the tutorial exactly :(

[@Sai_naga]

@@sarahq6497 me too... when i am running the spark-shell command from the exact spark location on cd, it works... but when i run it just after opening cmd, it doesn't it gives error like spark-shell is not found

[@susmayonzon9198]

Excellent! Thank you for making this helpful lecture! You relieved my headache, and I did not give up.

[@ampcode]

Thank you so much!

[@moathmtour1798]

hey , which version of hadoop did you install because the 2.7 wasn't available

[@indianintrovert281]

Those who are facing problems like 'spark-shell' is not recognized as an internal or external command On command prompt write 'cd C:\Spark\spark-3.5.1-bin-hadoop3\bin' use your own spark filepath(include bin too) And then write spark-shell or pyspark (It finally worked for me, hope it works for you too) If it worked, like this so that more people benefit from this

[@SharinH]

It worked .. Thank you

[@jagjodhsingh2358]

It worked, thanks :)

[@Manishamkapse]

Thank you 😊 so much it worked

[@Manishamkapse]

Thank you 😊 so much it worked

[@vishaltanwar2238]

why did we get this error?

[@ramnisanthsimhadri3161]

I am not able to find the package type: pre-build for Apache Hadoop 2.7 in the drop-down. FYI - my spark release versions that i can see in the spark releases are 3.4.3 and 3.5.1.

[@prateektripathi3834]

Did Everything as per the video, still getting this error : The system cannot find the path specified. on using spark-shell

[@srishtimadaan03]

On command prompt write 'cd C:\Spark\spark-3.5.1-bin-hadoop3\bin' use your own spark filepath(include bin too) And then write spark-shell or pyspark (It finally worked for me, hope it works for you too)

[@joshizic6917]

how is your spark shell running from your users directory? its not running for me

[@Sai_naga]

did it workfor you now? same issue ffacing here

[@rayudusunkavalli2318]

i did every step you have said, but still spark is not working

[@YohanTharakan]

Hi, I completed the process step by step and everything else is working but when I run 'spark-shell' , it shows - 'spark-shell' is not recognized as an internal or external command, operable program or batch file. Do you know what went wrong?

[@viniciusfigueiredo6740]

I'm having this same problem, the command only works if I run CMD as an administrator. Did you manage to solve it?

[@hulkbaiyo8512]

@@viniciusfigueiredo6740 same as you, run as administrator works

[@shivamsrivastava4337]

@@viniciusfigueiredo6740 same issue is happening with me

[@RohitRajKodimala]

@@viniciusfigueiredo6740same issue for me did u fix it?

[@santaw]

Anyone solved this?

[@antonstsezhkin6578]

Excellent tutorial! I followed along and nothing worked in the end :) StackOverflow told me that "C:Windows\system32" is also required in the PATH variable for spark to work. I added it and spark started working.

[@Manojprapagar]

helped

[@antonstsezhkin6578]

@@Manojprapagar happy to hear it!

[@ampcode]

Thank you so much!

[@conroybless]

This was the game changer, also check the the extracted spark folder isn't in a folder of another folder(3 clicks to see the files). Should just be the spark folder you created and inside that folder another folder with the extracted spark filies.(2 clicks to see the files)

[@amitkumarpatel7762]

I have followed whole instruction but when I am running spark -shell is not recognised

[@JustinLi-y6q]

same here

[@anandbagate2347]

'spark-shell' is not recognized as an internal or external command, operable program or batch file.

[@AnudeepKolluri]

use vscode dev containers to setup spark in minutes without any issues

[@Karansingh-xw2ss]

i'm facing this issue can anyone help me to fix this 'spark-shell' is not recognized as an internal or external command, operable program or batch file'.

[@nikhilupmanyu8804]

Try to add direct path at System Environment. It will fix the issue

[@rakeshkandula2318]

Hi, i followed exact steps (installed spark 3.2.4 as that is the only version available for hadoop 2.7). Spark-shell command is working but pyspark is thrwing errors. if anyone has fix to this please help me. Thanks

[@thedataguyfromB]

Step by step solution kzbin.info/www/bejne/oICcqI19etiDiNEsi=aaITbbN7ggnczQTc

[@nftmobilegameshindi8392]

spark shell not working

[@anthonyuwaifo8605]

I got the below error while running spyder even though i have added the PYTHONPATH. File ~\anaconda\lib\site-packages\spyder_kernels\py3compat.py:356 in compat_exec exec(code, globals, locals) File c:\users\justa\.spyder-py3\temp.py:26 df = spark.createDataFrame(data = data, schema = columns) File ~\anaconda\lib\site-packages\pyspark\sql\session.py:1276 in createDataFrame return self._create_dataframe( File ~\anaconda\lib\site-packages\pyspark\sql\session.py:1318 in _create_dataframe rdd, struct = self._createFromLocal(map(prepare, data), schema) File ~\anaconda\lib\site-packages\pyspark\sql\session.py:962 in _createFromLocal struct = self._inferSchemaFromList(data, names=schema) File ~\anaconda\lib\site-packages\pyspark\sql\session.py:834 in _inferSchemaFromList infer_array_from_first_element = self._jconf.legacyInferArrayTypeFromFirstElement() File ~\anaconda\lib\site-packages\py4j\java_gateway.py:1322 in __call__ return_value = get_return_value( File ~\anaconda\lib\site-packages\pyspark\errors\exceptions\captured.py:169 in deco return f(*a, **kw) File ~\anaconda\lib\site-packages\py4j\protocol.py:330 in get_return_value raise Py4JError( Py4JError: An error occurred while calling o29.legacyInferArrayTypeFromFirstElement. Trace: py4j.Py4JException: Method legacyInferArrayTypeFromFirstElement([]) does not exist at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318) at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326) at py4j.Gateway.invoke(Gateway.java:274) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182) at py4j.ClientServerConnection.run(ClientServerConnection.java:106) at java.base/java.lang.Thread.run(Thread.java:1623)

[@ampcode]

Sorry for late response. Could you please check if you are able to run spark-submit using cmd?

[@ankitgupta5446]

100 % working solution kzbin.info/www/bejne/oICcqI19etiDiNEsi=lzXq4Ts7ywqG-vZg

[@Saravanan_Gunasekaran]

is there any thing wrong with the latest version of the python and spark 3.3.1 ? i am still getting the error

[@AkshayNagendra]

I followed all the steps but I'm getting this error 'spark-shell' is not recognized as an internal or external command, operable program or batch file

[@Karansingh-xw2ss]

Yeah I'm also facing this same issue

[@ankitgupta5446]

100 % working solution kzbin.info/www/bejne/oICcqI19etiDiNEsi=lzXq4Ts7ywqG-vZg

[@riptideking]

'pyspark' is not recognized as an internal or external command, operable program or batch file. getting this error and tried it for whole day and same issue.

[@srishtimadaan03]

On command prompt write 'cd C:\Spark\spark-3.5.1-bin-hadoop3\bin' use your own spark filepath(include bin too) And then write spark-shell or pyspark (It finally worked for me, hope it works for you too)

[@Sai_naga]

@@srishtimadaan03 hello....but we added spark home in environment variables, what is the point of running it from the exact location? Environment variables should help system to find the command.

[@ChiragJiteshRana]

To the peeps getting 'cmd' cannot recognize the command, run the command prompt as administrator and try it again.

[@ampcode]

Thank you so much for your inputs!

[@solitudechronikles3521]

Thanks for that

[@ampcode]

@@solitudechronikles3521 thank you!

[@saurabh_bhagat_]

Thank you bro, thanks a lot

[@RohitRajKodimala]

Thanks , but how can we fix it ??

[@shahrahul5872]

on apache spark's installation page, under choose a package type, the 2.7 version seem to not be any option anymore as on 04/28/2023. What to do?

[@shahrahul5872]

I was able to get around this by copying manually the URL of the site you were opened up to after selecting the 2.7th version from the dropdown. Seems like they have archived it.

[@ampcode]

Sorry for late reply. I hope your issue is resolved. If not we can discuss further on it!

[@nagalakshmip8725]

I'm getting spark- shell is not recognised as an internal or external command, operable program or batch file

[@harshithareddy5087]

I don't have the option for Hadoop 2.7 what to choose now???

[@LLM_np]

did you get any solution? please let me know

[@geetalimatta2836]

@@LLM_np NO

[@meditationmellowmelodies7901]

I followed all the setps but getting error 'spark-shell' is not recognized as an internal or external command, operable program or batch file.

[@Mralbersan]

the same happens to me

[@indianintrovert281]

Facing same error, Did you find any solution for it?

[@badnaambalak364]

I followed the steps & Installed JDK 17, spark 3.5 and python 3.12 when I am trying to use map function I am getting an Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe Error please someone help me

[@rajkumarsahoo6937]

same problem 😢

[@prajakta-dh7fc]

'spark' is not recognized as an internal or external command, operable program or batch file. its not working for me i have follow all the steps but its still not working waiting for solution

[@nagarajgotur]

spark-shell is working for me, pyspark is not working from home directory, getting error 'C:\Users\Sana>pyspark '#' is not recognized as an internal or external command, operable program or batch file.' But when I go to python path and run the cmd pyspark is working. I have setup the SPARK_HOME and PYSPARK_HOME environment variables. Could you please help me. Thanks

[@ampcode]

Sorry for late response. Could you please also set PYSPARK_HOME as well to your python.exe path. I hope this will solve the issue😅👍

[@bintujose1981]

@@ampcode nope. Same error

[@coclegend715]

everything is working fine until i run "pyspark" in my command prompt which shoes an error "ERROR: The process with PID 38016 (child process of PID 30404) could not be terminated. Reason: There is no running instance of the task. ERROR: The process with PID 30404 (child process of PID 7412) could not be terminated. Reason: There is no running instance of the task."

[@basanthaider3238]

me too have find a solution?

[@muhammadazam8422]

Good day, I face this issue after configuration. scala> 23/07/25 12:55:00 WARN ProcfsMetricsGetter: Exception when trying to compute pagesize, as a result reporting of ProcessTree metrics is stopped 23/07/25 12:55:00 WARN GarbageCollectionMetrics: To enable non-built-in garbage collector(s) List(G1 Concurrent GC), users should configure it(them) to spark.eventLog.gcMetrics.youngGenerationGarbageCollectors or spark.eventLog.gcMetrics.oldGenerationGarbageCollectors

[@ankitgupta5446]

100 % working solution kzbin.info/www/bejne/oICcqI19etiDiNEsi=lzXq4Ts7ywqG-vZg

[@Jerriehomie]

Getthing this error: WARN ProcfsMetricsGetter: Exception when trying to compute pagesize, as a result reporting of ProcessTree metrics is stopped. People have mentioned to use python fodler path which I have as you have mentioned but still.

[@bukunmiadebanjo9684]

I found a fix for this. Change your python path to that of anaconda(within the environment variable section of this video) and use your anaconda command prompt instead. No errors will pop up again.

[@ampcode]

Sorry for late response. Could you please let me know if you are still facing this issue and also confirm if you’re able to open spark-shell?

[@shivalipurwar7205]

@@bukunmiadebanjo9684 Hi Adebanjo, my error got resolved with you solution. Thanks for your help!

[@chinmaymishra6381]

winutil file is not downloading from that github link

[@sriram_L]

Yes brother.Did u get it now from anywhere?

[@sibrajbanerjee6297]

I am getting a message of 'spark-version' is not recognized as an internal or external command, operable program or batch file. This is after setting up the path in environment variables for PYSPARK_HOME.

[@Sai_naga]

try running as administrator.

[@syamprasad8295]

while selecting a package type for spark, Hadoop 2.7 is not available now. Only Hadoop 3.3 and later is available. And winutils 3.3 is not available at the link provided at the git. What to do now? can I download Hadoop 3.3 version and can proceed with winutils2.7 ? Pls help.. Thanks In Advacnce

[@shivamrai162]

I got same issue

[@ankitgupta5446]

100 % working solution kzbin.info/www/bejne/oICcqI19etiDiNEsi=lzXq4Ts7ywqG-vZg

[@edu_tech7594]

my Apache hadoop which i downloaded previously is version 3.3.4 eventhough i should choose pre-built for Apache Hadoop 2.7?

[@sriram_L]

Same doubt bro. Did u install now

[@laxman0457]

i have followed all your steps,still i'm facing an issue. 'spark2-shell' is not recognized as an internal or external command

[@nayanagrawal9878]

Do everything that he said but not in User Variables but in System variables. I was facing the same problem but then I did the same in system variables and my spark started running.

[@thedataguyfromB]

Step by step spark + PySpark in pycharm solution video kzbin.info/www/bejne/oICcqI19etiDiNEsi=aaITbbN7ggnczQTc

[@user-zk4hm2cy8l]

If you tried all the steps mentioned above and it still does not work, try to add "C:\Windows\System32" to system variable "path". It fixed the error after 2 days of struggling

[@viniciusfigueiredo6740]

I followed the step by step and when I search for spark-shel at the command prompt I come across the message :( 'spark-shell' is not recognized as a built-in command or external, an operable program or a batch file). I installed windows on another HD and did everything right, there are more people with this problem, can you help us? I'm since January trying to use pyspark on windows

[@letsexplorewithzak3614]

Need to edit bottom "add this to env var path" path >> C:\Spark\spark-3.3.1-bin-hadoop2\bin\

[@kiranmore29]

@@letsexplorewithzak3614 Thanks worked for me

[@nayanagrawal9878]

Do everything that he said but not in User Variables but in System variables. I was facing the same problem but then I did the same in system variables and my spark started running.

[@jayakrishnayashwanth7358]

Even I'm facing the same issue ,can you tell in more detail like what to do add in system variables??As we already added Java , Hadoop, Spark and Pyspark_Home in the user varaibles as said in the video.@@nayanagrawal9878

[@penninahgathu7956]

@@nayanagrawal9878 thank you!!! I did this and it solved my problem

[@saikrishnareddy3474]

I’m little confused on how to setup the PYTHONHOME environment variable

[@thedataguyfromB]

Step by step kzbin.info/www/bejne/oICcqI19etiDiNEsi=aaITbbN7ggnczQTc

[@nihalisahu3857]

in CMD while running spark-shell getting error like ERROR SparkContext: Error initializing SparkContext.

[@ashwinnair2325]

thanks a lot pyspark is opening but when executing df.show() command on a dataframe i get below error Cannot run program "python3": CreateProcess error=2, The system cannot find the file specified is there any way to rectify it

[@venkataramakrishnamagapu7645]

Did you get the solution.. i am also facing the same issue.

[@Sarvesh_Ganesan]

PYTHONPATH- %SPARK_HOME%\python;%SPARK_HOME%\python\lib\py4j-0.10.9-src;%PYTHONPATH% Any idea about this text?

[@ampcode]

Have you added this in the env variables and is Spark running fine? Please let me know

[@anastariq1310]

After entering pyspark in cmd it shows "The system cannot find the path specified. Files\Python310\python.exe was unexpected at this time" please help me resolve it

[@mahamudullah_yt]

i face the same problem. is there any solution

[@BOSS-AI-20]

In cmd the comand spark-shell is running only under C:\Spark\spark-3.5.0-bin-hadoop3\bin directory not globally same for pyspark

[@s_a_i5809]

yeah man , same for me.. did you found any fixes... if, let me know :)

[@BOSS-AI-20]

@@s_a_i5809 add your Environment variables under system variables not user variables.

[@ankitgupta5446]

100 % working solution kzbin.info/www/bejne/oICcqI19etiDiNEsi=lzXq4Ts7ywqG-vZg

[@lucaswolff5504]

I added C:\Program Files\spark\spark-3.5.1-bin-hadoop3\bin to the system variables and it worked

[@BOSS-AI-20]

@@lucaswolff5504 yes

[@satishboddula4942]

I have done exactly you shown in tutorial but when I am running the spark-shell command in cmd getting "spark-shell The system cannot find the path specified."

[@ganeshkalaivani6250]

yes same error..did you find out the colustion

[@satishboddula4942]

@@ganeshkalaivani6250 yes the spark don't support with latest java and python version try with java 1.8 and python 3.7 and spark 2.7

[@ganeshkalaivani6250]

@@satishboddula4942 can you please share the java 1.8 download link jdk showing only 18,19 and 20 version

[@ganeshkalaivani6250]

@@satishboddula4942 still system path cannot find out error

[@shashankkkk]

C:\Apache Spark\spark-3.3.1-bin-hadoop2\bin\ add this to env var path

[@kchavan67]

Hi, following all the steps given in video, I am still getting error as "cannot recognize spark-shell as internal or external command" @Ampcode

[@psychoticgoldphish5797]

I was having this issue as well, when I added the %SPARK_HOME%\bin, %HADOOP_HOME%\bin and %JAVA_HOME%\bin to the User variables (top box, in the video he shows doing system, bottom box) it worked. Good luck.

[@thedataguyfromB]

Step by step spark + PySpark in pycharm solution video kzbin.info/www/bejne/oICcqI19etiDiNEsi=aaITbbN7ggnczQTc

[@omthorve7928]

(base) PS C:\windows\system32> cd "C:\\spark\\bin" (base) PS C:\spark\bin> pyspark 'cmd' is not recognized as an internal or external command, operable program or batch file. (base) PS C:\spark\bin> spark-shell 'cmd' is not recognized as an internal or external command, operable program or batch file. (base) PS C:\spark\bin> I did all recommended

[@omthorve7928]

Yet I am stuck pls help me

[@sanchitabhattacharya353]

while launching the spark-shell getting the following error, any idea?? WARN jline: Failed to load history java.nio.file.AccessDeniedException: C:\Users\sanch\.scala_history_jline3

[@iAmDecemberBorn]

resolved hua ?

[@RajalSolanki-r8z]

Use JDK 8 or 11 you wont see any issues with shell command

[@AnuragPatel-y9j]

ERROR Shell: Failed to locate the winutils binary in the hadoop binary path I am getting above error while running spark or pyspark session. I have ensured that winutils file is present in C:\hadoop\bin

[@ampcode]

Could you please let me know if your all the env variables are set properly?

[@ganeshkalaivani6250]

FileNotFoundError: [WinError 2] The system cannot find the file specified getting this error even i have installed all required intalliation

[@ampcode]

Sorry for late reply. I hope your issue is resolved. If not we can have a connect and discuss further on it!

[@karthikeyinikarthikeyini380]

hadoop 2.7 tar file is not available in the link

[@ankitgupta5446]

100 % working solution kzbin.info/www/bejne/oICcqI19etiDiNEsi=lzXq4Ts7ywqG-vZg

[@Bujdil-y8z]

not working for me i set up everything except hadoop version came with 3.0

[@sriramsivaraman4100]

Hello when I try to run the command spark_shell as a local user its not working (not recognized as an internal or external command) and it only works if I use it as an administratror. Can you please help me solve this? Thanks.

[@ampcode]

Sorry for late response. Could you please try once running the same command from the spark/bin directory and let me know. I guess there might be some issues with your environment vatiables🤔

[@dishantgupta1489]

@@ampcode followed each and every step of video still getting not recognised as an internal or external command error

[@ayonbanerjee1969]

​@@dishantgupta1489 open fresh cmd prompt window and try after you save the environment variables

[@obulureddy7519]

In Environment Variables you give the paths in Users variable Admin. NOT IN System variables

[@touhidalam4825]

Im getting bad constant pool index error. Please help

[@ganeshkalaivani6250]

can any one please help...last two days tried to install spark and give correct variable path but still getting system path not speicifed

[@ampcode]

Sorry for late reply. Could you please check if your spark-shell is running properly from the bin folder. If yes I guess there are some issues with your env variables only. Please let me know.

[@SHIVAMSHARMA-pb2lt]

bhai jo comments pe logo ne query likha he unpe bhi REPLY kar do, sirf thank you wale pe reply nai kiya jata he. Followed each and every step as mentioned, but once I quit scala shell and type pyspark, it is giving error - file not found. Can anyone help here?? Kyuki admin to sirf thank you wale pe reply karta he, usse ek bar ho gaya, baki usko nai pata kuch

[@khushboojain3883]

Hi, I have installed Hadoop 3.3 (the lastest one) as 2.7 was not available. But while downloading winutils, we don't have for Hadoop 3.3 in repository. Where do i get it from?

[@sriram_L]

Same here.Did u get it now?

[@khushboojain3883]

@@sriram_L yes, u can directly get it from google by simply mention the Hadoop version for which u want winutils. I hope this helps.

[@hritwikbhaumik5622]

@@sriram_L it still not working for me though

[@Nathisri]

I have some issues in launching python & pyspark. I need some help. Can you pls help me?

[@JesusSevillanoZamarreno-cu5hk]

same, did you fix it? it worked for scala for me but not spark

[@DEMON-jg3zl]

Those who are facing problems like "'spark-shell' is not recognized as an internal or external command" even after copying everything from video or discription Do This: - Go to environment variables. - your SPARK_HOME path will be "C:\spark\spark-3.5.2-bin-hadoop3" - now copy and paste the following path to change it to "C:\spark\spark-3.5.2-bin-hadoop3\spark-3.5.2-bin-hadoop3" - or if you have other path than this then sorry for you and if you copy the video then your path must be the one I mentioned. - Lastly now restart cmd and run the command spark-shell. NOTE: in "C:\spark\spark-3.5.2-bin-hadoop3\spark-3.5.2-bin-hadoop3" 3.5.2 is my version, yours could be different so you better chek it so.

[@MScProject-u9n]

i have faced an issue :'spark-shell' is not recognized as an internal or external command, operable program or batch file." and i use the same path mentioned above

[@DEMON-jg3zl]

@@MScProject-u9n try chatgpt in my case it solved my problem just tell everything to gpt what are you trying to do and copy paste every code and then I'm 100%sure that it will help

[@pratikshyapriyadarshini4677]

Very Helpful.. Thankyou

[@itsshehri]

hey pyspark isnt working at my pc. I did everything how you asked. Can you help please

[@ampcode]

Sorry for late response. Could you please also set PYSPARK_HOME env variable to the python.exe path. I guess this’ll do the trick😅👍

[@bramhanaskari3152]

you haven't give solution for that warn procfsMetricsGetter exception is there any solution for that ?

[@ampcode]

Sorry for late response. This could happen in windows only and can be safely ignored. Could you please confirm if you’re able to kick off spark-shell and pyspark?

[@Mralbersan]

I can't see Pre-Built for Apache Hadoop 2.7 on the spark website

[@meriemmouzai2147]

same problem for me! I tried the "3.3 and later" version with the "winutils/hadoop-3.0.0/bin", but it didn't work

[@alpha_ak-p3h]

not getting the ui says: docker refused to connect

[@Analystmate]

C:\Users\lavdeepk>spark-shell 'spark-shell' is not recognized as an internal or external command, operable program or batch file. Not working

[@syamprasad8295]

which winutil file did u download? Its Hadoop 2.7 or later version?

[@AmreenKhan-dd3lf]

Apache 2.7 option not available during spark download. Can we choose Apache Hadoop 3.3 and later ( scala2.13) as package type during download

[@ganeshkalaivani6250]

can any one help - spark-shell is running fine. but pyspark "The system cannot find the path specified."

[@shashankkkk]

C:\Apache Spark\spark-3.3.1-bin-hadoop2\bin\ add this to env var path

[@ampcode]

Sorry for late reply. Have you also set PYSPARK_HOME, PYTHON_HOME env variables. That'll do the trick :)

[@richardalphonse2680]

Bro while executing spark-shell getting an error ReplGlobal.abort: bad constant pool index: 0 at pos: 49180 [init] error: bad constant pool index: 0 at pos: 49180 while compiling: during phase: globalPhase=, enteringPhase= library version: version 2.12.17 compiler version: version 2.12.17 reconstructed args: -classpath -Yrepl-class-based -Yrepl-outdir C:\Users\HP\AppData\Local\Temp\spark-f4a4c1ed-e79a-4179-9492-a41e66431c1b epl-3fc51940-943d-416d-ab37-074575e4ad8d last tree to typer: EmptyTree tree position: tree tpe: symbol: null call site: in == Source file context for tree position == Exception in thread "main" scala.reflect.internal.FatalError: bad constant pool index: 0 at pos: 49180 while compiling: during phase: globalPhase=, enteringPhase= library version: version 2.12.17 compiler version: version 2.12.17 reconstructed args: -classpath -Yrepl-class-based -Yrepl-outdir C:\Users\HP\AppData\Local\Temp\spark-f4a4c1ed-e79a-4179-9492-a41e66431c1b epl-3fc51940-943d-416d-ab37-074575e4ad8d

[@nilobrayanvallejopablo564]

ejecuten cmd como administrador en el último paso

[@AnuragPatel-y9j]

PYTHONPATH- %SPARK_HOME%\python;%SPARK_HOME%\python\lib\py4j-0.10.9-src;%PYTHONPATH% can you explain the above line which you have mentioned under environment variables ?

[@ampcode]

This is required for the windows installation in which you have multiple values present for python path variable seperated by semi colon

[@shankarikarunamoorthy4391]

sir, spark version is available with Hadoop 3.0 only. Spark-shell not recognized as internal or external command. Please do help.

[@Manoj-ed3lj]

installed successfully but when i am checking hadoop version, i am getting an like hadoop is not recognized as internal or external command

[@ganeshbansode9380]

if not working install spark-3.5.0-bin-hadoop3 and go to folder and then use command spark-shell

[@AgustinArcos-fm4nf]

anytime you're confused by some weird computer program installation bullshit, the youtube desi tech gurus got your back

[@alireza2295]

Great. I followed the instructions and successfully installed spark. Thank you!

[@chinmayapallai8452]

I have followed same thing what ever u have done while u have explained, I have observed and same thing I did but both spark and pyspark is not working,Can you please help me how to resolve the issue as after giving cmd then typing spark-shell it's showing spark- shell is not recognised as internal or external command same thing for spark also . Please help me how to overcome from this 🙏🙏🙏🙏🙏🙏🙏

[@nayanagrawal9878]

Do everything that he said but not in User Variables but in System variables. I was facing the same problem but then I did the same in system variables and my spark started running.

[@moathmtour1798]

hello, which Hadoop Version should i install since the 2.7 is not available anymore ? thanks in advance

[@ampcode]

You can go ahead and install the latest one as well. no issues!

[@venkatramnagarajan2302]

@@ampcode Will the utils file still be 2.7 version ?

[@infamousprince88]

I'm still unable to get this to work. I've been trying to solve this problem for nearly 2 weeks

[@nuzairmohamed5345]

I get a noModuleError saying pyspark does not contain numpy module. I followed all the steps. Can you please help??

[@ampcode]

Hello, Are you trying to use numpy in your code. If so, have you installed pandas package? Please let me know so we can solve this issue😃

[@nuzairmohamed5345]

​@@ampcode how to install pandas in pyspark

[@ampcode]

@@nuzairmohamed5345 you can run command as below: pip install pandas Please let me know if any issues.

[@abhinavtiwari6186]

where is that git repository link? Its not there in the description box below

[@ampcode]

Extremely sorry for that. I have added it in the description as well as pasting it here. GitHUB: github.com/steveloughran/winutils Hope this is helpful! :)

[@shivamsrivastava4337]

i am still facing issues.. all the variables have been set by me but still facing issues. one more strange thing is happening i can run pyspark as administrator but not as normal user though i have followed steps in normal user as well

[@RohitRajKodimala]

Same issue for me, did u fine the fix??

[@nayanagrawal9878]

Do everything that he said but not in User Variables but in System variables. I was facing the same problem but then I did the same in system variables and my spark started running.

[@thedataguyfromB]

Step by step spark + PySpark in pycharm solution video kzbin.info/www/bejne/oICcqI19etiDiNEsi=aaITbbN7ggnczQTc

[@thedataguyfromB]

​@@RohitRajKodimala Step by step spark + PySpark in pycharm solution video kzbin.info/www/bejne/oICcqI19etiDiNEsi=aaITbbN7ggnczQTc

[@ashwinkumar5223]

Gettin as spark shell is not recognized as internal or external commnad

[@shashankkkk]

C:\Apache Spark\spark-3.3.1-bin-hadoop2\bin\ add this to env var path

[@ampcode]

Sorry for late reply. I hope your issue is resolved. If not we can have a connect and discuss further on it!

[@ed_oliveira]

Thank you! 👍

[@ampcode]

Thank you so much! Subscribe for more content 😊

[@sisterkeys]

What I was doing in 2 days, you narrowed to 30 mins!! Thank you!!

[@ampcode]

Thank you so much! Subscribe for more content 😊

[@pulkitdikshit9474]

hi i installed but when I restarted my pc it is no longer running from cmd? what might be the issue?

[@ankushv2642]

Did not work for me. At last when I typed the pyspark in command prompt, it did not worked.

[@shivangveeravlogs7164]

Your content is very helpful, but also can you make in Hindi?

[@basanthaider3238]

I have an issue with the pyspark it's not working and it's related to java class I can't realy understant what is wrong ???

[@theefullstackdev]

i have fallowed all these steps and installed those 3 and created paths too, but when i go to check in the command prompt... its not working.. error came... can anyone help me please to correct this

[@nikhilupmanyu8804]

Hi, Thanks for the steps. I am unable to see Web UI after installing pyspark. It gives This URL can't be reached. Kindly help

[@shiviagrawal8302]

PySparkRuntimeError( pyspark.errors.exceptions.base.PySparkRuntimeError: [JAVA_GATEWAY_EXITED] Java gateway process exited before sending its port number. pls help me what to do

[@comedyspikes9800]

Heyy where u able to solve the issue and if yes what did u do?

[@manikantaperumalla2197]

java,python and spark should be in same directory?

[@arpitakar3384]

bro Apache spark runs for on jdk8 and jdk-11 version misinformation sensed to me i think

[@ampcode]

Hello there! Sorry for any inconvenience. Are you facing any issues with the installation?

[@arpitakar3384]

@@ampcode yes, at udemy teacher tolds me that apache spark supports only jdk-8 and jdk11 only

[@arpitakar3384]

@@ampcode please help , 'spark-shell' is not recognized as an internal or external command 'pyspark' is not recognized as an internal or external command i checked enviroment variable 100 times vro

[@tal3003]

Can you please help me to download it via zoom?

[@ampcode]

Hello there...Could you please et me know what exactly is your issue. If you can provide an error, this may be helpful.:)

[@ArtificialIntelligenceColombia]

WHAT A PROCESS!! It worked for me just by run spark-shell in cmd as ADMIN. thank you for the video!

[@syafiq3420]

how did you download the apache spark in zipped file? mine was downloaded as tgz file

[@ampcode]

Sorry for late response. You’ll get both options on their official website. Could you please check if you are using the right link?

[@georgematies2521]

@@ampcode There is no way now to download the zip file, only tgz.