Spark Executor Memory Calculation | Number of Executors | Executor Cores | Spark Interview Q&A

#sparkmemoryconfig #executormemory #drivermemory #SparkSubmit #CleverStudies
Follow me on LinkedIn
/ nareshkumarboddupally
-----------------------------------------------------------------------------
Follow this link to join 'Clever Studies' official WhatsApp groups:
chat.whatsapp....
Community: chat.whatsapp....
--------------------------------------------------
Follow this link to join 'Clever Studies' official telegram channel:
t.me/+eMaiZNWT...
--------------------------------------------------
(Who choose Paid Membership option will get the following benefits)
Watch premium YT videos in our channel
Mock Interview and Feedback
Gdrive access for Bigdata Materials (Complimentary)
--------------------------------------------------
PySpark by Naresh playlist:
• PYSPARK BY NARESH
--------------------------------------------------
PySpark Software Installation:
• Latest Hadoop 3.2.2 Sp...
--------------------------------------------------
Realtime Interview playlist :
• How To Explain Project...
--------------------------------------------------
Apache Spark playlist :
• How Spark Executes A P...
--------------------------------------------------
PySpark playlist:
• PySpark | Tutorial-9 |...
--------------------------------------------------
Apache Hadoop playlist:
• APACHE HADOOP
--------------------------------------------------
Bigdata playlist:
• BIGDATA
--------------------------------------------------
Scala Playlist:
• SCALA TUTORIALS
--------------------------------------------------
SQL Playlist:
• SQL
Hello Viewers,
We ‘Clever Studies’ KZbin Channel formed by group of experienced software professionals to fill the gap in the industry by providing free content on software tutorials, mock interviews, study materials, interview tips, knowledge sharing by Real-time working professionals and many more to help the freshers, working professionals, software aspirants to get a job.
If you like our videos, please do subscribe and share within your friends circle.
Contact us : [e-mail]
Thank you !

[@kharwar755]

Thanks a lot for making such lucid videos....You are doing a great job....❤️.. And I intentionally dont skip the ads on your videos to support you as much as possible...Thanks man.

[@cleverstudies]

No words to say for your comments. This is a great 👍 motivation for us to go ahead and do more videos for you people 😊.

[@ddsa211]

Each Executor memory 18 GB will be further divided into below things. Reserved Memory: Fixed 300 Mb We left with = 17.7 GB Out of this 17.7 GB 40% goes to User Memory : 7.08 GB 30% goes to execution Memory (to run join, groupby, etc) : 5.31 (approx 5 GB) 30% gees to storage Memory (which is referred for cache the data into memory, altough if we are not caching anything then all 60% can goes to execution part) : 5.31 (approx 5 Gb) Then your each executor having the 5 cores, then each core approximately gets 1 GB of Execution Memory and 1 GB of storage Memory.

[@shrutisharma2301]

This is the best explanation. Thanks a lot.

[@2412_Sujoy_Das]

Wonderful Explanation, Sir. But please clarify me one thing: Do we have dedicated Application Master for every worker node or just a single Application master for the entire cluster?? If the latter is true, why are we dedicating one core and 1.5 GB RAM from every worker node? Thank you

[@Sathishkumar-rl7gj]

Thanks for the info. We need to discuss cluster creation based on input data size sir.

[@haribachala]

Thanks for the explanantion, can you please cover about driver memory config also in coming vedios?

[@ayooanand858]

Thank you for the video. please explain how you computed 7% of 21 = 3

[@pravinpathak7934]

Memory overhead is set to either 10% of executor memory or 384mb, whichever is higher. it should be 2.1 gb.

[@cleverstudies]

It's a typo

[@Amarjeet-fb3lk]

If no. of cores are 5 per executor, At shuffle time, by default it creates 200 partitions,how that 200 partitions will be created,if no of cores are less, because 1 partition will be stored on 1 core. Suppose, that My config is, 2 executor each with 5 core. Now, how it will create 200 partitions if I do a group by operation? There are 10 cores, and 200 partitions are required to store them, right? How is that possible?

[@ultimo8458]

i have applay 4x memory in each core for 5Gb file but no luck can you please help me to how to resolve this issue Road map: 1)Find the number of partition -->5GB(10240mb)/128mb=40 2)find the CPU cores for maximum parallelism -->40 cores for partition 3)find the maximum allowed CPU cores for each executor -->5 cores per executor for Yarn 4)number of executors=total cores/executor cores -> 40/5=8 executors Amount of memory is required Road map: 1)Find the partition size -> by default size is 128mb 2)assign a minimum of 4x memory for each core -> what is applay ??????? 3)multiple it by executor cores to get executor memory ->????

[@RangaReddy1988]

Hi Very good explanation. You didn't speak about Disk memory?

[@cleverstudies]

Thanks for watching. We will discuss disc space utilization in persistence topic.

[@guptaashok121]

As per this calculation we have about 540 gb memory available with all 39 executor, does it mean we can not process file larger than this size. If it has to read 1080 gb file, how will it process. Just trying to understand

[@shreyansh1023]

How to decide these parameters based on the data size that need to be processed, as there can be multiple spark applications running in big clusters.

[@cleverstudies]

Valid question. We will discuss in our coming videos.

[@pravinpathak7934]

@@cleverstudies is the video out on this??

[@rahulpatidar5109]

@@pravinpathak7934 @Clever Studies @Shreyansh Chougule same confusion...asked many people but couldn't find a clear answer...did you get the answer? kindly share if you guys have any idea .

[@kundansingh-zy9cb]

How to calculate cluster configuration based on our data size?

[@cleverstudies]

To know more, Please join our mini project execution group by sending your mobile number to [e-mail]

[@o0aleex0o]

What about driver memory???

[@Sharath_NK98]

10 nodes 16 cores each 64 GB ram Calculate Executors ? This is the interview question i faced today ??😢

[@amanraj3038]

21 of 7% is not 3….how

[@cleverstudies]

It's a typo mistake.

[@pratyushbhatt1712]

7% of 21 is not 3GB, even 10% of 21 is 2.1GB, @ 4:15

[@cleverstudies]

Yes you are correct. It's a typo

[@jnana1985]

7 % of 21 GB is 3 GB ? How

[@cleverstudies]

It was a typo. Pls ignore

[@jnana1985]

@@cleverstudies So can you pls explain the calculation for yarn memory overhead?