Spark Executor Tuning | Decide Number Of Executors and Memory | Spark Tutorial Interview Questions

As part of our spark Interview question Series, we want to help you prepare for your spark interviews. We will discuss various topics about spark like Lineage, reduceby vs group by, yarn client mode vs yarn cluster mode etc. As part of this video we are covering
what is role of spark context.
Please subscribe to our channel.
Here is link to other spark interview questions
• 2.5 Transformations Vs...
Here is link to other Hadoop interview questions
• 1.1 Why Spark is Faste...

[@rishi190]

Number of cores = Concurrent tasks as executor can run So we might think, more concurrent tasks for each executor will give better performance. But research shows that any application with more than 5 concurrent tasks, would lead to bad show. So stick this to 5. This number came from the ability of executor and not from how many cores a system has. So the number 5 stays same even if you have double(32) cores in the CPU.

[@DataSavvy]

Thanks for inputs... Useful for community

[@shaiksubhani9802]

Really appreciate your efforts.. thanks for sharing your knowledge.

[@manieshsh]

You are doing exceptional work. Love 👍🏻

[@DataSavvy]

Thanks Manish :)

[@ananthasubramanian7355]

Basic but important concept w.r.t Spark. Well explained...!! Thank you for this video..!!

[@DataSavvy]

Thanks Avantha :)

[@baritos4]

Hi - I have question regarding reserved memory (300MB) that is no available for any spark allocation. During your calculation of 64GB per node , why wsnt this 300MB deducted . You deducted 1GB for OS operations (did it include this reserved memory of 300MB)?Also if we enable spark.memory.offheap , should we deduct the offheap memory size as well from 64gb?

[@sandeepmaheshuni]

Can you elaborate, how many cores and memory need to be allocated individually directly w.r.t spark-submit configuration. coz we have driver/executor memory, driver/executor cores. This is most expected question in interview.

[@abhinavsingh9333]

Nice video, well explained.

[@karanrewari6482]

Thanks for the great video..really helped me understand thing from a fundamental level. One question - given the same context, how would you fit in driver memory requirements assuming you need 4GB of driver memory. How about the cores for it as well ? The driver memory only needs to be on 1 machine, while others can play the role of workers (how is this configured ?)

[@Sunkarakrwutarth9389]

thanks for your video, how to decide the number of workers for spark cluster in the data bricks environment.

[@remoram8141]

Hi , We are running HDFS/SPARK on same machines and total we have 7 nodes in cluster.each node having 94GB ram ,16 cores....so,how much resource we need to give for spark daemons and hdfs..we are getting heavy memory usage alerts from spark workers..can u plz help me in this...

[@amruthpesari7922]

Hi Sir, Could you please elaborate on " contention between threads for IO from HDFS " which is the answer for "why 5 is no. of cores per executor"

[@arryanderson]

I have a question. If there is 1 core assigned for the OS operation each machine, then doesnt that mean 4GB is assigned for OS? Little confused on how we arrived at 1GB for OS per machine. Also, if 21 GB per executor is divided among 5 cores, does that mean each core has 4.2 GB in its disposal?

[@sjdee13]

Please make a video on debugging spark code in an ide like eclipse. Thanks

[@zillala4553]

Great.... It's so clear explanation 👍👍

[@sundaramoorthysivasamy9784]

Hi , if it is shared cluster and we are provided with a resource queue of 1000 cores and 4000 GB memory , if we have 10 hourly jobs that needs to run in that resource queue , how to provide executor core and memory for each job. Some job might read TB volume , some in GB.

[@lavanyareddy310]

Great video sir,thank u so much for all u r videos

[@DataSavvy]

Thanks Lavanya :)

[@nimishbajaj6247]

Can there be scenarios when 5 cores per executor be a bad design, lets say we need more memory per executor? In that case, it might be better to have 3 executors in total with all the cores and memory we can give to the individual executors. (Will this scenario occur when the data is not properly partitioned/ cannot be equally paritioned)

[@study-channel6301]

What if there are a lot of Spark jobs running in the same cluster being developed by different developers with the same machine configurations? What considerations should be taken in that case?

[@ibrarahmed8563]

i get the concept of least and most granular aproaches but in both aproaches we have no resources left, the points you have made for most granular aproach,can also be considered for least granular approach( first approach ) bcz we also have no resources left there as well ? pls explain!!

[@phanikumar4915]

Hi Is it possible to decide on which executer/machine our job should run

[@mubasshirali1986]

Thank you so much for the informative video, Sir. You said there's a formula for allocating the memory for Yarn Overhead. As a kind reminder you said at 8:27 that you'd share it in comment/description section... Thanks in advance

[@shubhamshingi4657]

many ppls keep it 7 % of executor memory, but now a days 10% is also used

[@mubasshirali1986]

@@shubhamshingi4657 Thanks for the knowledge Sir 🙏

[@chiranjeevikatta8116]

So if we allocate all resources to our single spark job. What will happen if I have to run others jobs parallel.

[@saurabhpant44]

Hello, thanks for a very clear explanation of the concept. I have a question though. You have mentioned that, if we use a lot of cores for a single executor, all cores will compete with each other to read data from HDFS and will cause IO contention. But, how is it different from the case when we have 5 cores each for 3 executors. All these 3 executors and hence 15 cores will still try to read the data from HDFS in parallel. Isn't it or am I missing something here?

[@sachinyadahalli5745]

Yes, had same question in my mind. Could anyone comment on this ?

[@peace-ye2lm]

we can mention many parallel task at a time to run in a executor but spark decides internally at that point of time only certain number of tasks will run in parallel remaining will wait till the other tasks complete , ones a task completes the one which was in waiting state will start running

[@raghuram9130]

Hi, no of executor core you have decided as 5.can you please confirm how you decided that value?

[@sunilsambarekar3144]

Thank you, this is very helpful

[@DataSavvy]

Thanks sunil... Please provide feedback on other videos also

[@udaykumarreddygajjala7242]

I have a doubt If a spark job is running How do we check how much memory we allocated to that job? Apart from yarn UI?

[@lazyengineer1994]

$ yarn top

[@rameshbabuy9254]

can you configure below properties based on above example cluster in the video spark.executor.memory = spark.executor.cores = spark.executor.instances = spark.yarn.executor.memoryOverhead = spark.default.parallelism =

[@prakashmudliyar4834]

Can Dynamic Resource allocation takes are of all this?

[@ankush_sagane]

can u please share dynamic memory allocation as well?

[@shyamsundar8665]

Please do create a video on explaining 1) yarn memory overhead(off and on heap ) - what it is and its significance 2) garbage collection - what is this memory and how to resolve gc out of limit error

[@tejasriin]

Hi could you please give a good suggestion for my cluster 12*256 memory and 12*56 core and disk of 12*14tb

[@DataSavvy]

You need to understand nature of your job to precisely tell what is configuration required by your job... You job can be cpu heavy or Io heavy... Assign resources based on that

[@maheshkumarpurushothaman6167]

It is really helpful video. If i run spark job with shuffle partitions would that need to be considered to have additional cores/ memory for executors?

[@DataSavvy]

You should always try to reduce shuffle... Increasing cores will not help here... Increasing memory overhead for shuffle service will help but still shuffle is expensive operation

[@Raghav54321]

Please share the formula. As you have mentioned in video

[@user-he9nw7th8u]

Spark and Kafka and scala,how to tell the rolls and Responsibility

[@sachinyadahalli5745]

dont we have to decide any cores or memory for driver ? Or is it handled by spark itself. bcz I have seen many jobs where we specify as --driver-memory 10G etc. how about the cores for driver ?

[@DataSavvy]

You are right...Driver memory can also be set for your job... It is generally used when you collect big data in your job or your driver is very memory intensive... It is not considered good thing as if you are trying to do your processing in driver then that mean you are not using power of spark properly

[@ankitsrivastava2914]

Interview question - f job is running in 1 hour in production what you will do?

[@santoshkumar-jd9ns]

Is there any formula to take executor cores to be 5

[@RAVIKUMAR74]

how to decide calculate no of executor and memory when multiple jobs are runing by multiple proresses in cluster? what is the best way to tune our process. is this good way to give 50 executor with 16gb each. could you please explain with example.

[@bhaswatirout5764]

I have been asked this question twice in interview: Assume the table has 1TB(certain amount) data , how do you decide no of executors in spark submit? In this case is there any formula for this or only we can say based on past experience and hit n trial.

[@DataSavvy]

I will create a video on this

[@ajaypratap4025]

Data Savvy when?

[@rikuntri]

@@DataSavvy is this video available?

[@sashanksah4864]

@Data savvy: is the video available? also couldn't find the yarn overhead formula.. plz provide us.

[@ayushjain139]

@@DataSavvy 1TB/128MB(HDF blocksize) would be the cores you would want to have. Now with 5 cores per executor idealogy. Roughly comes down to ~1562 executors for max parallelism. Of course, based on available processing power this would be limited. Please correct this answer if someone knows the correct answer.

[@harekrushnamishra691]

Hi Sir ....Could you please explain how many executors we should go ahead with if we have 100 gb input data, 4 node cluster with 16gb ram and 8 cores each ? Asked in one of the interview question.

[@neelbanerjee7875]

Similar question i too had..

[@nehabansal677]

Very helpful for interviews

[@DataSavvy]

Thanks

[@DataSavvy]

Please watch full spark interview series

[@tolasebrisco6565]

Keep the good work #Prinetechs I can clearly see all the good reviews about you man…i never believed my account can fixed after 7 months hahaha

[@TheRn35]

1. So this says at best what can be assigned if only one job is running, right? What we have multiple independent batch jobs scheduled in parallel? 2. The calculation only factors in the cluster configuration. Shouldn't the data volume and type of operations being performed also be deciding factors? 3. Also why bother about these when dynamic allocation is available?

[@SpiritOfIndiaaa]

Thanks a lot sir , giving some good idea about how to allocate resources in spark. I am working on ETL tool but having real problem in tuning my environment...can you please help me.

[@DataSavvy]

Hi... Thank you... I will love to help on this... Which Etl tool are you using... I have knowledge about big data related things... Not much aware of Informatica etc

[@sangamchoubey2772]

How can we decide on no of executors and memory based upon the data input , like 1 TB ?

[@DataSavvy]

You should check how many partitions you have, what is approximate size of each partition and if you are doing any joins or aggregation how number of partitions were changing after that... This will give you good hints about hope many executors and executor cores you should use

[@abhishekkrishna9757]

Thanks for sharing. If possible, could you please share some scala intervew questions?

[@DataSavvy]

Sure, will create a series on Scala interview questions... Please subscribe to our channel 😊

[@abhishekkrishna9757]

Thanks for your reply. Already subscribed :)

[@user-he9nw7th8u]

Good job ,pls use Paint(board)

[@lifefactsfun]

Can you make video on the complete life cycle of a spark project.Including how unit testing is done by developers.

[@DataSavvy]

Sure I will create a video on this

[@riyasmohammad9234]

Can you make a playlist for spark from scratch?

[@JS-gg4px]

I am so confused when you mention cores and executors back force , which one is one ? lol

[@bhargavhr1891]

Very good content, I would like to know how will we increase the number of cores as a part of cluster upgrade, also every time the cores are increased in the cluster, does the formula change?

[@DataSavvy]

As you add more machines... More CPU are added, hence more cores are added... Sorry for delay in response

[@4ukcs2004]

even though cluster upgrade happens ,maximum cores to be used is 5 as per Hortonworks documentations.

[@aneksingh4496]

Please share some real time problem like job stuck or some use case .. that will be useful

[@IrfanShaikh-jt6yu]

awsome sir please make same as for yarn tunning with atleast 10 machine

[@sumitkrs2]

Very well explained..I have got the question here. We decide number of executors for each spark job but what will happen when we submit multiple jobs with same number of executors.

[@DataSavvy]

If those amount of resources are available, then all jobs will get resources... If those resources are not available then job priority and when job was submitted comes in picture

[@DataSavvy]

I am happy that you liked the video :)

[@gouthamanush]

Hi, I have been following your content for quite sometime. Really good content. In this example, you have assumed that we have to utilise all the existing resources. But what if the data is just 200MB. Would you still use the same configuration? Shouldn't the decision be based on how much input data is and then decide accordingly? I am confused here.

[@ratnawaliparshetti904]

Hi..Even i have same question.Did you recive this answer.

[@DataSavvy]

Hi... You are right. In my example, I assumed that we have huge data and we have to use all resources. But when data size is small we need to allocate based on amount of data we process and what will be in memory footprint... Spot on :)

[@AmitPrasadbangalore]

Hi , You are doing great job bro. I have a question though at 7:17 , how did we decide to have no of executor as 5 cores ?

[@mohitmanna7308]

broo do u have answer now?

[@krishnarupeshnagubandi596]

can you please elaborate.

[@mdmoshiuzzaman]

Hi, First of all thanks for your great efforts!!! :) Now coming to my doubt,this may sound very childish to others :( but I am curious to know how we will loose parallelism in case of tiny executor approach? cant different task run on different executor parallely ?

[@suvasishudgata7326]

Hi, yes your question is valid... you will not lose parallelism in the first case. A rather better statement would be you will not be using the parallelism to its max potential, in the 1st case you will have 1core/executor and in the 2nd one you will have 3 executor/machine with 5 core for each executor.

[@gouravchoubey860]

How did we derived 5 cores per executor?Was it mere guess?Also how did we reached number of executors

[@DataSavvy]

5 cores is usually a standard when you are defining no of cores... Having more than 5 threads(executor cores) will increase contentions among threads for resources... However there is no hard and fast rule.. u can try with 4 or 6 if that gives u better performance...

[@amarnathkal6467]

What happens if one of the Machine/ Workernode goes down in middle of the data processing? Can someone help please.

[@DataSavvy]

If a machine fails, executors on that machine will fail. They will be restarted on another machine

[@amarnathkal6467]

@@DataSavvy Thanks Bro👍👍👍

[@bhargavhr1891]

Hi Harjeet, could you please share the formula for finding the exact cores and memory for an cluster who's configuration would be different from what you have explained here

[@DataSavvy]

Sure, Bhargav... Will put it together and post here

[@Raghav54321]

@@DataSavvy please share...still waiting

[@harjeetkumar4632]

Great quality of content... Can you make more videos on this topic

[@DataSavvy]

Thank you

[@lifefactsfun]

How to resolve a spark job that is stuck for a long time during runtime?

[@DataSavvy]

There can be multiple reason for job to stuck... One most common reason is excessive shuffle... Other can be failure of executor again and again.. other could be job waiting to aquire resources as the resources on cluster may not be free

[@ShubhamJain-qx4hx]

@@DataSavvy What should be approach to solve this?

[@rameshthamizhselvan2458]

I have one doubt for a long days .. how number of excecutor core becomes 5 suddenly ????? is this assumption ??

[@DataSavvy]

Hi Ramesh... Number of cores = Concurrent tasks as executor can run So we might think, more concurrent tasks for each executor will give better performance. But research shows that any application with more than 5 concurrent tasks, would lead to bad show. So stick this to 5. This number came from the ability of executor and not from how many cores a system has. So the number 5 stays same even if you have double(32) cores in the CPU.

[@DataSavvy]

You may decrease it from 5 to 3 while executors on samne machine may increase... More than 5 cores create contention for resources within executor and reduction in write througput

[@chaithanyakrishnaravindrak7032]

Hi, Please share the formula as described in your video.

[@DataSavvy]

Will try to put this down

[@kaladharnaidusompalyam851]

Hi. Why spark is in memory? Expecting clear solution from you?

[@DataSavvy]

Hi... Spark processing is not always in memory. U can chose not to cache data and do processing. Caching shd be used only when u are doing operations on same data again and again..

[@rajudakoju]

Please have videos about Data Bricks

[@neelbanerjee7875]

Please make an video on optimization based on data size.. how to choose executor num, memory, core depending upon file size..

[@rohithgorthy971]

Thin vs Fat executor, optimum selection!

[@ranganath031]

why is the no of executor cores is 5?

[@areyoufreetolisten]

So nbr of executor per m/c = (no. of core - 1)/5 and mem per executor = ((tot mem - 1gb)/no. of executors) - 2gb overhead

[@DataSavvy]

Yup... That's the max on machine... U will need lesser number of executors on a single machine in actual

[@ayushjain139]

​@@DataSavvy So, --executor-memory will be ((tot mem - 1gb)/no. of executors) or ((tot mem - 1gb)/no. of executors) - 2gb overhead?

[@venkatraju6553]

Thanks for the content...audio was low...Please do increase in future video's.....!

[@DataSavvy]

Yes Venkat... Audio quality is going down when I upload it on KZbin... I am trying to find a solution for this

[@vishal7598]

Hi ...Your videos are great to see. Could you please tell me how you decide the No of executor cores to 5?

[@DataSavvy]

Good question my friend... This number is more from experience perspective. If u give more than 5 cores to executor , then it is observed there is contention between threads for IO from HDFS. So you data read and write will become slow.. 5 is generally more accepted number.. Hope this helps...

[@vishal7598]

Thanks for your reply. So it totally depends on heat and trial to get to know the performance.

[@DataSavvy]

VISHAL ANAND not exactly... Only the 5 number of executors per executor are outcome of experience... Rest is still math

[@maheshrichhariya6731]

@@DataSavvy i think one executor we eliminate for yarn..

[@Rahulwagh234]

How did you reach to Number of Executor Core should be 5

[@DataSavvy]

That number is arrived with hit and trial... If you job is Io intensive, then it may change

[@rikuntri]

He explained that if we apply less core than we will not utilise the jvm and if high io response will increase .so executor core should be in mid range not granular or too high

[@DataSavvy]

Number 5 is also driven by capability of executor to handle parallelism...

[@SpiritOfIndiaaa]

Can you please elobarate on No of executor cores : 5 and per machine : 3 , really confusing

[@DataSavvy]

Hi... Could not understand this question... Please elaborate

[@sandeepmaheshuni]

After calculation, Out of 5 cores, 3 will be allocated to avoid I/O contention in HDFS.

[@rikuntri]

@@sandeepmaheshuni no .Kindly go through the video again or basic architecture of spark.The thing you missed is Executor core and executor

[@rikuntri]

no .Kindly go through the video again or basic architecture of spark.The thing you missed is Executor core and executor .

[@surabhiyadav7673]

Hi can you please share the formula to decide number of executors.

[@DataSavvy]

Sure... Will post that

[@sandeeppandita1]

Does each executor run on separate jvms?

[@31bikashdash]

yes

[@31bikashdash]

different jvm instance

[@lavanyareddy310]

How much salary we expect for spark developer for 3 years experience

[@DataSavvy]

That depends on your skill, current salary and role and company you have applied for... It will be misleading to give you a single answer

[@inkuban]

Nice work sir!! But where is the formula for overhead memory allocation? Please provide code recipes also please. If you make udemy videos let me know.

[@31bikashdash]

8 % percent of memory allocated to each executors

[@jeevithat6038]

Poor audio quality. You speak in low voice. Can you play improve this. Apart from this your videos are informative.

[@adityapratapsingh7649]

True.

[@insaniyat_1512]

audio thoda mute hi krdo ya tez rkho plz. aapki cdo k liye thank you aise.

[@mubasshirali1986]

Another Interview Question: How to distribute resources between Yarn, Impala and Spark ?

[@DataSavvy]

Thats a good question Mubasshir.. This question is asked usually for admin roles... will plan a video on this

[@mubasshirali1986]

@@DataSavvy Thank you so much Sir. . Actually I'm looking for an admin job profile. Will be waiting for your video on my request. Gratitude.

[@mubasshirali1986]

@@DataSavvy Sir, Hope you're doing great. Kind reminder for requested query about: How to decide/share resources between Yarn, Impala and Spark. Thanks in advance.

[@anusha0504]

What are advanced spark technologies

[@DataSavvy]

Can you elaborate on your question... Do u mean to ask what are new features of spark?

[@anusha0504]

@@DataSavvy Yes please

[@DataSavvy]

There are a lot... Delta lake, AQE, dynamic Partition Pruning, auto coalesce, structured streaming etc

[@anusha0504]

@@DataSavvy Thanks for the information

[@mohitmanna7308]

Simple Words: 1. Leave 1 Core & 1 GB for OS (per machine) So, 16-1 =15 Cores / Machine | 64-1= 63 GB RAM/machine 2. Now each executor should have 4 or 5 Cores. if less cores / executor then, under utilization of cluster. JVM / multithreading not being used to its fullest if more cores / executor then, too many IO on HDFS and thus HDFS IO Contention thus slow RW on HDFS So lets fix it at 5 in this case. Means 5 Cores / Executor means No. of executors / machine = 3 3. RAM We are left with 63 GB RAM and we have 3 Executors to divide into. Each gets equal share. 21 GB RAM/Executor Now, this was about Node. each Node has 21 GB RAM, 5 Cores. Each Node needs some YARN Memory Overhead where it stores variables, objects, strings. This space is also used for shuffling / optimization. Generally it should be 7-10% so for 21 GB Node it would be roughly 2 GB. So Memory left for "actual" Executor is 19 GB

[@adityatavde406]

why are you taking Number of Executor Core should be 5 . please make a video for this.

[@nafisaslam4605]

The number of concurrent task which an executor can process is directly proportional to the number of cores. More than five task per executor degrades the performance (it's the tried and tested number). That's the reason here 5 cores have been selected per executor.

[@adityatavde406]

@@nafisaslam4605 ..do you have any example for that as you said..'degrades the performance (it's the tried and tested number)'

[@nafisaslam4605]

@@adityatavde406 I can't share the screenshot due to data security policy of our client, but I suggest you can change the number of cores on your job and check the metrics on spark web UI.

[@SpiritOfIndiaaa]

Would have explained pictorial diagram...

[@DataSavvy]

Good suggestion SHA... I will create a better version of this

[@snehakavinkar2240]

Does it mean if we have 10 jobs running in a cluster then all 10 would be using the same configuration? Thank you.

[@DataSavvy]

It does not mean that... based on how much data ur job is processing, how many partitions u have, and is ur job io intensive or compute intensive, u have to tweak these parameters

[@neelbanerjee7875]

@@DataSavvy please make a video on this...would be helpfull

[@vishalaaa1]

are you telling secret in low volume ????

[@DataSavvy]

Yes :)

[@srinidhisridharan9676]

Your audio level is bit low, kindly raise your decibel in your future videos.

[@insaniyat_1512]

audio level bdao yr plz

[@DataSavvy]

Yes Manish... I need to record this video again... I will fix this in new video

[@realMujeeb]

Barely able to hear you Sir!

[@shivamgarg9698]

didn't properly explained the disadvantages of small and large size executors

[@Rarchit]

Here is the formula Consider this aws ec2 machine r7g.4xlarge CPU core - 16 RAM - 128g Node - 5 ========================= tiny big approach --> 5 core per executors # spark.executor.core = 5 standard number of executor per node = (total vcpu - 1 [hadoop daemon] )/ spark.executor.core (15/5) = 3 # spark.executor.memory = (total RAM per instance / number of executors per instance) (128g / 3) = 42g 42 * 0.9 = 37 (round down) // 90% of total towards executor mem # spark.yarn.executor.memoryOverhead = 10% of total memory = 42g * 0.1 = 5 (round up) # spark.driver.memory = spark.executor.memory # spark.driver.cores = spark.executor.core # spark.executor.instances = (number of executors per instance * number of core instances) minus 1 for the driver) 3 * 5 - 1 = 14 # spark.default.parallelism = spark.executor.instances * spark.executors.cores * 2 14 * 5 * 2 = 140 =========================

[@adityatavde406]

Great video series!!! one suggestion: please make a already written script for explanation for video cause your fillers(e.g. ee ,aa uu) are really annoying.

[@tripathi123shailesh]

Please speak little more Lauder

[@DataSavvy]

Ok Shailesh

[@tolasebrisco6565]

Keep the good work #Prinetechs I can clearly see all the good reviews about you man…i never believed my account can fixed after 7 months hahaha

[@tolasebrisco6565]

Keep the good work #Prinetechs I can clearly see all the good reviews about you man…i never believed my account can fixed after 7 months hahaha