Spark Performance Tuning | EXECUTOR Tuning | Interview Question

[@RohanKumar-mh3pt]

Very Nice and clear explanation before this video i was very confused regarding executor tuning part now after this video it is now crystal clear.

[@TheFaso1964]

Dude. I feel like I knew nothing about spark in particular before I got my hands dirty with your performance improvement solutions. Appreciate a lot, got my subscription. Cheers from Germany !

[@TechWithViresh]

Thanks a lot :)

[@sankarn6016]

Nice Explanation!! can we use this approach for tuning/triggering multiple jobs in cluster ??

[@nivedita5639]

Very very helpful. Thanks

[@fahad_ishaqwala]

Excellent videos brother. Much Appreciated. Can you do a video on Performance Tuning for Spark Structured Streaming jobs as well.

[@TechWithViresh]

Surely, Working on a video for the same.

[@aneksingh4496]

As always best !!! Please include some real simulation example s

[@whatever-genuine7945]

How to allocate executers, core and memory if there are multiple jobs running on the cluster?

[@giyama]

This calculation is for just one job, what would be the calculation for multiple jobs running simultaneously? And how to calculate based on the volumetry? (Great job btw, tks!)

[@SidharthanPV]

Dynamic allocation is currently supported. You can set the max limit, yarn takes care of managing it in case of multiple instances running parallel.

[@umeshkatighar3635]

What If each node has only 8cores?? How does spark allocate 5cores per jvm ?

[@KNOW-HOW-HUB]

To process 1TB data what could be the best approach we have to follow

[@ranju184]

excellent explanation. Thanks

[@Dipanki-c7k]

What if I have multiple spark jobs in parallel in on spark session

[@DilipDiwakarAricent]

If not configure , so what will be the default number choose by spark.

[@inferno9004]

@5:10 can you explain how 20GB + 7% of 20GB is 23GB and not 21.4GB ?

[@rockngelement]

calculation mistake bhai, anyway it doesn't affect the info in this video

[@manisekhar4446]

According to your eg. How much GB if data can be processed by spark job??

[@mdmoniruzzaman703]

Hi, 10 nodes means including the master node? i have a configuration like this: "Instances": { "InstanceGroups": [ { "Name": "Master nodes", "Market": "SPOT", "InstanceRole": "MASTER", "InstanceType": "m5.4xlarge", "InstanceCount": 1 }, { "Name": "Worker nodes", "Market": "SPOT", "InstanceRole": "CORE", "InstanceType": "m5.4xlarge", "InstanceCount": 9 } ], "KeepJobFlowAliveWhenNoSteps": false, "TerminationProtected": false },

[@sivavulli7487]

Hi Sir , thank you for your nice explanation but if only one job is running over the cluster , that is more meaningful and understandable ..what if there are so many jobs running on the same cluster ??

[@TechWithViresh]

Based on the executor params passed for the each , that defines the container boundaries or running scope for that.If there are not enough resources available to be allocated, then that job(s) would be in queue.

[@sivavulli7487]

@@TechWithViresh so executor core can run only one job task at a time ..so if that is the case , in your examples , if there are 2 jobs on the same cluster, we need to take half of the resources mentioned in that video or better to take whatever you mentioned ..then first job runs successfully , it will take second job??( Until first job completed, second will be in queue).. could you please suggest best approach...alltogather before giving spark resource configurations for any job , just if we look at the cluster configuration is enough or need to look at how many other jobs running on the same cluster??

[@TechWithViresh]

@@sivavulli7487 Yes, we should take into account, how many concurrent jobs need to be run .How better approach followed these days to have interactive clusters for each job..

[@sivavulli7487]

@@TechWithViresh okay ..thank you sir ..if possible , pls can you make a video how to give the resources if there are multiple concurrent jobs running on the same cluster...

[@anusha0504]

What are advanced spark technologies

[@SpiritOfIndiaaa]

thanks bro , really wonderful explanation.... bro , can you make some vid on how to analyze Stages , Physical Plans etc on SparkUI ...based on that how to fix the issues regarding optimization ... its always confusing a lot to interpret these sql explain plans?

[@TechWithViresh]

Thanks very much, check out the video on stage details

[@SpiritOfIndiaaa]

@@TechWithViresh i dont find it, any url plz

[@snehakavinkar2240]

How to decide these configurations for a certain volume of data? Thank you.

[@TechWithViresh]

idea is to make sure max 5 tasks per executor, and the partition size is within the memory allocated to exec

[@snehakavinkar2240]

Is there any upper or lower limit to the amount of memory per executor?

[@TechWithViresh]

depends on the total memory resource available in your cluster.

[@rikuntri]

One executor is having four core so it can handle one task or 4 at a time

[@the_high_flyer]

No of cores = no of parallel task

[@girijapanda1306]

7% of 21GB = 1.4 GB am I missing something here

[@RAB-fu4rw]

7% of 21 gb is 3gb ????? how come it is 1.47 GB how did u arrive at 3 GB ???

[@komalkarnam1429]

Yes had the same question

[@divyar7991]

For yarn you can choose between 6 to 10 per

[@KiranKumar-cg3yg]

Means what I know is nothing.