Processing 25GB of data in Spark | How many Executors and how much Memory per Executor is required.

[@shafimahmed7711]

Your explanation of the Spark cluster and memory configurations was excellent. I really appreciate it!

[@anubhavsingh2290]

Simple explanation Great sir 🙌

[@cleverstudies]

Thanq

[@shivamchandan50]

plz make video on pyspark unit testing

[@anirudh2704]

Good explanation. Spark is all about good resource allocation or use and optimization

[@kusumal190]

Amazing explanation 👍🏻👏🏻👏🏻👏🏻

[@sravankumar1767]

Superb explanation 👌 👏 👍

[@yadi4diamond]

You are simply superb.

[@cleverstudies]

Thank you 🙏

[@sureshpujari2510]

Awesome explanation

[@arindamnath1233]

Wonderful Explanation.

[@aditya9c]

If num of partition is 200 ... And so it the number of core required ... So core size is 128mb ... Right ? Then how in 3rd block core size turn to 512mb and thus executer is then 4*512 ????

[@PravinUser]

in each core memory should be minimum 4 times of data it is going to process(128mb) roughly it should be minimum 512 mb of memory.

[@snagendra5415]

How many nodes does it requires ? Can anyone help me

[@tanushreenagar3116]

perfect video sir

[@VikasChavan-v1c]

for example you are assigning 25 executors instead of 50 then in each executors there will be 8 cores and parallel task will be run(25*8). Then also it will take 5 mins only to complete the job then how 10min. can you please explain this point once again?

[@vamshi878]

For each executor 2-5 cores should be there, so he is saying he is going to take 4 this number is fixed, if the data size increased or increased

[@kingoyster3246]

what if we have limited resource? what configuration would you recommend to process 25GB? (16 cores and 32GB)

[@paulinaadamski8233]

You would have to choose between an increased partition size or lowered parallelism with an increased number of partitions.

[@Rakesh-q7m8r]

Hi, Does the same study applies if we are working in Data Bricks?

[@bhanuprakashtadepalli7248]

yes, its same logic

[@Fresh-sh2gc]

In my company the cpu per executor is 5 min and 8 max.

[@cleverstudies]

It depends on the use case and resources availability.

[@Fresh-sh2gc]

@@cleverstudies depends on cluster. We have a state of the art one over $1b data center that can support high cpu’s per executor

[@shibhamalik1274]

Is it that each core would take 4 * partition size memory ?

[@anirudh2704]

1 executor.the best configuration is1 executor = 4 cores, 512 mb There's concept of fat and thin executors

[@shibhamalik1274]

There are 200 cores in total . Each core will use one partition at a time so will use 128MB Each executor has 4 core so each executor requires 4*128 MB which is 512 mb. Where does extra 4 multiplier came from ?😊

[@bhanuprakashtadepalli7248]

by default, to process a file in one core, we need 4 times the file size memory.

[@anirudh2704]

Spark is in memory processing. So it requires min 512mb of memory to perform cache, persist, shuffling and overhead tasks. 1 core handles 1 block of data.

[@kamatchiprabu]

Sir,I want to join Job ready program.How to join .Link is not enabled.pls help

[@cleverstudies]

Sorry, we are not conducting CSJRP sessions at present. Please check our website www.cleverstudies.in for more details.

[@dineshughade6741]

Zuper

[@Amarjeet-fb3lk]

What is use of giving each core 512 mb,if blcok size is 128 MB. Each block process on a single core,so if each block is 128 mb, why we should give 512mb To each core? There will be wastage of memory,Am I right? Please explain this. Thanks

[@debayanmitter]

The memory is for processing, not for storage.

[@anirudh2704]

The min req of executor is 4-5 cores and 512 mb memory. 1 core can handle 1 block data. And as spark is in memory processing so it requires memory space for cache, persist, shuffling etc