Processing 25GB of data in Spark | How many Executors and how much Memory per Executor is required.

#pyspark #azuredataengineer #databricks #spark
Use the below link to enroll for our free materials and other course.
www.cleverstud...
You can talk to me directly on Topmate by using the below link:
topmate.io/nar...
Follow me on LinkedIn
/ nareshkumarboddupally
-----------------------------------------------------------------------------
Clever Studies Official WhatsApp Group joining link:
Clever Studies 2.0: chat.whatsapp....
Clever Studies: chat.whatsapp.... (Full)
--------------------------------------------------
Follow this link to join 'Clever Studies' official telegram channel:
t.me/+eMaiZNWT...
--------------------------------------------------
Facebook: www.facebook.c...
Instagram: / cleverstudiesindia
PySpark by Naresh playlist:
• PYSPARK BY NARESH
--------------------------------------------------
Realtime Interview playlist:
• How To Explain Project...
--------------------------------------------------
Apache Spark playlist:
• How Spark Executes A P...
--------------------------------------------------
PySpark playlist:
• PySpark | Tutorial-9 |...
Hello Viewers,
We ‘Clever Studies’ KZbin Channel formed by group of experienced software professionals to fill the gap in the industry by providing free content on software tutorials, mock interviews, study materials, interview tips, knowledge sharing by Real-time working professionals and many more to help the freshers, working professionals, software aspirants to get a job.
If you like our videos, please do subscribe and share within your circle.
Contact us: [e-mail]
Thank you!

[@shafimahmed7711]

Your explanation of the Spark cluster and memory configurations was excellent. I really appreciate it!

[@anubhavsingh2290]

Simple explanation Great sir 🙌

[@cleverstudies]

Thanq

[@shivamchandan50]

plz make video on pyspark unit testing

[@anirudh2704]

Good explanation. Spark is all about good resource allocation or use and optimization

[@kusumal190]

Amazing explanation 👍🏻👏🏻👏🏻👏🏻

[@yadi4diamond]

You are simply superb.

[@cleverstudies]

Thank you 🙏

[@sravankumar1767]

Superb explanation 👌 👏 👍

[@jkarunkumar999]

Well explained

[@sureshpujari2510]

Awesome explanation

[@arindamnath1233]

Wonderful Explanation.

[@aditya9c]

If num of partition is 200 ... And so it the number of core required ... So core size is 128mb ... Right ? Then how in 3rd block core size turn to 512mb and thus executer is then 4*512 ????

[@PravinUser]

in each core memory should be minimum 4 times of data it is going to process(128mb) roughly it should be minimum 512 mb of memory.

[@tanushreenagar3116]

perfect video sir

[@snagendra5415]

How many nodes does it requires ? Can anyone help me

[@Rakesh-q7m8r]

Hi, Does the same study applies if we are working in Data Bricks?

[@bhanuprakashtadepalli7248]

yes, its same logic

[@VikasChavan-v1c]

for example you are assigning 25 executors instead of 50 then in each executors there will be 8 cores and parallel task will be run(25*8). Then also it will take 5 mins only to complete the job then how 10min. can you please explain this point once again?

[@vamshi878]

For each executor 2-5 cores should be there, so he is saying he is going to take 4 this number is fixed, if the data size increased or increased

[@kingoyster3246]

what if we have limited resource? what configuration would you recommend to process 25GB? (16 cores and 32GB)

[@paulinaadamski8233]

You would have to choose between an increased partition size or lowered parallelism with an increased number of partitions.

[@Fresh-sh2gc]

In my company the cpu per executor is 5 min and 8 max.

[@cleverstudies]

It depends on the use case and resources availability.

[@Fresh-sh2gc]

@@cleverstudies depends on cluster. We have a state of the art one over $1b data center that can support high cpu’s per executor

[@uoops87k8j76]

memory for each exectuor is 4*512 MB, what is this 512MB ?

[@kamatchiprabu]

Sir,I want to join Job ready program.How to join .Link is not enabled.pls help

[@cleverstudies]

Sorry, we are not conducting CSJRP sessions at present. Please check our website www.cleverstudies.in for more details.

[@shibhamalik1274]

Is it that each core would take 4 * partition size memory ?

[@anirudh2704]

1 executor.the best configuration is1 executor = 4 cores, 512 mb There's concept of fat and thin executors

[@shibhamalik1274]

There are 200 cores in total . Each core will use one partition at a time so will use 128MB Each executor has 4 core so each executor requires 4*128 MB which is 512 mb. Where does extra 4 multiplier came from ?😊

[@bhanuprakashtadepalli7248]

by default, to process a file in one core, we need 4 times the file size memory.

[@anirudh2704]

Spark is in memory processing. So it requires min 512mb of memory to perform cache, persist, shuffling and overhead tasks. 1 core handles 1 block of data.

[@dineshughade6741]

Zuper

[@Amarjeet-fb3lk]

What is use of giving each core 512 mb,if blcok size is 128 MB. Each block process on a single core,so if each block is 128 mb, why we should give 512mb To each core? There will be wastage of memory,Am I right? Please explain this. Thanks

[@debayanmitter]

The memory is for processing, not for storage.

[@anirudh2704]

The min req of executor is 4-5 cores and 512 mb memory. 1 core can handle 1 block data. And as spark is in memory processing so it requires memory space for cache, persist, shuffling etc