Processing 25GB of data in Spark | How many Executors and how much Memory per Executor is required.

#pyspark #azuredataengineer #databricks #spark
Use the below link to enroll for our free materials and other course.
www.cleverstudies.in/
You can talk to me directly on Topmate by using the below link:
topmate.io/naresh_kumar_boddu...
Follow me on LinkedIn
/ nareshkumarboddupally
-----------------------------------------------------------------------------
Clever Studies Official WhatsApp Group joining link:
Clever Studies 2.0: chat.whatsapp.com/BXyqRBHsXcr...
Clever Studies: chat.whatsapp.com/C70cGzAEKC1...
--------------------------------------------------
Follow this link to join 'Clever Studies' official telegram channel:
t.me/+eMaiZNWTPmZkYmVl
--------------------------------------------------
Facebook: profile.php?...
Instagram: / cleverstudiesindia
PySpark by Naresh playlist:
• PYSPARK BY NARESH
--------------------------------------------------
Realtime Interview playlist:
• How To Explain Project...
--------------------------------------------------
Apache Spark playlist:
• How Spark Executes A P...
--------------------------------------------------
PySpark playlist:
• PySpark | Tutorial-9 |...
Hello Viewers,
We ‘Clever Studies’ KZbin Channel formed by group of experienced software professionals to fill the gap in the industry by providing free content on software tutorials, mock interviews, study materials, interview tips, knowledge sharing by Real-time working professionals and many more to help the freshers, working professionals, software aspirants to get a job.
If you like our videos, please do subscribe and share within your circle.
Contact us: [e-mail]
Thank you!

[@arindamnath1233]

Wonderful Explanation.

[@shivamchandan50]

plz make video on pyspark unit testing

[@anubhavsingh2290]

Simple explanation Great sir 🙌

[@cleverstudies]

Thanq

[@yadi4diamond]

You are simply superb.

[@cleverstudies]

Thank you 🙏

[@aditya9c]

If num of partition is 200 ... And so it the number of core required ... So core size is 128mb ... Right ? Then how in 3rd block core size turn to 512mb and thus executer is then 4*512 ????

[@shibhamalik1274]

Is it that each core would take 4 * partition size memory ?

[@Amarjeet-fb3lk]

What is use of giving each core 512 mb,if blcok size is 128 MB. Each block process on a single core,so if each block is 128 mb, why we should give 512mb To each core? There will be wastage of memory,Am I right? Please explain this. Thanks

[@dineshughade6741]

Zuper

[@kingoyster3246]

what if we have limited resource? what configuration would you recommend to process 25GB? (16 cores and 32GB)

[@paulinaadamski8233]

You would have to choose between an increased partition size or lowered parallelism with an increased number of partitions.

[@user-nv6ho7uk8b]

Hi, Does the same study applies if we are working in Data Bricks?

[@bhanuprakashtadepalli7248]

yes, its same logic

[@user-dv1ry5cs7e]

for example you are assigning 25 executors instead of 50 then in each executors there will be 8 cores and parallel task will be run(25*8). Then also it will take 5 mins only to complete the job then how 10min. can you please explain this point once again?

[@vamshi878]

For each executor 2-5 cores should be there, so he is saying he is going to take 4 this number is fixed, if the data size increased or increased

[@kamatchiprabu]

Sir,I want to join Job ready program.How to join .Link is not enabled.pls help

[@cleverstudies]

Sorry, we are not conducting CSJRP sessions at present. Please check our website www.cleverstudies.in for more details.

[@shibhamalik1274]

There are 200 cores in total . Each core will use one partition at a time so will use 128MB Each executor has 4 core so each executor requires 4*128 MB which is 512 mb. Where does extra 4 multiplier came from ?😊

[@bhanuprakashtadepalli7248]

by default, to process a file in one core, we need 4 times the file size memory.

[@Fresh-sh2gc]

In my company the cpu per executor is 5 min and 8 max.

[@cleverstudies]

It depends on the use case and resources availability.

[@Fresh-sh2gc]

@@cleverstudies depends on cluster. We have a state of the art one over $1b data center that can support high cpu’s per executor