3. RDD partitioning | Repartition() vs Coalesce

[@ManaviVideos]

Thanks for sharing!

[@manu77564]

Thankyou

[@ideekshithreddyg]

Thank you for making the video,Could you please make a video on spark architecture on how data is shuffled across different executors

[@ABQ06]

Is hive metastore still relevant in current market? Or do i need to learn unity catalog? I mean for interview preparation I have working experience on hive metastore setup Well i will learn UC but as of now is it necessary to get a job? Please answer

[@nitinpandey4857]

Hi Annu, The reason why you are having 8 default partitions is because the number of cores in your Spark cluster is having 8 says, supposedly you have a spark cluster with four cores then the default partition will be 4

[@azurecontentannu6399]

​@@nitinpandey4857Appreciate your clarification.. But If you see I have 4 vCores . Probably the calculation behind goes 2*no. Of vCores for default partition?

[@azurecontentannu6399]

Even when I changed the spark to large cluster having 16 vCores, I could see sc.defaultparallelism gave 8 as result.