6.7 Decide Number Of Buckets in Hive and spark | Partition and Bucketing

[@gauravmathur56]

Awesome information... thank you harjeet for the great insights.....

[@DataSavvy]

Thanks Gaurav

[@diptiranjannayak5168]

How we will restrict the bucket size as per the block size dynamically? if I will mention 4 buckets and then what will happen if 1 bucket size will gradually increase above 1 GB or above? how I will achieve optimization?

[@DataSavvy]

You have to check your historical data and analyse what number makes sense for you... If you are expecting same size of data in future then just extrapolate those numbers

[@r.kishorekumar1388]

Is it possible to alter bucketted table to change number of buckets ?

[@jimitshah7636]

no, 1) create anther table with changed bucket number, 2) insert data from old table, 3) Drop the old table, 4) Rename the new table. Also you may need to stop the jobs running on this table till you are completed with this process.

[@simplecooking1323]

How we can decide whether we should do partition or bucketing ?

[@soumyakantarath5078]

Thank you so much 😊

[@routhmahesh9525]

Can we create buckets on top of partitioning ...can you please explain this?

[@rahulsamyal6159]

I got confused, block size is 128 mb and our memory can be 4 gb in size. In this case bucket should be 128 mb or 4 gb?

[@projjalchakraborty1806]

Can you plz explain what is Hcatalog and what is the use of it??

[@DataSavvy]

sure.. will create a video on this

[@rajareddy47444]

Please explain about oozie...how to schedule and workflows concepts...thank u

[@DataSavvy]

Ozzie is no more considered a good scheduler... It had lot of limitations... Most of companies are moving away from this...

[@rajareddy47444]

Data Savvy okay..can you please explain any scheduler most companies use..where we can talk about it and explain it... thank you

[@akashputti]

@@DataSavvy but a question is asked about this in interview

[@naresh5273]

Thank you

[@DataSavvy]

Please subscribe and share it

[@lokeshmvs]

I have a question? 1) buckets are created by writing clusters by. How we can implicitly give the number of buckets

[@DataSavvy]

Excuse me.. I could not understand your question... We give number of buckets while creating table. Are you asking how can give automatically fund out number of buckets?

[@vipulx1]

Thank you :)

[@divendughati6114]

Can you please explain how can we optimize if number of buckets get way too much about 1 million?

[@DataSavvy]

Hi Divendu, can you please elaborate more on this case. Like is bucketing creating lot of small files in this case? and what is ur usecase... if bucketing is creating small files then u should decrease number of buckets while creating table

[@AnkitaMishra-di9ub]

Can you please explain how to decide number of partition?

[@DataSavvy]

In hive, one partition is created per unique value... In spark it depends on no of blocks of a file

[@gauravpathak7017]

Harjeet-What is default no of bucket and partition??