Spark Interview Question | Bucketing | Spark SQL

[@vishalaaa1]

nice

[@eknathsatish7502]

Excellent..

[@TechWithViresh]

Thanks :)

[@SpiritOfIndiaaa]

Can you please the share the notebook URL please ? thanks a lot , really gr8 learnings

[@bhushanmayank]

How does spark know that other table attribute is identical on which it is bucketed while joining?

[@gauravbhartia7543]

Nicely Explained.

[@TechWithViresh]

Thanks:)

[@aashishraina2831]

excellent

[@TechWithViresh]

Thanks :)

[@RAVIC3200]

Again great content video, Viresh can you make video on those scenarios which interviewer usually ask like - 1) if you have 1TB of file how much time it takes to process (you take any standard cluster setup configuration to explain) and if i reduce to 500GB then how much time it will take. 2) DAG related scenarios questions ? 3) If spark job failed in middle then, will it start from starting if you re-trigger it again ? if not then why? 4) checkpoint related. Please try to cover such scenarios, if its inside one video then it will be really helpful.. thanks again for such videos.....

[@TechWithViresh]

Thanks, don’t forget to subscribe.

[@RAVIC3200]

@@TechWithViresh I'm your permanent viewer 🙏🙏

[@cajaykiran]

Is there anyway I can reach out to you to discuss something important?

[@TechWithViresh]

Send the details at [e-mail].

[@dipanjansaha6824]

When we directly write to adls i.e the files then how bucketing helps? 2. Also is that a correct understanding bucketing is good when we use a datafram for read purpose only.. as what I understood if there's a use case where in every build write operation happens.. bucketing would not be the best approach..

[@TechWithViresh]

Yes, bucketing is more effective in reusable tables involved in heavier joins

[@cajaykiran]

Thank you

[@gunishjha4030]

Great content!!!, You have used bucketBy in scala code to do the changes, can you tell how to handle the same in spark sql as well. do we have any function we can pass in spark sql for the same.

[@gunishjha4030]

found it thanks anyways PARTITIONED BY (favorite_color) CLUSTERED BY(name) SORTED BY (favorite_numbers) INTO 42 BUCKETS;

[@mdfurqan]

@@gunishjha4030 but are u able to insert the data in bucketed table using spark-sql underlaying storage is Hive?

[@sachink.gorade8209]

Hello Viresh sir, Nice explaination. Just one thing I did not understand when we create 8 partitions for these two tables as I could not find any code for it in video. So could you please explain?

[@TechWithViresh]

8 is the default partitions(round robin) created for the cluster used here with 8 nodes.

[@TechWithViresh]

8 is the default number of partitions (round robin) as the cluster used has 8 nodes

[@mateen161]

Nice explanation!...Just wondering how the number of buckets should be decided. In this example, you had used 4 buckets, can't we use 6 or 8 or 10. Is there a specific reason for using 4 buckets ?

[@TechWithViresh]

It can be any number, depending on your data and bucket column

[@himanshusekharpaul476]

Hey ..Nice explanation ..But here i have one doubt ... in above vedio you have given no of bucket is 4 . What are the criteria we should keep in mind while deciding no of bucket in real time.?? Is there any formula or bucket size constraints ??? Could you please help ??

[@TechWithViresh]

The idea behind these two data distribution techniques- partition and bucket is to have data distribution evenly and in such optimum size , which can be effectively processed in a single task

[@himanshusekharpaul476]

Ok.. What is the optimum bucket size that can be processed by single task??

[@aashishraina2831]

i think this video is reapted above. can be deleted.