34. Databricks - Spark: Data Skew Optimization

[@Prashanth-os5he]

This is by far the best databricks and spark tutorial series on youtube... great job Raja

[@rajasdataengineering7585]

Glad you think so! Thanks for your comment

[@sumanmondal8836]

Thanks, Raja, your explanations are really good...can you please make a video on salting techniques with example? It will be very helpful.

[@rajasdataengineering7585]

Thank you Suman. Sure, will make a video on salting

[@abhinavsingh1173]

Your course it best. But problem with you course is that you are not attching the github link for your sample data and code. Irequest you as your audience please do this. Thanks

[@srinubathina7191]

Awesome content Thank You So much Sir

[@rajasdataengineering7585]

Glad you liked it

[@skasifali4457]

Thanks Raja..Your video is really useful. Can you please create a video on debugging techniques and how we can use spark UI to debug and understand the bottleneck using use cases. Thanks a lot again

[@rajasdataengineering7585]

Sure Asif, will post a video on debugging

[@swapnilgosawi]

Do you have a document with all these details ?if yes, that would be great to share on git., Really Great explanation. Thank you !!

[@joyo2122]

You are the best Raja 🙌

[@rajunaik8803]

Hi Raja, QQ - Does AQE take care of salting and skew hint technique automatically in case of data skewness? Or do we have to explicitly apply them?

[@rajasdataengineering7585]

Yes AQE handles data skewness automatically. In later spark versions after 3.0, it is enabled by default. For prior versions of spark, we just need to enable AQE through spark config settings

[@rajunaik8803]

@@rajasdataengineering7585 thanks alot for your response. Do you have any telegram channel? And may I know your LinkedIn id please

[@iamkiri_]

Thanks for the video, I have a question.. Is salting technique applied while reading the data from source or during intermediate processing of the application..

[@rajasdataengineering7585]

It is applied during transformation stage, not at data extraction

[@iamkiri_]

Thanks Bro

[@VishalSharma-hv6ks]

You mainly focus on theoretical. It would be great if you write the code for salting as well.

[@rajasdataengineering7585]

Sure, will post another video with coding example

[@naveenkumarsingh3829]

why cant we use set maxpartitionbytes to get equal size of partitions and handle data skewness?

[@Personalcomments]

Your videos are very informative. Can you please post a video on Client mode vs Cluster mode vs local

[@rajasdataengineering7585]

Sure Merin, will post the video on this topic

[@balakrishna61]

@rajasdataengineering7585 Please explain salting in detail.It's not clear how you parition the German-1,_2 and so on .Each record will become one partition correct in this case?

[@SaurabhDestiny18]

Hi Tq for such useful videos, i have one question, i am still confused about executor boundary and cores/tasks boundary. In your first video you mentioned executor can have many cores/ram and then this video you mention executor runs in its own jvm process , which means all the cores/tasks are running under one jvm process? Or under than parent jvm process there are many more jvm process are running which are equal to number of cores/tasks?

[@sanskarsuman9340]

i have doubt: when u say data is partitioned on country and there are five different countries, out of which lets say Germany has 80% of data, so how can I say that germany data is in single partition only? coz partition is determined on the size of the block and 1 parttion = 128mb size, so depending on its size, germany data could be splitted into multiple partitions automatically?

[@ndbweurt34485]

same question i had

[@supriyakoura7755]

Same question

[@sravankumar1767]

Superb

[@rajasdataengineering7585]

Thank you

[@prathapganesh7021]

thank you

[@rajasdataengineering7585]

Welcome!

[@tanushreenagar3116]

nice

[@rajasdataengineering7585]

Thanks