AQE in spark | Lec-19

[@piyushjain5852]

Manish bhai, mai AQE pdh k gaya is video se, aaj interviewer ne puchha skew data ko kse optimize karoge, mene coleasce or split bataya pr usne puchha ki split kese hoga internally vo mujhe nahi samajh aya fir usne bataya SALTING krni hogi, bs vahi ek topic mene nhi padha tha qki mujhe laga important nhi hoga aapne b video me skip kr dia, pr vhi puch lia usne. Baki sb answer kr die mene thanks to your videos, but he rejected me, anyway I'm feeling more coinfident for upcoming interviews. Keep up the good work. You're helping guys like me a lot.

[@Matrix_Mayhem]

Thanks for sharing Piyush. Can we connect on LinkedIN? I am preparing too

[@vipul7010]

Best Explanation I found anywhere in youtube, Great Work!

[@deepjyotimitra1340]

Har ek video apka master class hain. ❤

[@greendaywithtrading7408]

GREAT EXPLANATION KINDLY PROVIDE DATA FOR PRACTICE

[@Fr4529]

Great .Very precise and detailed explanation on AQE. Do you also have the Github or the dbc file of the example.

[@Aryan-lu5js]

Azure Synapse was launched in Nov 2019 but Indian Recruiters wants someone who has 5+ years of experience in Azure Synapse 😢

[@manish_kumar_1]

😂😂

[@piyushjain5852]

Lol

[@isharkpraveen]

Recruiter pagla gaya he..😂

[@SouvindSajeev]

🤣🤣🤣

[@prashantmehta2832]

Hello sir, at 11.40, is there any rule for coalesce when we do AQE?

[@BishalKarki-pe8hs]

same question here

[@ajaypatil1881]

Best Explanation bhaiya

[@mission_possible]

Thanks for the videos Manish❤

[@RohitSingh-we9mo]

@Manish Kumar -Aapne bataya tha ki coalesce me partition size reduce hota hai but yha pe add kyu kr rhe dynamically coalescing me.Iska explain kr denge

[@aryankhandelwal8517]

Why does it create duplicate partitions for the join? It simply takes Table 2 partitions and joins them with both Table 1 partitions.

[@gauravkothe9558]

if we do split then how will join is perform on that split data please explain

[@isharkpraveen]

Sir ji,, bohoth time se video nai aara...please upload videos

[@stevedz5591]

Sir, plz make video on Orchestration & HDFS.

[@AyushMandloi]

Why AQE is not always enabled?

[@shadabalam17]

I have a small confusion, when the final plan is updated, and AQE chooses Broadcast join instead of Sort merge join as per DAG, then, we are avoiding shuffling infact, you told in the end of Dynamic switching of join strategy, that we cannot avoid shuffling, But we know, Broadcast join avoid shuffling, Can anyone clarify me pls

[@siddharthdas9952]

"AQE converts sort-merge join to broadcast hash join when the runtime statistics of any join side is smaller than the adaptive broadcast hash join threshold. This is not as efficient as planning a broadcast hash join in the first place, but it’s better than keep doing the sort-merge join, as we can save the sorting of both the join sides, and read shuffle files locally to save network traffic". From spark aqe documentation one side shuffling will be done other side can be avoided if the first one is below broadcast limit

[@SubhamKumar-or8vc]

Why would the shuffling still happen in case we've got 8gb and 10mb tables after filtering? The shuffling you talked about, is it before the tables are filtered or after the tables are filtered?

[@Matrix_Mayhem]

Before

[@kartikgupta2299]

@@Matrix_Mayhemare lekin pehele join hua kaha jo shuffling hui , pehele transformation hua aur uske baad to size kam hogya to na pehele hui shuffling aur baad m to hogi hi nhi kuki broadcast join hogya switch , vrna AQE ka fayda hi kya jab chidiya agar pehele hi khet chug jayegi to

[@shadabalam17]

@@kartikgupta2299 yes, Evn I feel you are correct, I was going through this video, and got confused, that shuffling will still heppen.

[@jai3863]

Have you created the discord group ? Link please if you have

[@manish_kumar_1]

discord.gg/6r9QF6TE3F

[@ansarmobile9758]

Sir, how many more lectures are pending, pls?

[@SouAzu-b8e]

Thank you, please explain the salting and how to implement it in next video.

[@manish_kumar_1]

Will do soon

[@Amarjeet-fb3lk]

At 27.54 Apne note me dala he, Skew join hone k baad, 1 partition me 8 gb data bhi aa sakta he,for example. But, 1 partition he,wo ek core pe store hogi,uspe process hogi. To, Agar 8 gb ka ek partition ban gaya, join k baad, But, core k pas memory hi ,4 gb ki hai, To wo kaise process karega?

[@manish_kumar_1]

Iteration me process hoga.

[@mission_possible]

Please make video on how to read physical plan and DAG

[@rohitsharma-mg7hd]

sir ye skewed fact table ka data de dijiye practice karne k lie

[@ritikpatil4077]

When i was going through your Job, Stage and Task video, i was only getting 1 Task even when i used groupBy and repartition. Now i know why :)

[@akhiladevangamath1277]

Bcz of AQE?

[@ritikpatil4077]

@@akhiladevangamath1277 yes

[@ajaypatil1881]

Top Class Video

[@tanushreenagar3116]

Best content 👌

[@kartikgupta2299]

Shuffling ku hui jab AQE ne sort merge ko change krke broadcast join m badal dia taaki shuffle na ho

[@shadabalam17]

Looking for the reply from Manish on this, I got confused with this line, that we cannot avoid shuffling !

[@tahersadri4220]

Hey manish can you provide autoloader and copyinto function of databricks using configs and incremental load

[@thedatamaster]

Databricks Auto Loader kzbin.info/aero/PL7S7dD8r4QdWGQ1XT8FEM0ebf7I1RRAy2

[@mohammedasif4349]

Ye AQE internally apne aap lag jata hai ya humein code me koi config set karna hota hai??

[@manish_kumar_1]

Apne aap hoga if you are using spark 3.x else you will have to add the config

[@madanmohan6487]

Hi Manish, can you make detailed video on fact and dimension tables.

[@manish_kumar_1]

Sure

[@sauravbhattacharjee6297]

Can you create a course of AWS for data engineers?

[@manish_kumar_1]

No. Because I am not good at any cloud technology

[@sanooosai]

great sir thank you

[@dharmendersingh7165]

Tell me one thing Manish and you need to be clear this time Assuming you have 2 executors and 2 partitions on each executor after shuffle So keys will lies in different partitions in same executor? Or Keys like in same partition in a executor?

[@younevano]

Sub-partitions formed for skewed partition after AQE skew handling might reside on different executors or physical partitions. They can be distributed across the cluster. But even when data for a single key (skewed partition) is physically split into sub-partitions (for AQE skew optimization), it is still logically grouped. Spark uses the logical partition ID and maintains metadata to track which sub-partitions belong to the same key. Spark ensures that all sub-partitions for the same key are processed as a unit during aggregation or joining. Spark ensures correctness by logically merging results post-processing based on the logical partition key.

[@saikumarjakki3802]

Hi guys, does any one have idea about how to solve java heap space in spark cluster?

[@KartikKumar-xr8gu]

bhai aap yeh bta skte ho ki or kitna time lgega spark pura hona mai ?

[@manish_kumar_1]

September tak ho jayega

[@arpittrivedi6636]

Sir aap data science sikhte ho kya from scratch? If yes then please share video link

[@manish_kumar_1]

No main data science nhi sikhata hu

[@unpluggedsaurav3186]

Salting: To mitigate data skew, you can apply a salting technique. Salting involves adding a random or semi-random prefix to each key or value. This prefix is generated in a way that ensures that the same original key or value always gets the same prefix. # Sample data with skewed keys data = [("A", 1), ("B", 2), ("A", 3), ("C", 4), ("A", 5)] # Apply salting by adding a random prefix to each key salted_data = [(str(hash(key) % 1000) + "_" + key, value) for key, value in data] # Now, data is more evenly distributed, and you can perform operations like groupBy result = spark.sparkContext.parallelize(salted_data) \ .groupByKey() \ .mapValues(lambda values: sum(values)) result.collect()

[@krupas8277]

no practical videos

[@agarwalankita504]

your videos are very useful but your speaking content is more which makes the video large and boring

[@sankuM]

24:24, I think it's Projection Pruning @manish_kumar_1 🤓🤓