AQE in spark | Lec-19

[@piyushjain5852]

Manish bhai, mai AQE pdh k gaya is video se, aaj interviewer ne puchha skew data ko kse optimize karoge, mene coleasce or split bataya pr usne puchha ki split kese hoga internally vo mujhe nahi samajh aya fir usne bataya SALTING krni hogi, bs vahi ek topic mene nhi padha tha qki mujhe laga important nhi hoga aapne b video me skip kr dia, pr vhi puch lia usne. Baki sb answer kr die mene thanks to your videos, but he rejected me, anyway I'm feeling more coinfident for upcoming interviews. Keep up the good work. You're helping guys like me a lot.

[@Matrix_Mayhem]

Thanks for sharing Piyush. Can we connect on LinkedIN? I am preparing too

[@deepjyotimitra1340]

Har ek video apka master class hain. ❤

[@vipul7010]

Best Explanation I found anywhere in youtube, Great Work!

[@Fr4529]

Great .Very precise and detailed explanation on AQE. Do you also have the Github or the dbc file of the example.

[@greendaywithtrading7408]

GREAT EXPLANATION KINDLY PROVIDE DATA FOR PRACTICE

[@RohitSingh-we9mo]

@Manish Kumar -Aapne bataya tha ki coalesce me partition size reduce hota hai but yha pe add kyu kr rhe dynamically coalescing me.Iska explain kr denge

[@shadabalam17]

I have a small confusion, when the final plan is updated, and AQE chooses Broadcast join instead of Sort merge join as per DAG, then, we are avoiding shuffling infact, you told in the end of Dynamic switching of join strategy, that we cannot avoid shuffling, But we know, Broadcast join avoid shuffling, Can anyone clarify me pls

[@ajaypatil1881]

Best Explanation bhaiya

[@dharmendersingh7165]

Tell me one thing Manish and you need to be clear this time Assuming you have 2 executors and 2 partitions on each executor after shuffle So keys will lies in different partitions in same executor? Or Keys like in same partition in a executor?

[@mission_possible]

Thanks for the videos Manish❤

[@stevedz5591]

Sir, plz make video on Orchestration & HDFS.

[@Aryan-lu5js]

Azure Synapse was launched in Nov 2019 but Indian Recruiters wants someone who has 5+ years of experience in Azure Synapse 😢

[@manish_kumar_1]

😂😂

[@piyushjain5852]

Lol

[@isharkpraveen]

Recruiter pagla gaya he..😂

[@SouvindSajeev]

🤣🤣🤣

[@SouAzu-b8e]

Thank you, please explain the salting and how to implement it in next video.

[@manish_kumar_1]

Will do soon

[@prashantmehta2832]

Hello sir, at 11.40, is there any rule for coalesce when we do AQE?

[@BishalKarki-pe8hs]

same question here

[@AyushMandloi]

Why AQE is not always enabled?

[@Amarjeet-fb3lk]

At 27.54 Apne note me dala he, Skew join hone k baad, 1 partition me 8 gb data bhi aa sakta he,for example. But, 1 partition he,wo ek core pe store hogi,uspe process hogi. To, Agar 8 gb ka ek partition ban gaya, join k baad, But, core k pas memory hi ,4 gb ki hai, To wo kaise process karega?

[@manish_kumar_1]

Iteration me process hoga.

[@aryankhandelwal8517]

Why does it create duplicate partitions for the join? It simply takes Table 2 partitions and joins them with both Table 1 partitions.

[@mission_possible]

Please make video on how to read physical plan and DAG

[@ajaypatil1881]

Top Class Video

[@tanushreenagar3116]

Best content 👌

[@gauravkothe9558]

if we do split then how will join is perform on that split data please explain

[@ansarmobile9758]

Sir, how many more lectures are pending, pls?

[@mohammedasif4349]

Ye AQE internally apne aap lag jata hai ya humein code me koi config set karna hota hai??

[@manish_kumar_1]

Apne aap hoga if you are using spark 3.x else you will have to add the config

[@madanmohan6487]

Hi Manish, can you make detailed video on fact and dimension tables.

[@manish_kumar_1]

Sure

[@kartikgupta2299]

Shuffling ku hui jab AQE ne sort merge ko change krke broadcast join m badal dia taaki shuffle na ho

[@shadabalam17]

Looking for the reply from Manish on this, I got confused with this line, that we cannot avoid shuffling !

[@sanooosai]

great sir thank you

[@SubhamKumar-or8vc]

Why would the shuffling still happen in case we've got 8gb and 10mb tables after filtering? The shuffling you talked about, is it before the tables are filtered or after the tables are filtered?

[@Matrix_Mayhem]

Before

[@kartikgupta2299]

@@Matrix_Mayhemare lekin pehele join hua kaha jo shuffling hui , pehele transformation hua aur uske baad to size kam hogya to na pehele hui shuffling aur baad m to hogi hi nhi kuki broadcast join hogya switch , vrna AQE ka fayda hi kya jab chidiya agar pehele hi khet chug jayegi to

[@shadabalam17]

@@kartikgupta2299 yes, Evn I feel you are correct, I was going through this video, and got confused, that shuffling will still heppen.

[@tahersadri4220]

Hey manish can you provide autoloader and copyinto function of databricks using configs and incremental load

[@thedatamaster]

Databricks Auto Loader kzbin.info/aero/PL7S7dD8r4QdWGQ1XT8FEM0ebf7I1RRAy2

[@sauravbhattacharjee6297]

Can you create a course of AWS for data engineers?

[@manish_kumar_1]

No. Because I am not good at any cloud technology

[@jai3863]

Have you created the discord group ? Link please if you have

[@manish_kumar_1]

discord.gg/6r9QF6TE3F

[@KartikKumar-xr8gu]

bhai aap yeh bta skte ho ki or kitna time lgega spark pura hona mai ?

[@manish_kumar_1]

September tak ho jayega

[@arpittrivedi6636]

Sir aap data science sikhte ho kya from scratch? If yes then please share video link

[@manish_kumar_1]

No main data science nhi sikhata hu

[@ritikpatil4077]

When i was going through your Job, Stage and Task video, i was only getting 1 Task even when i used groupBy and repartition. Now i know why :)

[@akhiladevangamath1277]

Bcz of AQE?

[@ritikpatil4077]

@@akhiladevangamath1277 yes

[@saikumarjakki3802]

Hi guys, does any one have idea about how to solve java heap space in spark cluster?

[@unpluggedsaurav3186]

Salting: To mitigate data skew, you can apply a salting technique. Salting involves adding a random or semi-random prefix to each key or value. This prefix is generated in a way that ensures that the same original key or value always gets the same prefix. # Sample data with skewed keys data = [("A", 1), ("B", 2), ("A", 3), ("C", 4), ("A", 5)] # Apply salting by adding a random prefix to each key salted_data = [(str(hash(key) % 1000) + "_" + key, value) for key, value in data] # Now, data is more evenly distributed, and you can perform operations like groupBy result = spark.sparkContext.parallelize(salted_data) \ .groupByKey() \ .mapValues(lambda values: sum(values)) result.collect()

[@krupas8277]

no practical videos

[@agarwalankita504]

your videos are very useful but your speaking content is more which makes the video large and boring

[@sankuM]

24:24, I think it's Projection Pruning @manish_kumar_1 🤓🤓