Advancing Spark - Understanding Low Shuffle Merge

[@WastedFury]

Really useful. New to Databricks and you're last couple of videos have really helped me understand how it will support some of the key concepts needed and some of the gotchas that are actually being resolved in the new releases. Thank you.

[@YoussefMrini]

I've been using it since the day 1. It has improved my merges :D

[@fb-gu2er]

Kudos for the whiteboard. You should do it more often

[@drummerboi4eva]

Useful feature , very well explained

[@AdvancingAnalytics]

Apologies - looks like I wiped out comments when clearing some initial spam. Apologies if anyone's actual comments got dropped! Simon

[@Monsalvo888]

really clear! thank!

[@dylanmccullough2679]

So cool!

[@ArcaLuiNeo]

Thanks for the explanation. What device are you using for the whiteboarding part?

[@AdvancingAnalytics]

Microsoft Whiteboard on a separate tablet, coming through a HDMI capture card! Probably a fairly overengineered approach!!

[@Vikasptl07]

Thanks for explanation. I am working on one such scenario where table (no efficient column for partition in table , not able to use predicate pushdown in merge )has 2 bn rows and my batch job run every 1 hour for loading(1mn rows every hour). Now merge is taking more time upwards of 50mins. I will try to implement low shuffle merge and also optimize z order by (once daily). Can you suggest any other optimization techniques?

[@tarun080311]

very time consuming explanation method.