66. Databricks | Pyspark | Delta: Z-Order Command

Azure Databricks Learning: Delta Lake - Z-Order Command
========================================================
What is Z-order Command in delta table and how to apply in delta lake development?
Z-order one of the performance optimization techinique used in delta lake. It is used along with optimize command and used to compact small files into optimal size and at the same time relevant data is co-located to improve the performance.
This video gives complete understanding of Z-order command
#DeltaZorder, #DatabricksZorder, #PerformanceOptimization, #Zorder,#Z-order, #Z-Ordering, #DeltaOptimize, #DeltaOptimizeZorder #DeltaCompactFiles, #DeltaSmallFileIssue, #DeltalakePerformance, #DeltaPerformanceImprovement ,#DeltalakeIntro, #IntroductionToDeltaLake, #Deltalake, #DeltaTable, #DatabricksDelta, #DeltaTableCreate, #DatawarehouseVsDataLakevsDeltaLake, #PysparkDeltaLake, #DeltalakevsDatalake, #SQLDeltaTable, #DataframeDeltaTable,#DeltaFormat ,#DatabricksRealtime, #SparkRealTime, #DatabricksInterviewQuestion, #DatabricksInterview, #SparkInterviewQuestion, #SparkInterview, #PysparkInterviewQuestion, #PysparkInterview, #BigdataInterviewQuestion, #BigdataInterviewQuestion, #BigDataInterview, #PysparkPerformanceTuning, #PysparkPerformanceOptimization, #PysparkPerformance, #PysparkOptimization, #PysparkTuning, #DatabricksTutorial, #AzureDatabricks, #Databricks, #Pyspark, #Spark, #AzureDatabricks, #AzureADF, #Databricks, #LearnPyspark, #LearnDataBRicks, #DataBricksTutorial, #azuredatabricks, #notebook, #Databricksforbeginners

[@ravisunkara6664]

Awesome explanation on Z-ordering. Greatly appreciated your efforts making this video.

[@rajasdataengineering7585]

Thank you

[@pratikparbhane8677]

Great Explain , Understood OPTIMISE , VACCUM() AND Z-ORDERING in One Video

[@rajasdataengineering7585]

Glad it was helpful!

[@navdeepjha2739]

Invaluable explanation sir! I went through many blogs but couldn't get it. You made it crystal clear😊

[@rajasdataengineering7585]

Glad to hear that! Thanks for your comment

[@shreeyashransubhe2537]

Sir, I have gone through lots of videos but never understood the concepts so simple yet very detailed manner. Thank you very much. I have shared your playlist with my colleagues too. They also liked it very much.

[@rajasdataengineering7585]

Thank you for your valuable comments. Really appreciated

[@rohitdanda]

Your videos are so simple that a kid can also understand. Thanks and salute sir🖖 for putting so much effort and making videos and helping us!

[@rajasdataengineering7585]

Thanks for your comment. Glad to know it helps data engineers

[@user-wy5tl2ev2r]

one such good video with neat explanation.

[@rajasdataengineering7585]

Thank you

[@YogeshBiguvu2208]

Excellent explanation with Examples.....Thank you so mcuh sir..

[@rajasdataengineering7585]

You are most welcome! Glad it helps

[@RanjeetkumarYadav]

Amazing and very intuitive example. Thank You!!

[@rajasdataengineering7585]

You're very welcome! Keep watching

[@mukilanlakshmanan8968]

Sir, I love your teaching method, you have explained it in detail.

[@rajasdataengineering7585]

Thanks Mukilan! Glad to hear that

[@user-hj2nv8gt4o]

Sir, Thanks for explaining in a very simple manner.

[@rajasdataengineering7585]

Thanks and welcome

[@ajaykiranchundi9979]

A very well explained . The way you broke down the data to explain the same is amazing. I am sure it would have taken good time to put it together. Indebted to you brother.

[@rajasdataengineering7585]

Thanks Ajay👍🏻

[@SumitAmbatkar]

i watched your nearly all playlist i loved your teching style, how to ogrip on concept, your explaination are fabulous keeping doing sir, best of luck. we are always here for you Thank you..:)

[@rajasdataengineering7585]

Thank you,Sumit! Keep watching

[@saurav0777]

Thanks for uploading . Very nice explanation

[@rajasdataengineering7585]

Thanks

[@sraoarjun]

Indeed an awesome video !! Great explanation !!

[@rajasdataengineering7585]

Glad you liked it! Thank you

[@terrificmenace]

Thank you sir 🙏🏻 I went through many udemy courses but never understood these concepts. Ur explanation is very good and easy to understand many many thanks sir 🙏🏻 🙏🏻

[@rajasdataengineering7585]

Thank you 👍🏻

[@vivek05117gece]

very well explained. Kudos to you.

[@rajasdataengineering7585]

Glad it was helpful!

[@shwetac2929]

you teaching methos is very good ....this video clear my all doubt

[@rajasdataengineering7585]

Glad to hear that

[@FreakONcW1]

Extremely helpful video.

[@rajasdataengineering7585]

Thanks Kinjal! Glad to know it was helpful!

[@3a8saisamireddi61]

detailed explanation👏

[@rajasdataengineering7585]

Thank you 🙂

[@dineshwaditake5248]

Nicely explained !!

[@rajasdataengineering7585]

Glad it was helpful!

[@omprakashreddy4230]

Your videos are definitely creating great impact. Thank you for that. Can you also please explain df.explain() command in great detail with examples.

[@rajasdataengineering7585]

Happy to hear that it's creating impact on data engineers. Thank you Sure, will post a video on explain plan

[@rajasdataengineering7585]

Hi Omprakash, created a video on explain plan as per your request. Hope it helps you - kzbin.info/www/bejne/bH_Vh4SKl9Chmrc

[@ravulapallivenkatagurnadha9605]

Please continue this videos

[@tanushreenagar3116]

Very nice sir 👌 cleared my concept now

[@rajasdataengineering7585]

Thank you

[@mohitupadhayay1439]

Raja please try to create a full project where all these optimizations can be shown at full scale.

[@rajasdataengineering7585]

Sure Mohit, will do!

[@viniciusguimaraessantana5455]

thank you very much.

[@rajasdataengineering7585]

You are welcome!

[@gil.0007]

Very nicely explained 🎉

[@rajasdataengineering7585]

Thanks, glad it was helpful!

[@AFSARAHMED4]

Excellent Explaination Sir

[@rajasdataengineering7585]

Thanks

[@manjit_singhh]

Very nice explanation 🙂

[@rajasdataengineering7585]

Thanks

[@tanushreenagar3116]

PERFECT CONTENT SIR

[@rajasdataengineering7585]

Thanks Tanu!

[@sathyahisto]

good Explaination, liked it when you demonstrated with excel. Just one suggestion syntax for zorder seems to be changed to "Zorder by ()"

[@rajasdataengineering7585]

Yes, you are right. Thanks

[@TotuBabyBird]

Great!

[@rajasdataengineering7585]

Thanks

[@satheeshkumarak6708]

Hello Sir, How do you determine the number of columns to be used in Z order and whether or not to use a particular column for Z order provided that you have calculated the cardinality percentage of all the columns?

[@venkatasai4293]

Thanks for the great explanation Raja. So are the statistics collected on all the columns ? What if we want to query on other columns ? Will it work ?

[@rajasdataengineering7585]

Yes Venkata, it will work first 32 columns. If your table contains more than 32 columns and you want to collect statistics for those columns, we can configure that separately

[@venkatasai4293]

@@rajasdataengineering7585 ok . So zorder is similar to bucketing right ? Colocating the data into same set of files ? If two tables contains same key and if we zorder them on the key While joining the data it will fetch only required files into the executor ?

[@shankar1556]

Hi Azar, Thank you for explanation. I have a dought. in this example it shows that z-order create new partitions with sorting emp_id. Does z-order really create new partitions?

[@rajasdataengineering7585]

Hi Shankar, this is Raja. When we perform z-order, data is being co-located within same set of files. It is not shuffling the data, nor creating new partitions

[@TheDataArchitect]

What about using multiple columns in z-order?

[@purnimasharma9734]

Hi Raja, how is the partition column determined e.g. how does it know that you have to use emp_id here? Is it based on the predicate column?

[@purnimasharma9734]

Never mind, when I watched your video completely, I found out.

[@rajasdataengineering7585]

Great

[@aswaniyettapu9992]

Can u do one video on lead and lag in pyspark..?

[@rajasdataengineering7585]

Sure, will post a video on lead and lag very soon

[@rajasdataengineering7585]

Hi Aswani, have posted a video on lead and lag function today as per your request

[@aswaniyettapu9992]

Tq so much

[@ravulapallivenkatagurnadha9605]

Please do video on how to convert pandas data pipilines to spark data pipiy

[@rajasdataengineering7585]

Sure will do