66. Databricks | Pyspark | Delta: Z-Order Command

Azure Databricks Learning: Delta Lake - Z-Order Command
========================================================
What is Z-order Command in delta table and how to apply in delta lake development?
Z-order one of the performance optimization techinique used in delta lake. It is used along with optimize command and used to compact small files into optimal size and at the same time relevant data is co-located to improve the performance.
This video gives complete understanding of Z-order command
#DeltaZorder, #DatabricksZorder, #PerformanceOptimization, #Zorder,#Z-order, #Z-Ordering, #DeltaOptimize, #DeltaOptimizeZorder #DeltaCompactFiles, #DeltaSmallFileIssue, #DeltalakePerformance, #DeltaPerformanceImprovement ,#DeltalakeIntro, #IntroductionToDeltaLake, #Deltalake, #DeltaTable, #DatabricksDelta, #DeltaTableCreate, #DatawarehouseVsDataLakevsDeltaLake, #PysparkDeltaLake, #DeltalakevsDatalake, #SQLDeltaTable, #DataframeDeltaTable,#DeltaFormat ,#DatabricksRealtime, #SparkRealTime, #DatabricksInterviewQuestion, #DatabricksInterview, #SparkInterviewQuestion, #SparkInterview, #PysparkInterviewQuestion, #PysparkInterview, #BigdataInterviewQuestion, #BigdataInterviewQuestion, #BigDataInterview, #PysparkPerformanceTuning, #PysparkPerformanceOptimization, #PysparkPerformance, #PysparkOptimization, #PysparkTuning, #DatabricksTutorial, #AzureDatabricks, #Databricks, #Pyspark, #Spark, #AzureDatabricks, #AzureADF, #Databricks, #LearnPyspark, #LearnDataBRicks, #DataBricksTutorial, #azuredatabricks, #notebook, #Databricksforbeginners

[@shreeyashransubhe2537]

Sir, I have gone through lots of videos but never understood the concepts so simple yet very detailed manner. Thank you very much. I have shared your playlist with my colleagues too. They also liked it very much.

[@rajasdataengineering7585]

Thank you for your valuable comments. Really appreciated

[@sundarkannan1856]

Thank you and very clear and helps to understand what is Zorder method in delta table

[@rajasdataengineering7585]

Glad it was helpful! Keep watching

[@pratikparbhane8677]

Great Explain , Understood OPTIMISE , VACCUM() AND Z-ORDERING in One Video

[@rajasdataengineering7585]

Glad it was helpful!

[@rohitdanda]

Your videos are so simple that a kid can also understand. Thanks and salute sir🖖 for putting so much effort and making videos and helping us!

[@rajasdataengineering7585]

Thanks for your comment. Glad to know it helps data engineers

[@navdeepjha2739]

Invaluable explanation sir! I went through many blogs but couldn't get it. You made it crystal clear😊

[@rajasdataengineering7585]

Glad to hear that! Thanks for your comment

[@AnupGupta-05]

Hi Brother you are best teacher, the way you explain its best, keep up the good work

[@rajasdataengineering7585]

Thank you!

[@annaduraip3182]

Great, thank you. You have explained in simpler way to understand anyone.

[@rajasdataengineering7585]

Thank you

[@ravisunkara6664]

Awesome explanation on Z-ordering. Greatly appreciated your efforts making this video.

[@rajasdataengineering7585]

Thank you

[@mukilanlakshmanan8968]

Sir, I love your teaching method, you have explained it in detail.

[@rajasdataengineering7585]

Thanks Mukilan! Glad to hear that

[@NileshPatil-b3u]

Sir, Thanks for explaining in a very simple manner.

[@rajasdataengineering7585]

Thanks and welcome

[@SumitAmbatkar]

i watched your nearly all playlist i loved your teching style, how to ogrip on concept, your explaination are fabulous keeping doing sir, best of luck. we are always here for you Thank you..:)

[@rajasdataengineering7585]

Thank you,Sumit! Keep watching

[@ajaykiranchundi9979]

A very well explained . The way you broke down the data to explain the same is amazing. I am sure it would have taken good time to put it together. Indebted to you brother.

[@rajasdataengineering7585]

Thanks Ajay👍🏻

[@terrificmenace]

Thank you sir 🙏🏻 I went through many udemy courses but never understood these concepts. Ur explanation is very good and easy to understand many many thanks sir 🙏🏻 🙏🏻

[@rajasdataengineering7585]

Thank you 👍🏻

[@shankar1556]

Hi Azar, Thank you for explanation. I have a dought. in this example it shows that z-order create new partitions with sorting emp_id. Does z-order really create new partitions?

[@rajasdataengineering7585]

Hi Shankar, this is Raja. When we perform z-order, data is being co-located within same set of files. It is not shuffling the data, nor creating new partitions

[@YogeshBiguvu2208]

Excellent explanation with Examples.....Thank you so mcuh sir..

[@rajasdataengineering7585]

You are most welcome! Glad it helps

[@satheeshkumarak6708]

Hello Sir, How do you determine the number of columns to be used in Z order and whether or not to use a particular column for Z order provided that you have calculated the cardinality percentage of all the columns?

[@RanjeetkumarYadav]

Amazing and very intuitive example. Thank You!!

[@rajasdataengineering7585]

You're very welcome! Keep watching

[@shwetac2929]

you teaching methos is very good ....this video clear my all doubt

[@rajasdataengineering7585]

Glad to hear that

[@venkatasai4293]

Thanks for the great explanation Raja. So are the statistics collected on all the columns ? What if we want to query on other columns ? Will it work ?

[@rajasdataengineering7585]

Yes Venkata, it will work first 32 columns. If your table contains more than 32 columns and you want to collect statistics for those columns, we can configure that separately

[@venkatasai4293]

@@rajasdataengineering7585 ok . So zorder is similar to bucketing right ? Colocating the data into same set of files ? If two tables contains same key and if we zorder them on the key While joining the data it will fetch only required files into the executor ?

[@purnimasharma9734]

Hi Raja, how is the partition column determined e.g. how does it know that you have to use emp_id here? Is it based on the predicate column?

[@purnimasharma9734]

Never mind, when I watched your video completely, I found out.

[@rajasdataengineering7585]

Great

[@omprakashreddy4230]

Your videos are definitely creating great impact. Thank you for that. Can you also please explain df.explain() command in great detail with examples.

[@rajasdataengineering7585]

Happy to hear that it's creating impact on data engineers. Thank you Sure, will post a video on explain plan

[@rajasdataengineering7585]

Hi Omprakash, created a video on explain plan as per your request. Hope it helps you - kzbin.info/www/bejne/bH_Vh4SKl9Chmrc

[@arabajshaikh8411]

Excellent, Thank you so much.

[@rajasdataengineering7585]

Glad it was helpful! You are welcome

[@sathyahisto]

good Explaination, liked it when you demonstrated with excel. Just one suggestion syntax for zorder seems to be changed to "Zorder by ()"

[@rajasdataengineering7585]

Yes, you are right. Thanks

[@sraoarjun]

Indeed an awesome video !! Great explanation !!

[@rajasdataengineering7585]

Glad you liked it! Thank you

[@tanushreenagar3116]

Very nice sir 👌 cleared my concept now

[@rajasdataengineering7585]

Thank you

[@saurav0777]

Thanks for uploading . Very nice explanation

[@rajasdataengineering7585]

Thanks

[@TheDataArchitect]

What about using multiple columns in z-order?

[@AFSARAHMED4]

Excellent Explaination Sir

[@rajasdataengineering7585]

Thanks

[@rahul_chilukamari]

one such good video with neat explanation.

[@rajasdataengineering7585]

Thank you

[@vivek05117gece]

very well explained. Kudos to you.

[@rajasdataengineering7585]

Glad it was helpful!

[@mohitupadhayay1439]

Raja please try to create a full project where all these optimizations can be shown at full scale.

[@rajasdataengineering7585]

Sure Mohit, will do!

[@FreakONcW1]

Extremely helpful video.

[@rajasdataengineering7585]

Thanks Kinjal! Glad to know it was helpful!

[@dineshwaditake5248]

Nicely explained !!

[@rajasdataengineering7585]

Glad it was helpful!

[@gil.0007]

Very nicely explained 🎉

[@rajasdataengineering7585]

Thanks, glad it was helpful!

[@ravulapallivenkatagurnadha9605]

Please continue this videos

[@tanushreenagar3116]

PERFECT CONTENT SIR

[@rajasdataengineering7585]

Thanks Tanu!

[@3a8saisamireddi61]

detailed explanation👏

[@rajasdataengineering7585]

Thank you 🙂

[@manjit_singhh]

Very nice explanation 🙂

[@rajasdataengineering7585]

Thanks

[@viniciusguimaraessantana5455]

thank you very much.

[@rajasdataengineering7585]

You are welcome!

[@anuragpaudyal3297]

awesome😃

[@rajasdataengineering7585]

Thank you! Cheers!

[@TotuBabyBird]

Great!

[@rajasdataengineering7585]

Thanks

[@aswaniyettapu9992]

Can u do one video on lead and lag in pyspark..?

[@rajasdataengineering7585]

Sure, will post a video on lead and lag very soon

[@rajasdataengineering7585]

Hi Aswani, have posted a video on lead and lag function today as per your request

[@aswaniyettapu9992]

Tq so much

[@ravulapallivenkatagurnadha9605]

Please do video on how to convert pandas data pipilines to spark data pipiy

[@rajasdataengineering7585]

Sure will do