Liquid Clustering in Databricks,What It is and How to Use,

Рет қаралды 7,545

9 ай бұрын

Liquid Clustering in Databricks,What It is and How to Use, #liquidclustering #clusterby #databricksLiquid
Clustering in Databricks,The Ultimate Guide,
Databricks Liquid Clustering,What It Is and How to Use It,
Improve Your Databricks Performance with Liquid Clustering,
How to Use Liquid Clustering to Optimize Your Databricks Workloads,
Databricks Liquid Clustering,A Step-by-Step Tutorial,
Databricks Liquid Clustering,The Future of Data Warehousing,
Databricks Liquid Clustering,Everything You Need to Know,
Databricks Liquid Clustering,A Deep Dive,
Databricks Liquid Clustering,Best Practices,
Databricks Liquid Clustering,Performance Benchmarking,
Databricks Liquid Clustering,Case Studies,
Databricks Liquid Clustering, New Features and Enhancements,
Databricks Liquid Clustering, The Future of Data Lakes,
Databricks Liquid Clustering,The Future of Data Engineering,

Пікірлер: 22

@TRRaveendra 9 ай бұрын

You can find the notebook in below github location : github.com/raveendratal/PysparkRaveendra/blob/master/Liquid%20Clustering.ipynb

@ajaykiranchundi9979 6 ай бұрын

Thanks Ravi! Great explanation

@TRRaveendra 6 ай бұрын

Thank you 🙏

@2007mnkumar 9 ай бұрын

What a great explanation. Ravi, Day by day the value of your presentations goes higher and higher. It would be greate, If you can share Notebook also.

@TRRaveendra 9 ай бұрын

github.com/raveendratal/PysparkRaveendra/blob/master/Liquid%20Clustering.ipynb

@jeetash1 9 ай бұрын

First table created using partitionBy on origin and filtering on dayofWeek = 1 and in second table you clustered by "dayofWeek" and filter on dayofWeek = 1 then it will obliviously take more time in case of partition table. I agree it will create files based on total number partitions and it would skip more files to read if table created using partitionBy dayofWeek and add filter on same column.

@TRRaveendra 9 ай бұрын

Partition by is not good for small tables The old approach was partition and Optimize with Zorder By . Instead of partition By We can use cluster By Then we can apply optimize. No need to use partition By and Zorder By for less than 1TB tables.

@dipalisabale6302 8 ай бұрын

Cluster by is alternate to partition by and z ordering and recommended table size to implement partition &z orderis 1TB . So does this conclude that we should not apply liquid clustering for table less than 1TB size ?

@oussemakeskes6275 18 күн бұрын

totally agree with @jeetash1. if you want to correctly compare and benchmark partitionBy and clusteredby you should use same column otherwise that comparison doesn't make sense. if you created using partitionBy on dayofWeek and filtering on dayofWeek = 1 and in second table you clustered by "origin " and filter on dayofWeek = 1 partitionby will take less time

@saimanideepallu5743 9 ай бұрын

I want to have personalized training from you. Could you please let me know about it please ?

@januaymagori4642 8 ай бұрын

On partition by why not using coalesce during writing so you can have few files

@udaybalerao4816 9 ай бұрын

thank you Sir! One question - will liquid clustering be same as Z order for NON Partitioned table?

@PrashantSamant-wp5yl 6 ай бұрын

On implementing liquid clustering, when I call desc detail table table name, I see clustering columns..but when I insert data to liquid clustering table using dataframe.write ND then execute same desc detail table, clustering columns are lost.i ran optimize but no use.i have datBricks runtime 13.2

@rajeshr4145 8 ай бұрын

Hi Ravi, This video was of great use. I have one question. Is it possible to convert an existing table with partitioned having data to liquid cluster? If so can you please suggest the steps?

@TRRaveendra 8 ай бұрын

as of now you can use only SQL Table DDL for liquid clustering like while creating a table using SQL CREATE TABLE Table_name(col...) cluster by (col1,col2.) after that you can alter a table for changing cluster by columns. using alter table ....

@ajaykiranchundi9979 6 ай бұрын

Hello Rajesh, Did you find an answer ? Did you try directly applying the clustering on the existing table ? was about to try it on one of the tables at my end.