78. Databricks | Pyspark | Performance Optimization: Delta Cache

[@phanisrikrishna]

Hi Raja, I have a question. 1) In this context what is the remote and local server? is a remote server a cloud? and local server on-prem? got confused here, can you please elaborate. 2) As the table is cached in the local server, How is local memory flushed when the job is completed and when that table is no more needed.

[@saysayeed]

Remote is cloud disk storage. Local is the Databricks disk storage.

[@rajunaik8803]

Thanks Raja, just QQ's, does this delta cache charge extra? I believe this feature is mostly useful for data analysts. Am I right? Is there any time limit for this delta cache before purge?

[@vishalaaa1]

awesome video !

[@rajasdataengineering7585]

Thanks!

[@sravanthiyethapu9970]

Hi Raja, Can you make a video on python loads vs load and dumps vs dump

[@rajasdataengineering7585]

Sure Sravanthi, will make a video on this requirement

[@suresh.suthar.24]

well explained

[@rajasdataengineering7585]

Thanks Suresh! Glad you liked it

[@natedubb88]

In the Databricks documentation for Disk Cache (both AWS and Azure) the section on configuration states the following properties should be set in the Spark config: spark.databricks.io.cache.maxDiskUsage 50g spark.databricks.io.cache.maxMetaDataCache 1g spark.databricks.io.cache.compression.enabled false Since the remote data is read/stored onto the worker SSDs, and those disk size can be set in the cluster config page, I am wondering if the 50g and 1g values for the top to parameters are still appropriate. I've tried looking through the docs but was unsuccessful finding an answer. What are you thoughts? Hope you are well!