Advancing Spark - Azure Databricks News October 2023

We have a HUGE month of updates to wade through for October, the folks over at Databricks have been busy pumping out a ton of goodies on the platform, and we're going to walk through what we have!
There's the huge shift of ML artifacts from their separate area into being part of Unity Catalog, there's a whole new Runtime with SQL Variables & XML wrangling, there's a ton of performance improvements for managed tables, and more!
In this video Simon walks through the release notes for October, and the new DBR 14.1 release, talking about the various features and which ones you should be excited about!
For the Databricks Release notes, see: learn.microsoft.com/en-us/azu...

[@drummerboi4eva]

Very instructive, thanks for sharing Simon !

[@robertwadsworth269]

Variables! Boom. Lifting and shifting here I come.

[@allthingsdata]

A few thoughts on the platform updates - I don't quite understand the read-only catalog binding as I can always set permissions to be read-only on any catalog object so is it just a convenience function if I want to give read-only to all objects in a catalog - table comments are only small part of data model docs, still helpful, what is actually cool is that you can programmatically feed those table comments into your company catalog (e.g. Purview or Alation) - I have recently changed my mind on managed tables; I was always on the external table side mostly for deletion and file org + access reasons; managed tables now seem to make sense for all mentioned reasons and due to features like undrop and UC REST API which external tools not coming through Databricks compute can always ask to find out where an object is stored (with all those random IDs) and thus access the object without going through Databricks compute (ofc permissions have to come from somewhere else then, e.g. storage ACLs); I would still always keep the managed tables in an external location for other reasons - I wonder how undrop table works technically; if the files are not removed immediately from the compute dropping a table, there needs to be some compute cleaning up after the retention period; if I'm not using serverless, which compute is that going to be

[@allthingsdata]

What also interesting is the pricing of predictive I/O. In sql warehouse classic photon is included but p. I/O not while it seems this distinction does not apply to regular clusters or I am misunderstanding sth.

[@Karenshow]

"You have no excuse to not document your data model " :)

[@fb-gu2er]

Deletion vectors as default table properties is not a good call