Advancing Spark - Reflecting on a Year of Unity Catalog

It's been a hugely busy year for the industry, and one of the real signs of maturity is a focus on data management principles: Governance, Security, Observability and Lineage! Looking back over the year, it's awesome to see how far Unity Catalog has come in that time - from a fairly narrow specialist tool to a truly end-to-end view of the data platform!
In this video Simon reflects on the various Unity Catalog features that we've seen coming through recently and how they come together to complete the picture of our data estate. If you've not been keeping up with Unity Cat recently, it's well worth a look!
If you've not implemented Unity Catalog yet, and need a boost getting things implemented properly, Advancing Analytics can help you on that journey in 2024!

[@AlessandroGattolin]

It has been a very exciting year about Unity Catalog, but also it has been a year full of very insightful contents from Advancing Analytics, so thank you so much!! Can’t wait to see what’s coming in 2024, which for me will be the year of the real Lakehouse :)

[@allthingsdata]

We currently migrate to UC with a few projects and had sessions with Databricks about it. One thing we asked was when to use lakehouse federation (LF) as we are also excited about it. We use Databricks as an integration tool getting data from many places incl databases and these can be large data so we leverage advanced jdbc driver settings like parallel read/write and custom batch sizes. Basically, what Databricks said was that for these workloads, it may still be better to not use LF as it is less flexible in the connection options. I have also not checked if LF would support SSO via Entra ID.

[@drummerboi4eva]

Nice recap on Unity's features, Simon !

[@saugatmukherjee8119]

Databricks should have put more documentation around why volumes should be chosen over external locations for “ drop zone files” or should have put in a “information dialog” in the external location documentation. We have been using external location for the drop zone (from where autoloader picks up) because volumes weren’t around, then.

[@DatabricksPro]

Good content. I actually enjoy Unity Catalog a lot.

[@DaSenf1860-]

Thanks, Simon, for this great recap. On the last slide you mentioned Attributed Based Access Control. I know that you can implement column and row-level security and dynamic data masking. Is it this what you mean or is there are some real ABAC you are referring to? I couldn´t find anything in the docs about it :(

[@jbwnknb]

Simon, you need a new and better mic!! Thanks for your videos.

[@joyo2122]

no one uses unity catalog :D

[@AdvancingAnalytics]

But they shoooooould, because a) it's free and b) I'd put money on DBX making it mandatory at some point!

[@shikokas]

@@AdvancingAnalytics nothing is free... you can consider it as a vendor locking as for b - that is a problem as its agents the OPEN lakehouse paradigm. connecting from outside of DBX to UC has many limitations at the moment,

[@cobrider2]

We do at my client. I believe you should