Advancing Spark - Azure Databricks News November 2023

We originally thought we had a fairly quiet month until we ran through the release notes, turns out there were quite a few big things happening in November '23 with Databricks. Didn't have time to go wade through the release notes yourself? Don't worry, we've got you covered!
There's some big ticket items like Databricks Vector Search out in preview, Unity Catalog by default, massive serverless news for the UK and a tasty new runtime release in 14.2!
In this video Simon walks through the release notes for November, and the new DBR 14.2 release, talking about the various features and which ones you should be excited about!
For the Databricks Release notes, see: learn.microsoft.com/en-us/azu...

[@azharuddinmohammad5786]

Good job

[@brads2041]

I'm experiencing concurrency issues now on a merge to a table by multiple notebooks. So, interested in the new row level thing

[@juandiegoechavarria1839]

Could you talk about the integration of unity catalog with Microsoft Fabric?

[@AdvancingAnalytics]

It's currently tricky - as I don't think there's an actual integration currently. The main piece of advice is that if you're looking to shortcut your databricks lake into Fabric, you'll need to avoid managed tables so Fabric can actually read the folder structure (rather than the managed table GUIDs that Unity uses). I'm hoping we'll see some actual collaborative integration between the two at some point!

[@andrewli7542]

(I am a Databricks employee) We are working with MSFT Fabric team on an official integration with Unity Catalog.

[@zycbrasil2618]

Hi Simon. I have issues with foreign catalogs (Azure SQL DBs) access for my team in notebooks and sql api. It works with sql editor. Clusters are single user and UC enabled. I as admin can run queries everywhere but the team got errors like "user is not an owner of connection" Any idea?

[@AdvancingAnalytics]

Sounds like a permissions issue with the catalog? Hard to say without looking into it, probably better to speak to someone on the databricks side?

[@zycbrasil2618]

@@AdvancingAnalytics Permissions are ok. It's very specific to foreign catalog. Will try with databricks . Tks

[@andrewli7542]

On single user clusters you need ownership of connection to access foreign catalogs, due to the security limitation of single user cluster where the assignee has root access and UC can't hide the foreign catalog credentials from the user. You can make the owner of a connection a group, and then assign single user cluster owners as members of the group.