Advancing Spark - Azure Databricks News Dec 2023 - Jan 2024

December and January are notoriously quiet months in software dev - so going through the release notes for Databricks was a bit of a shock. There have been a huge number of improvements and releases pushed over the last two months - not had time to catch up? Don't worry, we've got you!
In this video Simon walks through the release notes for December 2023 and January 2024 talking about the various features and which ones you should be excited about! Obviously, we know everyone is most excited about the new Entity Relationship Diagram in Unity Cat, but there's a lot more to see!
For the Databricks Release notes, see: learn.microsoft.com/en-us/azu...

[@roberttyler2861]

Can you do dates for widgets yet? I always thought that would be a great use case that could never be achieved

[@zycbrasil2618]

Hi Simon. Regarding XML as native format I've had problems even using library (before runtime 14.3). My question is how can we bypass rootTag and mainly rowTag if you have a nested dataframe ready to go. Also there's no support to add elements at root level. Any ideas? This needs improvements..

[@AdvancingAnalytics]

No idea - not even tried it. If anything, I've actively avoided it! - Simon

[@AlessandroGattolin]

What do you think about online tables as "back-end" for Webservice exposed to websites and mobile applications ? At the end of the day a classic old SOAP webservice was using a specific key to retrieve data, so the idea to have a "managed online tables" that scales automatically based on key-retrivial logic from web application seems interesting. Anyone thought about this or is just a bad idea ? :)

[@AdvancingAnalytics]

Definitely interesting, but I've not done any price benchmarking - you'd normally be looking at something like CosmosDB or even Redis Cache for that kind of thing, will be interesting to compare price/performance!

[@AlessandroGattolin]

@@AdvancingAnalytics I am also interested in the FinOps part, it’s time to study :) It could be interesting to have in general some FinOps comparison based on use cases for specific features for the thousands of new features coming to the Data Intelligent Platform!

[@alexischicoine2072]

There’s definitely the potential to end up paying more for less with these tangential products like Snowflake containers that have a massive markup from what’s basically kubernetes. I might use that kind of table to power processing tables for etl but I wouldn’t think powering a website is the use case.