Advancing Spark - Tracking Lineage with Unity Catalog

Tracking dependencies and lineage across the lake has been a long term objective for many teams, tools and vendors, but it's hard to get right! Unity Catalog recently pushed their Lineage functionality into public preview, so it's time to take a look and see how well it works!
In this video, Simon opens up his Unity Catalog-enabled workspace and creates some inter-dependent tables, tracking the lineage objects that are created in Unity Catalog as a result!
To get started with Lineage tracking, check out the docs here: docs.databrick...
And as always, if you're starting out on your Lakehouse journey, get in touch with Advancing Analytics to give your platform a much-needed boost!

[@paulroome3928]

Great video as always Simon! Files within lineage is coming soon 🙂

[@drummerboi4eva]

Amazing stuff SImon. I told my colleagues when Spark 3 first came out, that we need a lineage tool, and here it is. Fantastic, even though with limitations

[@alexischicoine2072]

Super interesting Simon. I know it’s just an example to explain, but might want to make sure others don’t have permissions on these poc tables ;)

[@arunr2265]

Great Video Simin. Waiting for the video on unity catalog set up ;)

[@swapnilsaxena9577]

Nice vid, Simon. With regards to extracting lineage programmatically, same doc covers REST API to extract table/column lineage too.

[@briancuster7355]

Unity catalog is a great start but I can see a load of potential from this service.

[@EvgenAnufriev]

Thanks for the video. Could you clarify if GA means that Unity catalog is free? If the company has Premium subscription does it requires to pay for the feature Unitiy catalog provides?

[@JLRocco43]

Can answer for you. I deployed UC for my company. Just need to have premium and you can deploy it for free

[@EvgenAnufriev]

@@JLRocco43 Thank you!

[@alexischicoine2072]

Databricks pretty much only charges you for compute and things like the meta store or notebooks are free. You have to pay a bunch of related stuff in your cloud provider though.