Core Databricks: Understand the Hive Metastore

A core part of the Databricks ecosystem is the Hive Metastore which enables Spark SQL. But how does Hive work and how do you use it? How does Hive relate to the new Unity Catalog? Join me as I answer these questions and more.
Support Me on Patreon Community and Watch this Video without Ads!
www.patreon.com/bePatron?u=63...
Link to slides, data, and code (Databricks Notebook in dbc format):
github.com/bcafferky/shared/b...

[@YiminWei-z6w]

great explanation. Thanks!

[@andrewpotts9948]

That's the right level of detail that I needed. Well explained. Thank you.

[@BryanCafferky]

You're Welcome!

[@haseebjehangir3249]

Finally a video on databricks hive metastore which is well explained, thanks Bryan

[@JLRocco43]

I was just pondering on doing a deep dive in this today and reading a lot of docs and then you put out the video 😂 awesome work Bryan!

[@kvin007]

Love the direct and clear content! Keep it going!

[@soumyavema6515]

Pretty clear ...very much needed before exploring Unity catalog ....Waiting for the next

[@joshuawagner5350]

Exceptional explanation. Thank you.

[@BryanCafferky]

Glad it was helpful.

[@martalopezjurado]

I love this video!! thanks a lot. Waiting for the unity catalog video!

[@BryanCafferky]

YW.

[@danhai7276]

Great video, waiting for the next one unity catalog.🙌

[@BryanCafferky]

Yeah. There's a lot to Unity Catalog. Also doing Databricks AI Assistant which is very cool.

[@sujitunim]

Thanks Bryan for this amazing session

[@BryanCafferky]

YW

[@etianemarcelino5706]

Great content... Like always

[@rabeMa]

Deadly clear, awesome 👌👌👌💯💯💯

[@mehulkhare8278]

Thanks for making it simple to understand.

[@BryanCafferky]

You're Welcome! Glad it helped.

[@vinit.khandelwal]

This gave real good idea

[@nargesrokni6348]

very good explanation, thank you very much man

[@BryanCafferky]

YW

[@renegade_of_funk]

You’re doing the Lord’s work. 👌

[@ravinarang6865]

Very Good.

[@CaponordRevHappy]

Superb! thank you.

[@BryanCafferky]

You're Welcome!

[@GhernieM]

Hey Bryan, do you plan to create something about Unity Catalog?

[@pal3201]

Can you tell us when are you releasing your take on Unity Catalog ? Looking forward to it.

[@BryanCafferky]

So many things to cover these days. Hopefully, soon. Thanks!

[@benjaminwootton]

Good video. Though I understand Hive Metastore, it confuses me why everything in data has a dependency on it. For instance, Iceberg seems to need it for everything even though it’s supposed to be a self describing table format.

[@BryanCafferky]

Technically, you don't need the Hive metastore to read Delta tables. But it provides a look up to where the table is physically stored. Otherwise, you need to provide the full path to the storage location. It also stores schemas for files that don't have built-in schemas like CSV and Text files.

[@jbab9618]

Hi @BryanCafferky if CSV file meta data is change then hive metastore automatically update metadata in hive store, is it right else we can do any steps for refresh metadata ?

[@BryanCafferky]

A Hive table definition over a CSV file is read only and to get the meta data reloaded, I believe you would need to drop and re-create the table.