Core Databricks: Understand the Hive Metastore

[@haseebjehangir3249]

Finally a video on databricks hive metastore which is well explained, thanks Bryan

[@bungaloebill4433]

Great video! I'm subscribing for the Red Green reference alone!

[@andrewpotts9948]

That's the right level of detail that I needed. Well explained. Thank you.

[@BryanCafferky]

You're Welcome!

[@JLRocco43]

I was just pondering on doing a deep dive in this today and reading a lot of docs and then you put out the video 😂 awesome work Bryan!

[@ambarishdashora5440]

This is what I was really looking for. Thank you very much for providing such an amazing explaination.

[@BryanCafferky]

You're welcome. Glad to help.

[@soumyavema6515]

Pretty clear ...very much needed before exploring Unity catalog ....Waiting for the next

[@jace_viz]

Very clear explanation. Thanks @Bryan!

[@TheAliakbarazad]

Thank you so much for despite your great knowledge about the subject, you take the time to explain it so even I can understand!!😍

[@BryanCafferky]

You're welcome. Glad it helps.

[@daminimohite3400]

super clear explanation, loved the analogy used in the beginning

[@BryanCafferky]

Thank You!

[@tiwlan]

Thank you very much for the video and the channel, I'm from Brazil and your work help me a lot!

[@BryanCafferky]

So glad my videos are helping you!

[@martalopezjurado]

I love this video!! thanks a lot. Waiting for the unity catalog video!

[@BryanCafferky]

YW.

[@kvin007]

Love the direct and clear content! Keep it going!

[@sumak151]

That's so good i enjoyed the video thoroughly..i am.just starting to understand more about azure data bricks

[@spursyou230]

thanks for video. but bit confused, when you do saveAsTable() and drop the table, will the physical data be deleted from original source? for example if I read data from AWS S3 and saveAsTable, but then drop the table, will the data in S3 also be deleted ?

[@BryanCafferky]

When you create a schema on top of an existing file, schema on read, it's really a read only pseudo table. You can also create tables that are unmanaged which means Spark will not delete them when you drop the table. If the table is defined as a managed table, dropping the table will also drop the underlying data. you need to make sure you know whether you have a managed or unmanaged table to avoid bad surprises.

[@awadelrahman]

Thanks A LOT! One question: at 17:05; did you mean "Delta Files" instead of "Delta tables" ? when you said "Detla tables are rather interesting ...."

[@BryanCafferky]

Just that a Delta file is really a Delta Table that has not been cataloged in the Hive Metastore or the Unity Catalog. But that just by pointing to the Delta file path, you can use as a table.

[@danhai7276]

Great video, waiting for the next one unity catalog.🙌

[@BryanCafferky]

Yeah. There's a lot to Unity Catalog. Also doing Databricks AI Assistant which is very cool.

[@mehulkhare8278]

Thanks for making it simple to understand.

[@BryanCafferky]

You're Welcome! Glad it helped.

[@Kete-Dude]

have some confused about unmanaged and managed, in the step `create delta table that stored in hive` the type of dimgeography is Managed but it still can drop by not get rid of the physical files like Unmanaged(External), so what's the difference point of it?

[@BryanCafferky]

Yes. It is confusing. Think of a managed table as being like a SQL Server table if that helps. SQL Server tables are created and dropped with all the data via a DROP TABLE statement. Spark supports similar functionality for Managed tables in which the table schema and underlying data are created at the same time. This is to mimic SQL database type of functionality. Unmanaged tables are when you already have an external file and you create a schema defining the columns names and types describing the table so Spark can allow you to use SQL queries against it. Since the file pre-exists and is maintained separately from the Hive Metastore or Unity Catalog, you don't want the physical file deleted when you issue a SQL DROP TABLE statement. Bottom line: if you want the table to be treated just like an RDBMS would treat it, i.e. catalog entry and physical data handled via SQL, you want Managed. If you want to use SQL queries against a pre-existing data file, you want to define it as Unmanaged. Make sense?

[@jbab9618]

Hi @BryanCafferky if CSV file meta data is change then hive metastore automatically update metadata in hive store, is it right else we can do any steps for refresh metadata ?

[@BryanCafferky]

A Hive table definition over a CSV file is read only and to get the meta data reloaded, I believe you would need to drop and re-create the table.

[@sujitunim]

Thanks Bryan for this amazing session

[@BryanCafferky]

YW

[@joshuawagner5350]

Exceptional explanation. Thank you.

[@BryanCafferky]

Glad it was helpful.

[@YiminWei-z6w]

great explanation. Thanks!

[@renegade_of_funk]

You’re doing the Lord’s work. 👌

[@GhernieM]

Hey Bryan, do you plan to create something about Unity Catalog?

[@pal3201]

Can you tell us when are you releasing your take on Unity Catalog ? Looking forward to it.

[@BryanCafferky]

So many things to cover these days. Hopefully, soon. Thanks!

[@rabeMa]

Deadly clear, awesome 👌👌👌💯💯💯

[@nargesrokni6348]

very good explanation, thank you very much man

[@BryanCafferky]

YW

[@devigugan]

Excellent narrative ❤❤❤

[@malaka123456]

Great video!

[@BryanCafferky]

Thanks!

[@etianemarcelino5706]

Great content... Like always

[@benjaminwootton]

Good video. Though I understand Hive Metastore, it confuses me why everything in data has a dependency on it. For instance, Iceberg seems to need it for everything even though it’s supposed to be a self describing table format.

[@BryanCafferky]

Technically, you don't need the Hive metastore to read Delta tables. But it provides a look up to where the table is physically stored. Otherwise, you need to provide the full path to the storage location. It also stores schemas for files that don't have built-in schemas like CSV and Text files.

[@ManishSharma-fi2vr]

Thanks Bryan!!

[@BryanCafferky]

You're welcome!

[@naveenagrawal_nice]

Loved it

[@ngneerin]

This gave real good idea

[@CaponordRevHappy]

Superb! thank you.

[@BryanCafferky]

You're Welcome!

[@ravinarang6865]

Very Good.