3.2 - Externalize Hive Meta-store with Cloud SQL | Apache Spark on Dataproc | Google Cloud Series

In this video, we'll see how we can externalize the Hive metastore by using a Cloud SQL based MySql Instance in GCP. We will achieve complete Compute-Storage Isolation by moving both metadata and data out of the cluster.
This video is part of the course Apache Spark on Dataproc. You can find all the videos for this course in the following playlist.
• Apache Spark on Datapr...
I regularly blog and post on my other social media channels as well, so do make sure to follow me there as well.
Medium : / sushil_kumar
Linkedin : / sushilkumar93
Github : github.com/kay...

[@shefalishrivastava8950]

These videos are literally SAVING my life, thank you!

[@javedmohammad6404]

Great job Sushil

[@balakrishnanramachandran8828]

great , to point the video .

[@balakrishnanramachandran8828]

* to the point

[@BriteRoy]

Question : what wil be the use-case of such a scenario where we are separating computing and storage and later deleting the dataproc cluster. The example you showed here is just for a one time job. What about a daliy batch processing where files are coming to GCS on a daily basis and pyspark job is required to compute. How will be the cluster availablility be managed in such a case ?

[@limeraghu579]

This is again great , can you show how to access that data using spark sql shell

[@SunnyG9]

it errors out - Service Hive server2 is not enabled ... localhost port 3306 (tcp) failed: Connection refused ... dataproc cluster is created but not able to connect to Hive metastore ...

[@DivyanandSharma-l8d]

Hi shushil, getting this error while creating shared dataproc cluster. There was no instance found at projects/projectid/instances/hive-mysql or you are not authorized to access it. I have enabled the cloudsql admin api

[@rutujapabale5080]

Same pinch

[@rutujapabale5080]

could you help me if you could solve it?

[@loke261989]

U created a mysql for metadata, by default will the data also be stored in cloud storage instead of hdfs?? Maybe bcs of ur wharehouse.dir property I guess, pls clarify

[@kaysush]

Hey, no for every hive database you create, you'll have to set the LOCATION property to `gs://your-bucket/` to do that. Otherwise default will still be HDFS.