Data Lakehouse: An Introduction

[@potnuruavinash]

Can we implement data lakehouse with open source tools like spark, presto & hive metastore ? is there any alternative for unity catalog in open source eco system

[@BryanCafferky]

Lakehouse is just Delta Lake, i.e., delta tables which are available in open source Spark so yes. Unity Catalog is really just a catalog of catalogs so you could build your own central catalog by extracting the meta data from local Hive metastores. I believe Spark tends to work one cluster at a time unlike Databricks which spins any number of clusters up as needed so not sure if UC could be implemented on open source Spark but perhaps?

[@DenisGorev-xj5hl]

It is amazing how concisely you put so much information in one video! Great!

[@gardnmi]

I'd love to see a non-bias comparison between delta lake, hudi, and iceberg.

[@BryanCafferky]

So would I. lol. Iceberg seems to be Snowflake's version of Lakehouse. Not sure about hudi.

[@BryanCafferky]

Looks like Amazon is promoting hudi.

[@sujithravindran7082]

I really enjoyed the perspective you brought into the evolution. Great work. Please keep bringing in these great videos. Thank you very much.

[@BryanCafferky]

Thank You! and you're welcome.

[@ayandapeter1681]

Sir, I just want to say thank you so much, I've gone through many videos but was still confused, u made this crystal clear with all your conceptual approach.

[@BryanCafferky]

Thank you for kind words. I'm so glad my videos are helping you. That's why I do them. I know this technology is not easy to learn so kudos to you for sticking with it.

[@antwanto]

wow that was very informative and amazing, thank you for your efforts

[@janni9789]

Again, perfectly explained. Thank you

[@rich111296]

do you have an example in any of your videos connecting to an s3 bucket specifying an endpoint within databricks? basically how to connect to an s3 bucket from a service other than aws? Thanks

[@BryanCafferky]

Hmmmm.... No have not tried that. Have you googled it?

[@rich111296]

@@BryanCafferky yeah ha, i did find a solution eventually, i think somewhere from stack overflow, searched around several places so i don't have the exact source "sc

[@rich111296]

and run the function obvi

[@prarthananeesh]

Can we use the lakehouse to replace a transactional system ?

[@BryanCafferky]

See my reply to your question about OLTP.

[@HasanCatalgol]

Underrated channel, really quality information.

[@prarthananeesh]

Is it mainly used for OLAP or can this be used for OLTP also ?

[@BryanCafferky]

It's meant for data warehousing, i.e., warehouse = lake + house, so warehouse on a data lake. OLTP has stringent requirements like high data transactions concurrency, referential integrity, etc. Delta logging is done at a file level whereas SQL databases log at a row level. See my video on Delta logs to get an understanding of what I mean.

[@BryanCafferky]

Delta Logs 1: kzbin.info/www/bejne/pnSrkKSEptOgqZI Delta Logs 2: kzbin.info/www/bejne/kIS3e3-cj96SftU

[@wennie2939]

Best video on this topic ever!

[@brokejohnnylive1530]

Dude you are on the money!! Agree all 100%.

[@ioannisnikolaospappas6703]

Life saver 🫡 Thank you sir!

[@BhaveshKumar-dz8hq]

you are a hidden gem

[@maheshthati1320]

Best explanation

[@jayashreetheagarajan2708]

Amazing contents.. Thank you Bryan

[@BryanCafferky]

You're Welcome! Glad it is helpful!

[@GILLOS21]

Amazing lecture! Thank you!

[@BryanCafferky]

You're Welcome!

[@ChristianWDegn]

Good presentation Thank!

[@BryanCafferky]

YW!

[@avishaysebban1515]

you're the best thank you.

[@BryanCafferky]

You're welcome! Thanks for watching.

[@WeAreTeamNovus]

Amazing stuff, as always!

[@BryanCafferky]

Thank you!

[@stu8924]

Thank you Bryan.

[@BryanCafferky]

You're welcome Stu.

[@kamalesht5942]

Your videos are really helping me improve the core knowledge on Data Engineering concepts. Thankyou!

[@BryanCafferky]

Great to hear! You're welcome.