Building a Lakehouse Architecture with Azure Databricks with Christopher Chalcraft

Join Christopher Chalcraft as he discusses building a lakehouse in Azure Databricks.
Today's companies are dealing with data of many different types, in many different sizes, and coming in at varying frequencies. These companies are looking beyond the limitations of traditional data architectures to enable cloud-scale analytics, data science, and machine learning on all of this data. One architecture pattern that addresses many of the challenges of traditional data architectures is the lakehouse architecture.
Lakehouse combines the low cost and flexibility of data lakes with the reliability and performance of data warehouses. The lakehouse architecture provides several key features including:
- Reliable, scalable, and low-cost storage in an open format
- ETL and stream processing with ACID transactions
- Metadata, versioning, caching, and indexing to ensure manageability and performance when querying
- SQL APIs for BI and reporting along with declarative DataFrame APIs for data science and machine learning
This video was recorded during the Calgary Azure Analytics User Group on Feb 23, 2021.
Join the Calgary Azure Analytics User Group LinkedIn Group and never miss a meeting!
bit.ly/CalgaryAzureUG
Christopher Chalcraft Social Profiles
LinkedIn: / cchalc
Twitter: / cchalc
----------------------------------------------------
Follow Iteration Insights!
LinkedIn: / iter. .
Twitter: / iterinsights
Facebook: / iterationins. .
Instagram: / iterationin. .
www.iterationinsights.com
Music: www.bensound.com

[@martinhorvath181]

Thanks, helped a lot to get first insights into the lakehouse topic, very well explained.

[@DilipDiwakarAricent]

Great explanation

[@jpb2085]

Well explained, thanks for sharing. Wanting to experiment building a data lakehouse myself but having difficulty finding appropriate open dataset(s) of structured and unstructured for both batch and stream. Any suggestions welcomed!

[@ranjansrivastava9256]

Very well explained Christopher, if possible could you please share the git link for the code which you have written in the Notebook.