Making Apache Spark™ Better with Delta Lake

  Рет қаралды 180,544

Databricks

Databricks

Күн бұрын

Пікірлер: 16
@sonagy23
@sonagy23 2 жыл бұрын
28:32 How does Delta Lake work? 28:50 Delta On Disk 29:59 Table = result of a set of actions 31:31 Implementing Atomicity 32:48 Ensuring Serializability 33:33 Solving Conflicts Optimistically 35:08 Handling Massive Metadata 36:32 Roadmap 38:20 QnA
@kbkonatham1701
@kbkonatham1701 2 жыл бұрын
hi kim thanks for support , you are from ? , i am from india.
@meryplays8952
@meryplays8952 4 жыл бұрын
The architecture comes with a nice VLDB 2020 paper (which the presenter did not mention).
@rakshithvenkatesh2773
@rakshithvenkatesh2773 4 жыл бұрын
I see this whole "Hierarchical Data Pipeline" strategy being talked about quite a bit these days. We did establish this as part of a ready solution we built for Manufacturing use case using Confluent Kafka + KSQL. But the Data Lake is something i believe will remain/continue to exist as a depot for long term retention of data where AI/DA platforms leverage data from these data lakes for batch processing. I see this story from DataBricks to be a Data-warehouse convergence towards Data Lakes !
@Sangeethsasidharanak
@Sangeethsasidharanak 4 жыл бұрын
27.28 on automating data quality. .. isn't it same as we do quality check before we save using custom code..Will there be any additional benefits?
@gustavemuhoza4212
@gustavemuhoza4212 3 жыл бұрын
It's probably the same, but not sure how you could do that on a datalake consistently. As described here, Delta appears to make it easier to do and making it possible to do it as if you were doing it on a relational database.
@srh80
@srh80 Жыл бұрын
Wait, people still use comcast and watch TV?
@hanssylvest8390
@hanssylvest8390 4 жыл бұрын
Please give all empl. a better audio recording microphone.
@jacekb4057
@jacekb4057 Жыл бұрын
Or use some AI audio cleaner :D
@moebakry3203
@moebakry3203 3 жыл бұрын
What is the best way to load data from Sql server to Delta lake every 5 seconds?
@NicholasGabriel04
@NicholasGabriel04 Жыл бұрын
debezium
@RossittoS
@RossittoS 3 жыл бұрын
Excellent features!!
@hidemisuzuki965
@hidemisuzuki965 3 жыл бұрын
Where can I download the slides? Thanks!
@rahulpathak3161
@rahulpathak3161 4 жыл бұрын
Thank you and can you please share PPT..
@張博凱-p7z
@張博凱-p7z 4 жыл бұрын
www.slideshare.net/databricks/making-apache-spark-better-with-delta-lake
@hanmuster
@hanmuster 4 жыл бұрын
@@張博凱-p7z Many thanks!
Delta Live Tables A to Z: Best Practices for Modern Data Pipelines
1:27:52
Simplify and Scale Data Engineering Pipelines with Delta Lake
57:53
“Don’t stop the chances.”
00:44
ISSEI / いっせい
Рет қаралды 62 МЛН
UFC 310 : Рахмонов VS Мачадо Гэрри
05:00
Setanta Sports UFC
Рет қаралды 1,2 МЛН
Master Databricks and Apache Spark Step by Step: Lesson 1 - Introduction
32:23
7 Best Practices for Implementing Apache Iceberg
57:01
Tabular
Рет қаралды 8 М.
Intro to Databricks Lakehouse Platform Architecture and Security
28:47
Simplify ETL pipelines on the Databricks Lakehouse
30:19
Databricks
Рет қаралды 27 М.
Delta Lake for Apache Spark - Why do we need Delta Lake for Spark?
18:57
Learning Journal
Рет қаралды 46 М.
Databricks, Delta Lake and You
48:02
SQLBits
Рет қаралды 19 М.
GraphRAG: The Marriage of Knowledge Graphs and RAG: Emil Eifrem
19:15