Introducing External Assets and Dagster Pipes -- Dagster Launch Week - Fall 2023

Nick Schrock (founder and CTO of Dagster Labs) shares details on two new powerful features of Dagster.
Read the blogpost on Dagster Pipes here: dagster.io/blog/dagster-pipes
Read the blogpost on External Assets here: dagster.io/blog/dagster-exter...
1) External Assets - the ability to tap into Dagster's control plane observability, lineage, and data quality capabilities without adopting Dagster as the orchestrator across all environments. You can now expand Dagster's span of control to legacy systems without having to do an infrastructure migration.
2) Dagster Pipes is a protocol to invoke remote computations seamlessly in hosted execution engines and remote environments. This reduces the incidental complexity of dealing with external computations and lets you bring in computations in any programming language.
00:00 Introduction and recap of Dagster Launch Week
03:10 Deconstructing Dagster:
05:05 The three layers of Dagster's system
07:06 Separating the layers of the stack (External Assets and Pipes)
08:54 An example use case for External Assets
14:34 Dagster Pipes overview
28:06 Coding example
35:57 What's launching this week?
Try Dagster today with a 30-day free trial: dagster.io/lp/dagster-cloud-t...

[@JohnoScott]

Pulling Dagster apart is a great Dr Manhattan move and will make you stronger with all these new use-cases you have unlocked. A great response to user feedback

[@tintrung6395]

Hi dagster admin, can you share the code of the above examples cause it is hard to find the full document of dagster elements. In some cases I need to read the code that helps me understand Dagster infrastructure.

[@dagsterio]

You will find the code broken down in the companion blogpost: dagster.io/blog/dagster-pipes

[@jordanfox470]

Is there a link to a databricks pipes example?

[@dagsterio]

We dive into the Databricks example (with code) in the blogpost found here: dagster.io/blog/dagster-pipes

[@congtinNguyen]

I have struggled with Dagster concepts for some time. Its concepts are numerous and quite overwhelming for beginners.

[@dagsterio]

No doubt that evey new powerful framework takes some investment up front to learn. Have you explored Dagster University? courses.dagster.io/courses/dagster-essentials

[@congtinNguyen]

@@dagsterioThank you so much. It has helped enlighten many things