Dagster Crash Course: develop data assets in under ten minutes

[@roberthedblad3044]

Would love to have more of these! Unfortunately there isn't a lot of tutorials and resources around to get inte dagster as beginner. So thank you for making this!

[@dagsterio]

Take a look at dagster.io/blog/dagster-google-sheets-tutorial which is an introduction tutorial for Dagster.

Joining other comments, I'd love to see more step-by-step tutorials and use cases. It took a few videos to grasp the concepts, and this one is a good one to start with. Docs are good, but videos are even better. I would love to see more of duckdb / dagster and ingestion cases.

[@jesperbagge2504]

Great example of how to actually do meaningful stuff with actual context (no pun intended) completely free from “foo=bar”. Love it!

[@gw1284]

Thank you for this great tutorial on dagster. Will explore dagster.

[@zuesbenz]

fk the learning curve on this shit.

[@krtmlry2719]

is it really hard? Im planning to learn this too. lmao

[@raticus79]

Aha, I was wondering if we could use context with assets like that - the docs had examples for using context with ops but not assets. Good demo, thanks

[@ttehir]

Thanks for the very interesting demo. I'm still struggling to grasp some concepts and how they apply to my use cases. Elaborating on your demo, my use case would have additional requirements - the data can't be fetched in its entirety on every run of the schedule, like if GitHub had a volume limitation on the amount of data you can fetch, or rate limiting so that you can't call their API for every day of data - some days, the API might not respond but we still want to retry to fill that in - I also want to run this pipeline for, say, multiple repos. All the logic is exactly the same, but the repo changes. I may have to tweak some extra info other than the name of the repo I've looked into the docs but I'm still confused whether partitioned assets, multi assets for example will allow me to run my use cases, in a way that will solve our problems and remain within the asset based paradigm of dagster. We've used DAGs, partitions and scheduling on a 0.* Version of dagster which worked but came with some issues (partitions growing too large). Now that dagster seems to be moving away from that approach, I'd like to realign our work with the paradigm of the tools we use to ensure we're not using soon to be deprecated features. Thanks a lot if you find the time to help out :)

[@kosmylo]

What is the difference between dagit and dagster-webserver?

[@user-if2kq8nh8m]

Why was it decided to showcase this demo with "@repository" as opposed to "Definitions"?

[@selimrbd]

Repositories were the previous way of defining code locations (at the time this video was made). The new way is using Definitions

[@mansibhatt2392]

can you make a video how to define sensors in your dagster project..

[@jesusmartinalcala4730]

When i execute dagster project scaffold --name ....., in the created directory does not appear workspace.yaml, how can i fix that? Thank you in advance

[@dagsterio]

As long as you have `[tool.dagster]` pointing to the right Python module in `pyproject.toml`, you don't have to use `workspace.yaml` since Dagster 1.1.7 Details: We introduced `Definitions` to replace `@repository` and `workspace.yaml`. `workspace.yaml` is no longer the preferred solution for convenient command-line-free usage of `dagit` and other tools. Instead we just rely on a `[tool.dagster]` of `pyproject.toml` a Python standard design with extensibility points with these uses cases in mind. However, if you do want to use the older pattern, we're still supporting `@repository` and `workspace.yaml` for the foreseeable future. You can find more details in github.com/dagster-io/dagster/discussions/11167

[@TheEsisia]

Am I the only one finding it weird that you basically coding inside the _init__.py file?

[@dagsterio]

It's definitely a shortcut we sometimes use in demos to keep things simple. If a fully built-out project, you would use a full file structure facilitated by the scaffolding process.

[@hiteshmusale7895]

Good Demo. Actually I am running dbt jobs on dagster in k8s Where i have scheduled some dbt assets which runs fine when it is materialized manually but gets failed sometime when tried to run on scheduled time. Though Dagster daemon is also set up. Do you have any inputs on it ? Appreciate your help.

[@dagsterio]

Hi - Join us on Slack at dagster.io/slack and we can support you there.

[@datalearningsihan]

I am trying to install it. It does not work. Had to spin up docker, still does not work. Whats wrong. Can I get an actual non buggy guideline?

[@dagsterio]

Hi Sihan, Feel free to join us on our Slack channel (dagster.io/slack) and we can help you out. Hundreds of users have successfully completed this tutorial so we are confident we can get you unstuck! We will just need some details on what your issue is.

[@aamirahmadansari2721]

I am disappointed with the content as it was rushed. As a beginner, I would have loved a more structured and detailed tutorial/ crash course.

[@dagsterio]

Hi Aamir. Sorry if the 'crash course' seemed a bit rushed. Were you watching the video embedded in the blog post here: dagster.io/blog/dagster-crash-course-oct-2022 ? If not, the written version will help. Also we would recommend the following tutorial which is more details, and geared towards people brand new to Dagster who want to take things one step at a time: dagster.io/blog/dagster-google-sheets-tutorial

[@aamirahmadansari2721]

@@dagsterio Hey, I will definitely have a look at it and would request the team to please add a detailed version of the video for mlops. I like the premise of Dagster and would like to pitch it in my company. Your documentation is real good but a video tutorial would be very helpful.

[@dagsterio]

Agreed. We have lots of Dagster users running ML pipelines, and we hope to bring more case studies to the community in the coming weeks. In the meantime, did you see this blog post from Sandy (Lead eng on the Dagster project)? dagster.io/blog/dagster-ml-pipelines Join us in the Slack channel and we would be happy to discuss the use case: dagster.io/slack