Getting started with Dagster | Create Python ETL | Orchestrate ETL Pipelines with Dagster

[@bralabala]

very helpful, but a few things have changed since the project was recorded. for example `dagster new-project ` is now `dagster project scaffold --name `

[@BiInsightsInc]

Yes, for the new versions this command has been updated. In the new version 1.1.21/0.17.21 (libs) the command to create a new project is updated to: dagster project scaffold --name my-dagster-project Here is there official docs: docs.dagster.io/getting-started/create-new-project

[@ianyoung_]

Thanks for the helpful tutorial. I'd love to see a follow-up on how to deploy to a production environment using CI/CD. The workflow from local changes to production deployment would be very useful.

[@tkeus991]

thanks a lot man ! i'm starting out with dagster and i'm completely clueless . this will help out a little bit :)

[@BiInsightsInc]

Glad to hear it is helping out!

[@rizzrak]

Helpful tutorial. Thanks for this. Pls make more videos

[@siddharthasahu2603]

I work in Windows Subsystem for Linux, Just so because Linux is more comfy for me.. Nice Tutorial btw

[@hungnguyenthanh4101]

hi, i watched the video and it's great. You said in the video that Dagster is only suitable for ETL with small to medium data sources, you rate Dagster as medium to good. But I have the following advice: your data pipeline is using python, so I think this ETL performance depends on the ETL tool here, python, not Dagster. If we use Dagster to manage the data pipeline for ETL work like Apache Kafka,Pyspark,Dbt tools then I think it's much faster. I'd say that ETL performance is in the technology used and not the management tool. thanks for reading.

[@BiInsightsInc]

Hi @hungnguyenthanh4101 thanks for stopping by. I am referring to the Dagster open source setup shown in the video. This excludes Dagster cloud offering. I cover ETL pipelines with Python and it's a common concern of the viewers where sheer data size can overwhelm the system's resources. The concern is not performance but available resources. Dagster and Python are both restricted by the resources available on the machine they are running on. Therefore, if you are trying the open source version on your machine I'd recommend small to medium size data load with this setup. Hopefully this provides you with some context. Other tools you mentioned, excluding DBT, are distributed in nature and are recommended to be set up on a cluster. If you have a cluster set up for Dagster install then by all means run any size data pipeline on it. I would be curious to see if you have done any setup for managing Apache Kafka and/or PySpark please feel free to share it with the rest of the community.

[@ishatripathi7125]

Thanks for the video it was really helpful. Could you make some more videos on dagster like a tutorial or something like that.

[@BiInsightsInc]

Thanks Isha. What sort of content would you like to see on dagster i.e. Overview? Use case?

[@ishatripathi7125]

@@BiInsightsInc a step-by-step guide on having a scheduler/sensor which gets triggered whenever a new row is inserted into a db and then do some other tasks and then stores it on amazon s3 or something like that. Once, again thanks a lot for replying.

[@kikecastor]

Thank you! Great video

[@alexzir]

Thanks 🙏 Continue about Dragster please

[@sbj6173]

Thanks for the great explanation 😊

[@user-gr4pv4qh6t]

Now im trying the exact same thing but getting errors. get the provide the new version video or documents that helps us

[@BiInsightsInc]

Here is the link to the whole Dagster series: hnawaz007.github.io/dagster.html Second video has the update install directions. Here is the video on how to navigate the channel's website: kzbin.info/www/bejne/ppvMp2mgbNmsrrs

[@Pasdpawn]

hi, great video. have one question though. how do i run the scheduled dagster job even when my pc is turned off? Cos when my pc is off, dagster daemon wont run and therefore the job will also not run. how do i overcome this?

[@BiInsightsInc]

You can subscribe to their cloud offering and this manner your jobs will run on specific time as the servers will be on. Another option is to install dragster on your server that’s always on so dagster daemon can run in the background and monitor schedules.

[@ExploreWithArcha]

command for creating a new project is not working, dagster new-project etl, what to do

[@BiInsightsInc]

Please check if dragster is installed properly and check the dagster version. In the new version 1.1.21/0.17.21 (libs) the command to create a new project is updated to: dagster project scaffold --name my-dagster-project Here is there official docs: docs.dagster.io/getting-started/create-new-project

[@BiInsightsInc]

@Yuvashree P what version of Dagster are you using? And share the detail error message you are receiving when create a new project.

[@BiInsightsInc]

For projects using newer version 1.1.20 or 0.17.20 the command includes an additional parameter: "scaffold". Thanks for sharing. To get started, you can run: pip install dagster dagster project scaffold --name my-dagster-project

[@lokendrasinghtanwar5917]

having issue in setting up environment variable , what will be the directory for DAGSTER_HOME variable

[@BiInsightsInc]

Hi Lokendra, your DAGSTER_HOME variable value should be the directory that contains the dagster.yml file. For example my yaml files exist in following directory: G:\dagster\etl this is my DAGSTER_HOME value. By default Dagster will look for an instance config file at $DAGSTER_HOME/dagster.yaml. This file contains each of the configuration settings that make up the instance.

[@pybokeh]

Aren't you missing a workspace.yaml file? You can't just run the dagit command @4:50 by itself without the workspace.yaml file.

[@pybokeh]

Nevermind, I mistakenly thought your current working directory was ../etl/etl. Probably need to mention that you would need to run the dagit command in the same directory containing the workspace.yaml file.

[@BiInsightsInc]

@@pybokeh I will add this to the description too. But this comment will help someone in the future.

[@harshitamehta2253]

command for creating a new project is not working, dagster new-project etl. Getting error, AttributeError: module 'pendulum' has no attribute 'Pendulum'

[@BiInsightsInc]

The command to create a new project has changed. You can issue the following command to create a new project: dagster project scaffold --name my-dagster-project

[@harshitamehta2253]

@@BiInsightsInc I tried this as well but still facing the same error. I am not able to figure out exactly why this is happening. Do you have any idea ?

[@BiInsightsInc]

@@harshitamehta2253 What do message do you get back when you issue the above command? You may want to check if you have dagster and/or Python installed. Issue following commands and see if you get the versions. dagster --version python --version

[@hungnguyenthanh4101]

I don't know if you can make a video on how to install it on docker.

[@BiInsightsInc]

I will cover the docker install in the future videos.

[@hungnguyenthanh4101]

​@@BiInsightsInc Tks u,i am forward to

[@alexzir]

What is better for you Airflow or Dragster?

[@BiInsightsInc]

It depends on your needs. If you want to simply orchestrate a workflow then Airflow is better. It is a mature tool with plenty of guides and ample documentation. However, if you want to extract data and then pass it to another function let's say to perform transformation then Dagster is better choice. It can handle small to medium size data well. Airflow does not handle data between task gracefully yet. Maybe future releases will address this issue.

[@alexzir]

@@BiInsightsInc thank you!

[@julesm6601]

No jobs Your definitions are loaded, but no jobs were found.

[@BiInsightsInc]

You can share your project and one of us can help you spot anything you have missed. Try it with a simple hell job to see if this get's picked up Dagster. Also, try copying the project from the GitHub and give it a try see if that works for you. I have tested this project on the latest version dagster, version 1.1.21 and it works as expected. Hope this helps.

[@BiInsightsInc]

Related videos on Dagster & ETL orchestration topic: Dagster updated video: kzbin.info/www/bejne/nGK3k4l9mc2gj8k&t Windows Task Scheduler: kzbin.info/www/bejne/f6TYcp-qhbSce9k&t ETL with Airflow: kzbin.info/www/bejne/m4vJdWmubqt9aqc&t