How to integrate with LDAP in this version? I couldn't find 🥲

I prefer the first method. Easier to read.

awesome class tks

Excellent content terrible accent

This was a great tutorial - easy to conceptualize and a great example which was easy to follow. It's a great feeling to look at your data team's airflow github repo and actually understand what the DAGs are doing when you didn't know how airflow worked an hour ago!

Hi , I am trying to run dbt transforms in Airflow, my table raw_invoices is already created in US location. still I am getting below error in Airflow. Error in model dim_customer (models/transform/dim_customer.sql)\x1b[0m', '\x1b[0m20:02:13 404 Not found: Table airflow-project-446808:retail.raw_invoices was not found in location US; Can you suggest here?

Thank you so much for this amazing video! It was very helpful and well-explained. However, I encountered the following error when running my code: airflow.exceptions.AirflowException: Cannot execute: spark-submit --master spark://spark-master:7077 --conf spark.master=spark://spark-master:7077 --num-executors 2 --executor-memory 2g --driver-memory 2g --name arrow-spark --verbose --deploy-mode client /usr/local/airflow/include/scripts/read.py /usr/local/airflow/include/1.csv. Error code is: 1. The only additional part in my code compared to yours is the inclusion of dropna. Could you please help me figure out what might be causing this issue? Thank you in advance!

Backfill cannot happen when the dag is not scheduled ? In that case airflow dags trigger command works, but it will not be able to trigger specific tasks but entire dag

@MarcLamberti : Can you help me fix it? Docker image I am using is 12.6.0-python-3.10. Broken DAG: [/usr/local/airflow/dags/retail.py] Traceback (most recent call last): File "/usr/local/airflow/dags/retail.py", line 12, in <module> from include.dbt.cosmos_config import DBT_PROJECT_CONFIG, DBT_CONFIG File "/usr/local/airflow/include/dbt/cosmos_config.py", line 1, in <module> from cosmos.config import ProfileConfig, ProjectConfig ModuleNotFoundError: No module named 'cosmos'

Cant Astro just incorporate utg-8 encoding parameter as part of it's load file method?

Can I pls ask that why didnt we just use dbt for quality checks. What the advantages of soda over dbt

Thanks for the great video, By way is the any possibility to do push and pull request to git from the airflow, could please share any useful resource to go through. Thanks!

This video is very informative, But Is there any way to pull and push to the git repo, if there is an resource is already there, can you please share me the link or can you please share any resource that would be very helpfull, Thanks!

Done it thank you Marc. How can I query the table using athena. I tried cataloging the data using glue but crawlers can only detect general purpose buckets.

Hi Marc!, I noticed you used the BashOperator in the example pipeline @13:16 Could you please explain why you chose BashOperator over PythonOperator in this case? Also, how do you typically decide which operator to use when designing your DAGs? Thank you again for sharing your knowledge...I really appreciate your efforts!

Thanks Marc for the great video 🎉🎉. Sorry I am a beginner, just out of curiosity, how much $ does AWS charge you for this project?

Hi Marc, After selecting the python interpreter still my autocomplete doesnt work. I have tried switching between all the available one in my machine, any suggestion.

I imagine this solution in a different way. I thought it would be just putting a file or updating a file in the dataset directory (or the file) without the need for a "producer" dag. I have an external system and not a DAG that unloads the data into a directory. For me, the "TriggerDagRunOperator" already fulfills exactly the same role as the Datasets. Help me understand if that's not it. Thanks for the content Marc!

could you pls share knowledge on how to integrate airflow v2 with custom oauth2 provider (not in the supported list from the airflow doc) ?

Hey marc , I'm a beginner. loved your content. It would be really helpful if you could number the videos in this playlist to make it easier going with the right order.

Thanks for the video ! Is soda essentiel and how much does it cost, i couldnt find the price on their website

really thanks for lesson)

Good job.

Sir, when I try to inspect the network, these Airflow containers and other containers are on a different network, even though I followed the instructions. Could you please post a video to help fix this?

Great this worked fine locally. But when I tried to deploy the same code for AF deployed on using cloud.astronomer instance using celery executor, Th DAG is failing with this exceptions: raise AirflowException( airflow.exceptions.AirflowException: Cannot execute: spark-submit --master spark://spark-master:7077 --name arrow-spark --verbose --deploy-mode client ./include/scripts/read.py. Error code is: 1

Thanks for lesson. I have a problem - TypeError: _choose_best_model() missing 1 required positional argument: 'ti'. Can you help?) returned values 8, 9 and 6

i am trying to upload the csv file to google cloud bucket but am getting a TIMEOUT ERROR. I tried uploading another file that was less heavy it worked. please can someone help me @MarcLamberti

Hey Marc, I have deferred tasks in my DAG and when I use the backfill command to it, the task isn't going to deferred state, it's just marking all the tasks as successful ..... does the backfill command not work with deferrable operators ??

I would like to ask you about the DAG being scheduled on a Dataset or Dataset Alias. I assume that such DAG will only run if those datasets are modified by another DAG, not by any event that modifies those files outside the Airflow architecture?

Is it possible to run pyspark jobs from azure synapse? (for example a notebook)?

how to get the compose file with the latest images? Is the doc updated regularly or I grab latest images manually from docker hub and customize my compose file

I’ve one doubt sir what’s the usage of Installing airflow any specific reason for that ? And in what kind of scenario can we use this one ?

Great tutorial, thanks Marc!

Now waiting for video where we deploy multinode worker of spark and airflow with Kubernetes in production 🎉

Hey great video! Could you show us how to set this up using Kubernetes and helm?

If you're using Apprise, it's possible to tag people on Teams?

They are in same network i verified using inspect command🍖 able to ping spark-master from airflow webserver as well

Getting this error raise PySparkRuntimeError( pyspark.errors.exceptions.base.PySparkRuntimeError: [JAVA_GATEWAY_EXITED] Java gateway process exited before sending its port number.

Hey Marc, great video! Indeed that seems to be really easy to play around with Spark and Airflow in a dev environment with your explanation, but I have 3 questions though: (1) At the end of the video when you present the TaskFlow API way of triggering a PySpark job, you import pandas at a "top-level import" which is not really recommended if I am not wrong (should be imported within the function of the task)? I get that you need to do so since your function is returning a pandas DataFrame though :). (2) Same context as before: your function returns a pandas DF, it will be returned as a XCOM right? I remember that XCOM used to be used to pass around "little amount" of data from tasks to tasks, but I think this limitation is gone given you can set an XCOM backend to be a cloud object store (GCS, S3, ABFS). But do we really want to do that given the fact that Spark is designed to deal with big amount of data (although I totally get that it can make sense for small amount of data as showcased in your example) ? To rephrase my question: should not it be better to code your PySpark job so that the result is written in a cloud object storage instead of passing it around through XCOM? (3) In a higher environment (prod), would you still deploy Spark as a Kubernetes application (docker in your case but it is a container), or would it better to leverage the fact that PySpark applications can use Kubernetes as a cluster manager? Thanks again for the great video :)

Nice, need to YARN + TEZ and HDFS and will be grate e2e project

i dont have memberOf attribute in user's internal atrribute. How did you configure your LDAP so it creates memberOf attribute?

I think this is going to work great. I have a hundred or so PHP files to schedule and call from Airflow leveraging SSHOperator, and these calls only differs in name of php file, and sometimes location of php file, all other tasks are the same (solves a legacy issue left over from years past)... I can control scheduling in batches, and arrange for any specifics with properties...

Loving your energy!

You are simple "The Best" <3

Hey.. Marc Now please tell ne how to stop repetition of tasks as no one told in comments 😂

Hi Marc, Glue job finishes in 5 minutes but it shows as running in astro airflow local instance

I am unable to restart the airflow instance after 29:30 , can anyone plzzzzzz help

Wow this was such a great tutorial! Very easy to understand and can’t wait to try it myself!