Skip to main content
11,976 questions
Newest Active Bountied Unanswered
Filter by
Sorted by
Tagged with
0 votes
0 answers
33 views

I have an airflow podman on local which I have started with this command: podman run -d --name airflow --network airflow-net --cpus 8 --memory 8192m -e AIRFLOW__CORE__EXECUTOR=LocalExecutor -e ...
ng.newbie's user avatar
  • 3,312
0 votes
1 answer
27 views

I’m using Apache Airflow 2.x and I want to trigger a downstream DAG only when specific tasks in an upstream DAG succeed. Here’s the situation: I have an upstream DAG: dag_A Inside dag_A, I have ...
Prathima Sarvani Alla's user avatar
  • 1
0 votes
1 answer
43 views

In an Airflow Dag file, I'm trying to handle following jinja template variables as DateTime values. Specifically {{ data_interval_start }} and {{ data_interval_end }}, which (according to https://...
Jelly's user avatar
  • 1,434
Best practices
0 votes
0 replies
42 views

Are there any best practices for handling roles in code? Most of our setup is managed via Terraform. It would be great if there was a possibility. There is a Terraform provider, however using the ...
Moritz Schmitz v. Hülst's user avatar
  • 3,731
0 votes
0 answers
45 views

I’m running Apache Airflow inside a Docker container and trying to use the KubernetesPodOperator to run a simple “hello world” pod in an external Kubernetes cluster (not the same one where Airflow ...
Denis Victor's user avatar
  • 1
0 votes
0 answers
30 views

Airflow Docs say that the LocalExecutor supports an "unlimited Parallelism". Source They do not specify exactly how to run LocalExecutor in this mode, but I assume it is through the core....
Alkis Mavridis's user avatar
  • 1,255
0 votes
1 answer
112 views

I'm trying to pass Airflow logical_date to dbt model so that I can use it in the model (sql). I'm using Airflow 2.11.0. I'm doing below, but the dag couldn't get constructed, with an error says: '...
user3735871's user avatar
  • 367
-1 votes
1 answer
42 views

I'm currently working with Apache Airflow and trying to configure retries for my DAG tasks using the retry_delay (or retry_delay_sec) parameter in a YAML-based DAG definition. However, I'm running ...
Lucas Lino's user avatar
  • 1
0 votes
0 answers
48 views

I’m using Cloud Composer (Airflow) and have two BigQuery operators like this: run_aggregation = BigQueryInsertJobOperator( task_id='aggregation_task', configuration={ "query":...
efesabanoglu's user avatar
  • 1
0 votes
0 answers
74 views

I have created a DAG in Airflow that will detect a file in Azure blob storage. This is a sensor that will detect the existence of a blob in Azure blob storage. But it is failing repeatedly with the ...
awesome_sangram's user avatar
  • 89
0 votes
0 answers
25 views

I would like to use Airflow for some ETL operations, where the source data does not have (indexed) timestamp columns. The source data is a database table where new records of events are appended ...
MrGumble's user avatar
  • 5,766
0 votes
0 answers
75 views

I’m running into an issue with Airflow 3.1 using CeleryExecutor inside Docker Compose, and I can’t figure out what’s wrong. This is for a small project I’m building for my business. The idea is to use ...
Aneeta Karan's user avatar
  • 1
1 vote
0 answers
96 views

I’m trying to run Apache Airflow 3.1. with the CeleryExecutor using Docker Compose (together with Redis + Postgres). My problem is, when I trigger a DAG (directly over the command line or how it's ...
Qohelet's user avatar
  • 1,661
0 votes
0 answers
68 views

I'm using a newer version of airflow (airflow:3.0.6) with docker compose. Essentially I have just copied and slightly modified the official docker-compose file provided by Airflow itself: https://...
Qohelet's user avatar
  • 1,661
0 votes
0 answers
34 views

I am working on a platform allowing user to register DAGs to an airflow instance. Users upload the code for the DAGs they want to import along with a mapping of permission for roles on those DAGs. My ...
Jérôme Fink's user avatar
  • 176
0 votes
2 answers
80 views

I have a dag, which runs several sql scripts on certain tables. There are two options to run this dag: On the production tables On frozen archived tables I want to be able to select which tables to ...
Serge Kashlik's user avatar
  • 421
-1 votes
1 answer
73 views

I have Airflow running on Kubernetes. So, as you know, every task is spun up as it's own pod by the KubernetesExecutor. I have tried to override the individual container images with slim python images,...
Q170's user avatar
  • 1
2 votes
1 answer
253 views

I want to periodically delete old XCom records (e.g. older than 15 days) from inside a DAG. In Airflow 2 I could connect directly to the metadata DB, but in Airflow 3 direct DB access is no longer ...
Sara Brajkovski's user avatar
  • 21
0 votes
2 answers
102 views

I'm working on a task group that needs to pass a variable from a BashOperator to another BashOperator. Each bash operator is invoking Python, and the first Python script needs to return a string in a ...
BMac's user avatar
  • 302
0 votes
0 answers
62 views

I am using Google Cloud Composer environment and Apache Airflow to run my DAG. Almost all tasks are run in deferrable mode, since they are long-running. I noticed that when running a backfill job for ...
Murphy's user avatar
  • 1
0 votes
0 answers
71 views

I have a airflow DAG which calls databricks job that has a task level parameters defined as job_run_id (job.run_id) and has a type as python_script. When I try to access it using sys.argv and ...
Diksha Bisht's user avatar
  • 1
-2 votes
2 answers
505 views

I am new to Airflow and was able to configure Airflow 3.0.0 with sample docker compose. It is up and running fine. I am able to login to UI and also run dags on UI. I actually want to trigger DAG via ...
atul ahire's user avatar
  • 179
0 votes
0 answers
30 views

I'm generating dynamically based on JSON files some DAGs. I'm creating a WHILE loop system with TriggerDagRunOperator (with wait_for_completion=True), triggering a DAG which self-calls itself until a ...
Martin REPPERT's user avatar
  • 31
0 votes
0 answers
52 views

I am trying to run a batch process using Apache Airflow. The Extract and Transform stages work very fine but the load stages is giving an error. Here is my code: from airflow.decorators import dag, ...
Nwaogu Eziuche's user avatar
  • 19
0 votes
0 answers
78 views

I'm trying Apache Airflow for the first time and built a simple ETL. But after loading the data and proceeding to the transform phase, it throws an error because it says pyarrow was not found. Im ...
Enzo Martins's user avatar
  • 1

1
2 3 4 5
…
240