97 questions
-2
votes
2
answers
505
views
Airflow 3.0.0 : Rest API : Not authenticated
I am new to Airflow and was able to configure Airflow 3.0.0 with sample docker compose. It is up and running fine. I am able to login to UI and also run dags on UI.
I actually want to trigger DAG via ...
- 179
0
votes
0
answers
449
views
Airflow 3.0.0 Cant Login to Web UI
This was a project that i was following through a youtube tutorial but since the video is old, the exact commands were not working so i had to change some of them. This is a project about ELT with ...
- 31
0
votes
0
answers
441
views
In Apache Airflow 3.x, what does the web_server_path parameter control, and how is it used?
I'm working with Apache Airflow 3.0.0 and I'm trying to serve the web UI under a custom subpath.
I updated my airflow.cfg file like this:
[webserver]
web_server_path = /myairflow
and restarted the ...
0
votes
0
answers
41
views
Airflow DAG Failing in UI but Succeeding via CLI
I’m trying to make an HTTPS call from an Airflow DAG.
I've set up the connection with the appropriate Connection ID, Connection Type, and Host (URL).
When I run the following command from the terminal,...
0
votes
0
answers
37
views
How to find run-time for a set of tasks in an Airflow DAG
I have a pretty simple DAG which has some primary operational tasks, and then some cleanup and reporting tasks. I would like to get the runtime of the operational bits either by the difference of the ...
- 230
0
votes
0
answers
59
views
Authenticating to RestAPI inside self-hosted Airflow from python script
I have a self-hosted Airflow which has Google Oauth implemented. I need to extract data from an REST API within an Airflow DAG that requires authentication. To do this, I want to use a Python script ...
- 1
0
votes
0
answers
301
views
How to Handle Skipping Subsequent Tasks for Specific Indices in Dynamic Task Mapping in Airflow Task Groups
I’ve recently started working with Airflow and I’m using Airflow 2.9.1 with the Taskflow API. I’ve created a task group with dynamic task mapping and I’m using task decorators exclusively (no ...
- 171
0
votes
1
answer
89
views
In Airflow how to avoid a DAG to run a child DAG which is disabled in the Airflow UI?
I was told to disable the dbt DAG in Airflow, which I did, but the dbt DAG in my case is called by a parent "main" DAG, which calls an "extract" DAG before calling the dbt one.
I ...
- 120
0
votes
0
answers
117
views
Endpoint added in apache airflow plugin not accessible
I'm experimenting with apache airflow for a project, installed
standalone version ( 2.9.0 ). I have tired add a new custom endpoint
in airflow by using a custom plugin.
I have created a folder '...
- 3,735
0
votes
1
answer
142
views
In Airflow Pool Dagrun status is scheduled but in List Dag Run Dagrun status is running
I am testing Airflow pool, created a pool with 2 slots, whenever i run a dag more than 2 times, 2 dagruns start and remaining in scheduled state, but for same dag when i check it on grid view all ...
0
votes
1
answer
255
views
from airflow.providers.amazon.transfers.local_to_s3 import LocalToS3Operator ModuleNotFoundError: No module 'airflow.providers.amazon.transfers'
actually my dag file count increases but dag file not shows on dag file list and broken dag: module not found error but I install required packages..but still got an error,what can I do?
I installed ...
1
vote
1
answer
1k
views
Creating json variable with Airflow web API
I'm trying to create variable with json value in Airflow using web api.
My script:
curl -X POST "${AIRFLOW_URL}/api/v1/variables" \
-H "Content-Type: application/json" \
...
0
votes
0
answers
480
views
Airflow Dag run goes into queue
facing the a issue when i run the Airflow DAG by using the api. 2nd run goes into queue. after first run only its starting. ideally we want to run in parallel. same rest api call is working in Airflow ...
0
votes
1
answer
2k
views
Airflow: from connexion.decorators.validation import RequestBodyValidator ModuleNotFoundError: No module named 'connexion.decorators.validation'
After installing Airflow in Aws or Azure Vm whenever I'm entering the command "Airflow standalone" it ends up with an error below. I have tried it in different instances of EC2 and Azure Vm ...
0
votes
1
answer
185
views
How to clear the DagRun object (<DagRun dag_id@2023-01-01 00:15:00+00:00:scheduled__2023-01-01T00:15:00+00:00, externally triggered: False>)
Goal is to find failed runs for some DAGs and trigger them (by clearing the scheduled run) programmatically without manual intervention.
I was able to retrieve the DagRun objects of failed runs using ...
0
votes
1
answer
326
views
Obtaining next scheduled run time of a DAG using Airflow 2.2.5 REST API
I used Airflow 2.4.2 REST API and python apache-airflow-client to retrieve the next scheduled run time of a DAG.
from airflow_client.client.api import dag_api
with airflow_client.client.ApiClient(...
- 553
0
votes
1
answer
279
views
How to get the status of DAG_RUN_STATUS for a DAG in airflow
How to get the execution status of overall DAG, which can be used to trigger the mail.
def on_success_dag(context):
dag = context.get("task_instance").dag_id
state=context.get("...
- 123
0
votes
0
answers
1k
views
Airflow SimpleHttpOperator and PythonOperator to extract data using API
I am new to airflow and using airflow 2.6.1 on a macOS with M1. I have a simple ETL process which uses GET request to ingest data in JSON and transform it and then save it to a db. I can do it without ...
- 1
1
vote
1
answer
2k
views
How to Handle Automatically Generated DAG Run ID in the URL of Airflow REST API?
I'm on Airflow v2.2.5 and am trying to use this API to fetch an xcom value.
My DAG runs are all done through the UI (scheduled/manual), and they have dag_run_ids with the format of <manual/...
- 1,546
0
votes
1
answer
191
views
Why my Airflow REST API returns deleted dags
I have set of REST API in Airflow , and I have multiple dags inside my dag bag, but I have deleted one dag. After that, I restarted my Airflow scheduler and web server, but I get the deleted dag as a ...
- 45
2
votes
0
answers
2k
views
Why airflow marks kubernetes pod operator tasks in airflow as failed on the UI while the containers are still running on k8s?
I am having a for loop in my DAG which creates some tasks using kubernetes pod operator. These tasks run in parallel and there are enough resources available on the k8s nodes to let them run in ...
- 2,264
0
votes
1
answer
287
views
Airflow no module found error 'airflow.providers.airbyte' on UI
Broken DAG: [/usr/local/airflow/dags/dag_airbyte_example.py] Traceback (most recent call last):
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File &...
0
votes
1
answer
475
views
DAG successfully completed even though its one of the task is getting failed internally with script
I am getting below DAG as "successfully completed instead of failed DAG, because one of the tasks failed with the SFTP connection. Can you please help me with how this update DAG with failed task....
- 123
0
votes
1
answer
449
views
Secure way to utilize AWS SSM parameter store to make API call
I need to write a lambda function which makes an API call (to Airflow) using credentials stored in AWS SSM parameter store. I have been supplied with the key id for the credentials.
How can I securely ...
- 460
2
votes
1
answer
2k
views
Airflow set dag run note
How to use the note present in DAG runs panel from the ui?
I would want to programmatically fill it. For example changing the content depending the on the params passed to the DAG run
- 89