Why is Airflow Bash operator not passing XCom to another operator?

Question

I'm working on a task group that needs to pass a variable from a BashOperator to another BashOperator. Each bash operator is invoking Python, and the first Python script needs to return a string in a variable that the second bash operator will take and continue with:

first_status = BashOperator(
    task_id=1st_taskid_str,
    bash_command=f"python myscript.py --dag_id '{dag.dag_id}' \
                            --task_id '{1st_taskid_str}' --dag_conf configuration_string",
    retries=10,
    dag=dag,
    retry_delay=timedelta(minutes=1),
    do_xcom_push=True,
)

#step 2 

second_status = BashOperator(
    task_id=process_file_task_id
    ,bash_command=f"python secondscript.py --dag_id '{dag.dag_id}' \
                            --task_id '{2nd_task_id}' --dag_conf configuration_string --file '{{ ti.xcom_pull(task_ids=\"{1st_taskid_str}\") }}'"
    ,dag=dag
)

first_status >> second_status

When I view the XCom from the first_status task, I'm not seeing the variable that is logged by the Python script invoked.

How can I get this variable passed from the first bash operator to the second bash operator?

what do you have in myscript.py? Maybe it has wrong code.

furas
– furas

2025-09-02 15:54:19 +00:00
Commented Sep 2 at 15:54 — furas
– furas, Commented Sep 2 at 15:54

Cp Gowtham · Accepted Answer · 2025-09-03 06:59:05Z

1

BashOperator with do_xcom_push=True only stores the last line of stdout in XCom.
So in myscript.py, make sure the very last line just prints the value you want:

print("/tmp/output.csv")

Then pull it in the next task like you did:

--file '{{ ti.xcom_pull(task_ids="first_task") }}'

That will pass the variable correctly.

answered Sep 3 at 6:59

Cp Gowtham

3611 silver badge7 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

BMac · Accepted Answer · 2025-09-04 13:32:10Z

0

the issue was that the function that myscript.py was calling was not returning the value needed by xcom.

further, the xcom needed to be assigned to a variable
`xcomvar = '{{ ti.xcom_pull(task_ids="1st_taskid_str" }}'

then using that in the bash operator

bash_command=f"python secondscript.py --dag_id '{dag.dag_id}' \
                            --task_id '{2nd_task_id}' --dag_conf configuration_string --file {xcomvar}'"

answered Sep 4 at 13:32

BMac

3021 gold badge2 silver badges12 bronze badges

Collectives™ on Stack Overflow

Why is Airflow Bash operator not passing XCom to another operator?

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related