Branches created using BranchPythonOperator do not merge?

Question

I am using BranchPythonOperator to make branches in airflow. My use case is I need to make two branches from mainstream. Branch A (which has few tasks) will be followed when somefile.csv is present otherwise Branch B(which has no task) to follow. At last both branches should be merged to make a mainstream again. Now I am able to follow either Branch A or Branch B but the issue is if I follow Branch B final mainstream tasks executed and if I follow Branch A final mainstream tasks are being skipped.

MainstreamTaskA.setDownStream(MainstreamTaskB)
MainstreamTaskB.setDownStream(BranchATaskA)
BranchATaskA.setDownStream(MainstreamTaskC)
MainstreamTaskB.setDownStream(MainstreamTaskC)

I have set trigger rule as "all_done" in MainstreamTaskB and MainstreamTaskC.

Can somebody guide me through this?

Please share your process code.

joebeeson
– joebeeson

2018-11-25 00:17:56 +00:00
Commented Nov 25, 2018 at 0:17 — joebeeson
– joebeeson, Commented Nov 25, 2018 at 0:17

kaxil · Accepted Answer · 2018-11-25 22:31:13Z

1

I cannot see the other branch in your dependencies. The only branch is BranchATaskA. But based on what you have mentioned you should have the following task dependencies and have two branch tasks BranchATaskA and BranchATaskB.

MainstreamTaskA >> MainstreamTaskB
MainstreamTaskB >> BranchATaskA >> MainstreamTaskC
MainstreamTaskB >> BranchATaskB >> MainstreamTaskC

You should have trigger rule as all_done on MainstreamTaskC.

answered Nov 25, 2018 at 22:31

kaxil

19.1k3 gold badges67 silver badges87 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Jitesh Sharma Over a year ago

I understand your doubt. But there are no tasks in BranchB. It is like following a mainstream only but at one task based on certain condition I need to include 1 or more tasks in between MainstreamTaskA and MainstreamTaskC.

kaxil Over a year ago

You can use DummyOperator for BranchB. There can't be a no-task in the 2nd branch.

Jitesh Sharma Over a year ago

Thanks for your answer. It worked. I was also searching and got this, airflow.apache.org/concepts.html and search for this string "If you want to skip some tasks, keep in mind that you can’t have an empty path, if so make a dummy task". I added this link here just for reference.

Collectives™ on Stack Overflow

Branches created using BranchPythonOperator do not merge?

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related