Execute Python Script After PutFile (Apache - NiFi)

Question

I am trying to do something that I feel should be relatively simple, but I am struggling immensely to conceptualize how to execute this process.

At a high level I am simply unpacking a zip file into a directory. Once that is complete, I need to execute a python script.

The issue I am having is that I can't determine the best method to achieve this goal. I assume Notify/Wait will be the answer, but setting the release signal identifier and only releasing when all flowfiles have been placed, does not appear to be possible? I have also tried MergeContent using a like attribute, but I only want a single merged flowfile, and I cannot set the minimum # of entries = to ${fragment.count}. Please see below for any screenshots that may be beneficial. Thank you.

What does the python script do? Does it execute once for each FlowFile or once for all 16? — mattyb
– mattyb, Commented Oct 25, 2021 at 21:51
It will execute a series of batch scripts which handle logic related to the files. But we only want to execute the batch scripts if all the files are in the directory. So in shorter terms - once for all 16. — lildakota
– lildakota, Commented Oct 26, 2021 at 12:19

Sdairs · Accepted Answer · 2021-10-26 13:38:55Z

2

If you are using UnpackContent, all unpacked flowfiles get fragment identifier, index and count attributes - see the Writes Attributes section of the docs.

If you want to collect all of these into a single FlowFile, you can use MergeContent with a Merge Strategy of Defragment. This will take all of the FlowFiles created by the UnpackContent and merge them into a single FlowFile - it will group by the identifier and wait until it has seen every index up to the count - thus if 16 FlowFiles were produced, it will merge all 16 into 1.

You can use that single FlowFile to trigger your script.

answered Oct 26, 2021 at 13:38

Sdairs

2,0421 gold badge14 silver badges13 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

lildakota Over a year ago

Will check this out and report back. I assumed MergeContent could handle this, but I did not realize UnpackContent had those identifiers. Thank you.

lildakota Over a year ago

Marked as the answer. Thank you for this information. I didn't fully understand the mergecontent processor, but definitely have a better understanding now.

Collectives™ on Stack Overflow

Execute Python Script After PutFile (Apache - NiFi)

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related