Split values inside attribute into multiple flowfiles

Question

I have a flowfile with empty content and some attributes.

I'm interested in the attribute ids. It contains list of ids with , separator.

1018866556,1018878837,1018522766,1018522773,1018522788,1018522790,1018522797,
1018522959,1018522963,1018522968,1018522972,1018522981,1018511143,1018511174

I'm using this attribute in InvokeHTTP processor, but sometimes errors occur due to a large amount of data.

If an error occurs I want to split values inside this attribute. I should get two flowfiles instead of one with the same (or close to it) number of ids inside each flowfile.

So, after transformation I expect:

Flowfile #1 with ids 1018866556,1018878837,1018522766,1018522773,1018522788,1018522790,1018522797
Flowfile #2 with ids 1018522959,1018522963,1018522968,1018522972,1018522981,1018511143,1018511174

If error still occurs, then split them again, etc... Like binary search but without sorting.

How to reach it?

don't you think it's better to know the limit of items supported by api? in any case i think script is the best choice. — daggett
– daggett, Commented Aug 25, 2021 at 15:32
True, but api limit is - 2000 ids, but sometimes API can't handle even 50 ids due to a large amount of data (internal server error occurs). If I'll run with 1 id per request - i'll eat quota limits. So, I need to balance it. About script: how to split 1 flowfile into 2 diffrent flowfiles? Can you please provide example? — miroha
– miroha, Commented Aug 25, 2021 at 15:57

daggett · Accepted Answer · 2021-08-25 16:15:41Z

1

script for ExecuteGroovyScript processor:

def ff=session.get()
if(!ff)return

def idList = ff.ids?.split(',')
if ( idList.size()>1 ){
    def sublistSize = (idList.size()/2).round() as int
    def ffList = idList.collate(sublistSize).collect{sublist-> 
            ffOut = ff.clone(true)
            ffOut.ids = sublist.join(',')
            ffOut
        }
    REL_SUCCESS << ffList
    ff.remove()
}else{
    REL_SUCCESS << ff
}

answered Aug 25, 2021 at 16:15

daggett

29k3 gold badges52 silver badges61 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Split values inside attribute into multiple flowfiles

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related