Combining AWS step function task and map outputs in to one array

Question

I have a task state that outputs the following:

"batch": {
    "batch": "size",
    "currentTimestamp": 1596205376
  },

and a map state out outputs an array:

"batch": [
    {
      "batch": "product-batch-0",
      "currentTimestamp": 1596205376
    },
    {
      "batch": "product-batch-1",
      "currentTimestamp": 1596205376
    }
]

I would like to combine them so that the input to the state that follows the map state is this:

  "batch": [
    {
    "batch": "Size",
    "currentTimestamp": 1596205376
    },
    {
      "batch": "product-batch-22",
      "currentTimestamp": 1596205376
    },
    {
      "batch": "product-batch-8",
      "currentTimestamp": 1596205376
    }
]

Is this possible using the input/output processing available in aws step functions? I want to have them contained in one array so they can be processed together in an additional map state later in the state machine.

若槻龍太 · Accepted Answer · 2022-09-06 15:09:43Z

17

It is possible. try this.

ASL Definition

{
  "StartAt": "getArrayOfArray",
  "States": {
    "getArrayOfArray": {
      "Type": "Pass",
      "Parameters": {
        "arrayOfArray.$": "States.Array($.array, States.Array($.appendant))"
      },
      "Next": "mergeArray"
    },
    "mergeArray": {
      "Type": "Pass",
      "Parameters": {
        "mergedArray.$": "$.arrayOfArray[*][*]"
      },
      "End": true
    }
  }
}

input

{
  "array": [
    {
      "batch": "product-batch-0",
      "currentTimestamp": 1596205376
    },
    {
      "batch": "product-batch-1",
      "currentTimestamp": 1596205376
    }
  ],
  "appendant": {
    "batch": "size",
    "currentTimestamp": 1596205376
  }
}

output

{
  "mergedArray": [
    {
      "batch": "product-batch-0",
      "currentTimestamp": 1596205376
    },
    {
      "batch": "product-batch-1",
      "currentTimestamp": 1596205376
    },
    {
      "batch": "size",
      "currentTimestamp": 1596205376
    }
  ]
}

answered Sep 6, 2022 at 15:09

若槻龍太

1711 silver badge2 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

this-sam Over a year ago

Can you explain why this syntax results in the final mergedArray? "mergedArray.$": "$.arrayOfArray[*][*]" Are you using the [*][*] syntax to flatten the nested arrays created in the "Pass" state?

若槻龍太 Over a year ago

Exactly. Amazon States Language (ASL) follows JSONPath syntax. ref: aws.amazon.com/blogs/compute/… JSONPath syntax has the wildcard expression. The wildcard enables to retrieve all objects/elements regardless their names. ref: goessner.net/articles/JsonPath/index.html

LePuppy Over a year ago

Thanks a lot for this solution, it solved my current need and will probably solve future needs as well

Daeho Ro · Accepted Answer · 2020-08-01 03:51:02Z

2

You should use another Lambda function to merge the input and output because the step functions output feature cannot append the result into an array or merge the input and output as an array.

[1] : https://docs.aws.amazon.com/step-functions/latest/dg/input-output-resultpath.html#input-output-resultpath-append

answered Aug 1, 2020 at 3:51

Daeho Ro

13.7k4 gold badges25 silver badges50 bronze badges

Comments

ljmc · Accepted Answer · 2023-10-18 09:09:47Z

0

In case you need to do it several times in the step function you can use only one state using array flattening (some_array[*][*]) along with array merging (States.Array(array1, array2)) at the same time:

  "AddItemToArrayState": {
  "Type": "Pass",
  "Next": "NextState,
  "Parameters": {
    "property_with_array.$": "States.Array($.property_with_array[*][*], States.Array('item1', 'item2'))"
  }
},
"NextState": {
  "Type": "Pass",
  "Next": "FinalState,
  "Parameters": {
    "property_with_array.$": "States.Array($.property_with_array[*][*], States.Array('item1', 'item2'))"
  }
},
"FinalState": {
  "Type": "Pass",
  "Parameters": {
    "property_with_array.$": "$.property_with_array[*][*]"
  },
  "End": true
}

edited Oct 18, 2023 at 9:09

ljmc

5,3732 gold badges11 silver badges30 bronze badges

answered Jul 12, 2023 at 13:29

Victor Gomez de Juan

112 bronze badges

Collectives™ on Stack Overflow

Combining AWS step function task and map outputs in to one array

3 Answers 3

3 Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

3 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related