Combining JsonPath listAt and stringAt to make a single array in AWS Steps/CDK

Question

I am invoking an AWS Steps orchestration - and one of the contained steps is to invoke an ECS Fargate Task.

I currently pass some of the Steps' inputs as the "command" of the ECS Task using JsonPath.listAt. I would like to also add a single extra value to the command array - but I cannot work out the syntax as a construct within CDK.

Our step's input is:

{
  "indexes": [
    "s3://bucket/test1.bam",
    "s3://bucket/test2.bam"
  ],
  "reference": "hg38"
}

And the example of us passing the indexes as command arguments to the ECS Task is:

new EcsRunTask(this, "Job", {
      integrationPattern: IntegrationPattern.RUN_JOB,
      cluster: fargateCluster,
      taskDefinition: taskDefinition,
      containerOverrides: [
        {
          command: JsonPath.listAt("$.indexes"),

I want to also pass JsonPath.stringAt("$.reference") in as the first argument of the command but I cannot get the syntax correct.

JsonPath.listAt is the only JsonPath function that returns a string[] (needed by .command in typescript CDK).

JsonPath.array can be used to build arrays - but cannot take in inputs in the same way as listAt (as in it won't flatten out an existing array input).

command: [
  JsonPath.stringAt("$.reference"),
  ...JsonPath.listAt("$.indexes"),
]

fails with

Cannot use JsonPath fields in an array, they must be used in objects

command: JsonPath.array(
            JsonPath.stringAt("$.reference"),
            ...JsonPath.listAt("$.indexes")
          ) as any

fails with

Error: Resolution error: Resolution error: Resolution error: Found an encoded list token string in a scalar string context. Use 'Fn.select(0, list)' (not 'list[0]') to extract elements from token lists..

Yes - so that the command invoke ends up as [ "hg38", "s3://bucket/test1.bam", "s3://bucket/test2.bam" ] — Andrew Patterson
– Andrew Patterson, Commented Nov 8, 2023 at 3:44
Got it. Will the indexes array in the input always have 2 elements? — fedonev
– fedonev, Commented Nov 8, 2023 at 3:46
No, indexes needs to support variable lengths of input array — Andrew Patterson
– Andrew Patterson, Commented Nov 8, 2023 at 3:48

fedonev · Accepted Answer · 2023-11-08 20:35:10Z

2

The least bad option is a two step process: merge then flatten.

First, add a Pass State to merge the inputs. The state will output a nested array: [["hg38"], ["s3://bucket/test1.bam", "s3://bucket/test2.bam"]]:

const merge = new sfn.Pass(this, "merge", {
  parameters: {
    merge: sfn.JsonPath.array(
      sfn.JsonPath.array(sfn.JsonPath.stringAt("$.reference")),
      sfn.JsonPath.stringAt("$.indexes"), // CDK does not accept listAt here; the array param is of type []string
    ),
  },
  resultPath: "$.merge",
});

Now your command can use JSONPath wildcard syntax to flatten the array into the desired shape of ["hg38", "s3://bucket/test1.bam", "s3://bucket/test2.bam"]:

command: sfn.JsonPath.listAt("$.merge.merge[*][*]")

Note that this roundabout approach is needed because of how Step Functions' intrinsic functions work. It's not a limitation imposed by the CDK.

answered Nov 8, 2023 at 20:35

fedonev

26.1k2 gold badges43 silver badges60 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Andrew Patterson Over a year ago

Wish there was an easier way to do with Intrinsic functions - but as you say - I think this is the least bad option. Thanks (have tried this code and is working well)

fedonev Over a year ago

@AndrewPatterson Glad to help. Perhaps AWS will add a native intrinsic push function one day. In the meanwhile, there is a ludicrously complex nested intrinsic function expression that will append to an array in a single step. I didn't mention that alternative, though, because the CDK return type wouldn't match what your command param expects.

Collectives™ on Stack Overflow

Combining JsonPath listAt and stringAt to make a single array in AWS Steps/CDK

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related