How to pass machine input and Lambda function output to map states

Question

Current Setup:

I currently have a Step Functions state machine that kicks off a Task state (which calls a Lambda function), followed by a map state (which submits a job to Batch), defined as follows

State Machine Definition

(note: region and account id have been omitted and substituted for dummy variable ACCOUNT_INFO)

{
  "StartAt": "Populate EFS",
  "States": {
    "Populate EFS": {
      "Next": "MapState",
      "Type": "Task",
      "InputPath": "$",
      "ResultPath": "$.populate_efs_result",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "arn:aws:lambda:{ACCOUNT_INFO}:function:PopulateEFSLambda",
        "Payload.$": "$"
      }
    },
    "MapState": {
      "Type": "Map",
      "End": true,
      "ResultPath": "$.metadata.run_info",
      "InputPath": "$",
      "Iterator": {
        "StartAt": "TaskState",
        "States": {
          "TaskState": {
            "Type": "Task",
            "End": true,
            "InputPath": "$",
            "ResultPath": null,
            "Resource": "arn:aws:states:::batch:submitJob.sync",
            "Parameters": {
              "JobDefinition": "arn:aws:batch:{ACCOUNT_INFO}:job-definition/BatchJobDefCfn:1",
              "JobName": "test",
              "JobQueue": "arn:aws:batch:{ACCOUNT_INFO}:job-queue/BatchQueue123",
              "ContainerOverrides": {
                "Command": [
                  "sh",
                  "-c",
                  "entrypoint.pl -i /NGS/${sequencer}/${run_id}/ -s ${sample_name}"
                ],
                "Environment": [
                  {
                    "Name": "run_id",
                    "Value.$": "$.run_id"
                  },
                  {
                    "Name": "sample_name",
                    "Value.$": "$.sample_name"
                  },
                  {
                    "Name": "sequencer",
                    "Value.$": "$.sequencer"
                  }
                ]
              }
            }
          }
        }
      }
    }
  }
}

State machine input

{
  "metadata": {
    "run_info": [
      {
        "sample_name": "SAMPLE_X",
        "sequencer": "Nextseq"
      },
      {
        "sample_name": "SAMPLE_Y",
        "sequencer": "Nextseq"
      },
      {
        "sample_name": "SAMPLE_Z",
        "sequencer": "Nextseq"
      }
    ]
  }
}

Lambda output (shortened for simplicity)

{"populate_efs_result": {
    "ExecutedVersion": "$LATEST",
    "Payload": "RUN_1"}

Expected Outcome:

The second step (MapState) needs information from the machine input (sample_name and sequencer), as well as what the Lambda function returns in populate_efs_result.Payload (run_id), therefore both need to be included in the event object for the Map state input. However, in my attempts so far, the input for the map state has been either the machine input or the Lambda output, not both.

I've tried changing the InputPath and ItemsPath parameters in the Map state definition and have also tried including the following in the Map state definition, but none of these methods work: Parameters: {"new_run_id.$": "$.populate_efs_result.Payload"}.

Bwyss · Accepted Answer · 2021-09-27 17:23:47Z

1

A simple but not so elegant solution could be to move your lambda step within the Map state. The advantage would be that the response from the lambda would be in the context of the map state (this might be needed if your lambda response if specific to each iteration of the map state). The disadvantage is that your lambda function would need to be executed for each iteration of the map, while lambda function are fast and cheap it is still not a perfect solution.

Another approach would be to pass the execution input to that lambda and then extend the lambda to modify the run_info array to include the required data. Then pass this modified array to the map state as the InputPath

answered Sep 27, 2021 at 17:23

Bwyss

1,8363 gold badges28 silver badges50 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to pass machine input and Lambda function output to map states

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related