CloudFormation : How to parse mutiples key from secretmanager to container task definition?

Question

I want to define multiple AWS Batch jobs that all use the same environment variables defined in Secrets Manager. I understand CloudFormation does not supports YAML anchors and aliases. Is there a way to define the 'Secrets' configuration as a reusable block?

example: i don't want to define the "Secrets" manytimes for each job. (very long template)

  BatchRCJob01:
    Type: AWS::Batch::JobDefinition
    Properties:
      ...
      EcsProperties:
        TaskProperties:
          - ...
            Containers:
              - Name: TestContainer01
                ...
                Secrets:
                  - Name: APP_MODE_ENV
                    ValueFrom: "arn:aws:secretsmanager:ap-northeast-1:123456789:secret:dev/test-us7Vjm:APP_MODE_ENV::"
                  - Name: APP_API_DATABASE_HOST
                    ValueFrom: "arn:aws:secretsmanager:ap-northeast-1:123456789:secret:dev/test-us7Vjm:APP_API_DATABASE_HOST::"
                  - Name: APP_API_DATABASE_NAME
                    ValueFrom: "arn:aws:secretsmanager:ap-northeast-1:123456789:secret:dev/test-us7Vjm:APP_API_DATABASE_NAME::"
                  - Name: APP_API_DATABASE_PASSWORD
                    ValueFrom: "arn:aws:secretsmanager:ap-northeast-1:123456789:secret:dev/test-us7Vjm:APP_API_DATABASE_PASSWORD::"
                  - Name: APP_API_DATABASE_USERNAME
                    ValueFrom: "arn:aws:secretsmanager:ap-northeast-1:123456789:secret:dev/test-us7Vjm:APP_API_DATABASE_USERNAME::"
                  - Name: KEY_BASE
                    ValueFrom: "arn:aws:secretsmanager:ap-northeast-1:123456789:secret:dev/test-us7Vjm:KEY_BASE::"
                  # and many others secret
                  ...
                DependsOn: []

  BatchRCJob02:
    Type: AWS::Batch::JobDefinition
    Properties:
      ...
      EcsProperties:
        TaskProperties:
          - ...
            Containers:
              - Name: TestContainer02
                ...
                Secrets:
                  - Name: APP_MODE_ENV
                    ValueFrom: "arn:aws:secretsmanager:ap-northeast-1:123456789:secret:dev/test-us7Vjm:APP_MODE_ENV::"
                  - Name: APP_API_DATABASE_HOST
                    ValueFrom: "arn:aws:secretsmanager:ap-northeast-1:123456789:secret:dev/test-us7Vjm:APP_API_DATABASE_HOST::"
                  - Name: APP_API_DATABASE_NAME
                    ValueFrom: "arn:aws:secretsmanager:ap-northeast-1:123456789:secret:dev/test-us7Vjm:APP_API_DATABASE_NAME::"
                  - Name: APP_API_DATABASE_PASSWORD
                    ValueFrom: "arn:aws:secretsmanager:ap-northeast-1:123456789:secret:dev/test-us7Vjm:APP_API_DATABASE_PASSWORD::"
                  - Name: APP_API_DATABASE_USERNAME
                    ValueFrom: "arn:aws:secretsmanager:ap-northeast-1:123456789:secret:dev/test-us7Vjm:APP_API_DATABASE_USERNAME::"
                  - Name: KEY_BASE
                    ValueFrom: "arn:aws:secretsmanager:ap-northeast-1:123456789:secret:dev/test-us7Vjm:KEY_BASE::"
                  # and many others secret
                  ...
                DependsOn: []
...
   # and many others job

The stack deploy error when using yaml anchor/alias definiton

Caldazar · Accepted Answer · 2025-03-05 17:08:41Z

0

The way I solved it in a few projects is to actually pass the whole secret as one to your ECS task and then unpack it as first thing in your task. For example:

Secrets:
  - Name: SECRET_KEY
    ValueFrom: "arn:aws:secretsmanager:ap-northeast-1:123456789:secret:dev/test-us7Vjm"

Then run this as entrypoint:

echo $SECRET_KEY | jq -r 'to_entries|map("\(.key)=\"\(.value|tostring)\"")|.[]' | while read line; do echo "export $line" >>vars.sh; done
. vars.sh
rm vars.sh

In one instance had to run the script without export like this:

echo $SECRET_KEY | jq -r 'to_entries|map("\(.key)=\"\(.value|tostring)\"")|.[]' | while read line; do echo "$line" >>vars.sh; done
. vars.sh
rm vars.sh

answered Mar 5 at 17:08

Caldazar

3,91216 silver badges24 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

luan nguyen Mar 6 at 0:48

Thank you, I will test that. However, is there a CloudFormation approach? The ECS container is maintained by another team, so I don't have permission to modify it's dockerfile.

luan nguyen · Accepted Answer · 2025-03-06 09:35:28Z

Updated : I use Fn::Transform "AWS::Include" to solve it.

#JobDefinition

    TaskProperties:
         Containers:
          - Name: TestContainer01
            Fn::Transform:  # this is the entire of "Secrets"
              Name: "AWS::Include"
              Parameters:
                Location: "s3://xxx/secretfile.yaml"

#secretfile.yaml

Secrets 
 - Name: APP_MODE_ENV
   ValueFrom: "arn:aws:secretsmanager:ap-northeast-1:123456789:secret:dev/test-us7Vjm:APP_MODE_ENV::"
 - Name: APP_API_DATABASE_HOST
   ValueFrom: "arn:aws:secretsmanager:ap-northeast-1:123456789:secret:dev/test-us7Vjm:APP_API_DATABASE_HOST::"
  ...

I got below error, so i needed to parse entire "Secrets" object.

Transform AWS::Include failed with: The specified S3 object's content should be valid Yaml/JSON

Below config did not work.

#JobDefinition
                TaskProperties:
                     Containers:
                      - Name: TestContainer01
                        Secrets:
                           Fn::Transform:  # **parse only value of Secrets**
                           Name: "AWS::Include"
                           Parameters:
                             Location: "s3://xxx/secretfile.yaml"

#secretfile.yaml

- Name: APP_MODE_ENV
  ValueFrom: "arn:aws:secretsmanager:ap-northeast-1:123456789:secret:dev/test-us7Vjm:APP_MODE_ENV::"
- Name: APP_API_DATABASE_HOST
  ValueFrom: "arn:aws:secretsmanager:ap-northeast-1:123456789:secret:dev/test-us7Vjm:APP_API_DATABASE_HOST::"
  ...

Collectives™ on Stack Overflow

CloudFormation : How to parse mutiples key from secretmanager to container task definition?

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related