EmrServerlessCreateApplicationOperator networkConfiguration with multiple subnetIds

Question

If I pass more than one subnet Id to EmrServerlessCreateApplicationOperator via the networkConfiguration attribute, I receive an error.

If I use a single subnet Id the operator works fine. This is the network configuration and also shown is an extract of the operator:

network_config = {
    'subnetIds': [
        subnet_ids
    ],
    'securityGroupIds': [
        security_group_ids
    ]
}

EmrServerlessCreateApplicationOperator(
    task_id=task_identifier,
    job_type="SPARK",
    release_label="emr-6.6.0",
    config={"name":config_name, "networkConfiguration": network_config}
)

If I try passing more than one subnet Id (via the subnet_ids string value), whether comma separated either with or without quotes, I receive the following error from Airflow:

(An error occurred (ValidationException) when calling the CreateApplication operation: 1 validation error detected: Value at 'networkConfiguration.subnetIds' failed to satisfy constraint: Member must satisfy constraint: [Member must have length less than or equal to 32, Member must have length greater than or equal to 1, Member must satisfy regular expression pattern: [-0-9a-zA-Z]+.*]; 2522)

Does anyone have an example of how to correctly format multiple subnet Ids for the network configuration?

Can you add the full traceback to your question? Have you tried adding the second subnet to the array? 'subnnetIds': [subnet_id1, subnet_id2] — 0x26res
– 0x26res, Commented Jul 13, 2023 at 7:56

singleton · Accepted Answer · 2023-07-13 17:59:21Z

0

As mentioned by @0x26res, if added as an additional item in the array rather than the string, this works as expected.

answered Jul 13, 2023 at 17:59

singleton

1714 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

EmrServerlessCreateApplicationOperator networkConfiguration with multiple subnetIds

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related