AWS Lambda Execution Failing with AWS Common Runtime Error [duplicate]

Question

I'm attempting to deploy a lambda function using python and the boto3 dependencies to interact with a CloudFront key store. I'm getting an error I don't quite understand suggesting I don't have botocore[crt] available in the lambda environment but as far as I can tell I do.

Error: Missing Dependency: This operation requires an additional dependency. Use pip install botocore[crt] before proceeding.

I'm following this set of instructions: https://docs.aws.amazon.com/lambda/latest/dg/python-package.html#python-package-create-dependencies with one small exception: instead of pip install --target ./package boto3 I'm using pip install --target ./package -r requirements.txt. I'm on a mac with an m1 chip using python 3.11.9. I've tried the arm64 and x86_64 environments.

Sample code:

import json
import boto3

cf_kvs_client = boto3.client("cloudfront-keyvaluestore")
cf_client = boto3.client("cloudfront")
KVS_ARN = "blah"

def lambda_handler(event, context):
    return fetch_current_values()

def fetch_current_values() -> str:
    current_values, etag = get_keys()
    return json.dumps({"etag": etag, "versions": current_values})

def get_keys() -> tuple[str, str]:
    kvs_etag = cf_kvs_client.describe_key_value_store(KvsARN=KVS_ARN)["ETag"]
    kvs_response = cf_kvs_client.list_keys(KvsARN=KVS_ARN)["Items"]
    return kvs_response, kvs_etag

requirements.txt

boto3[crt]==1.35.62
botocore[crt]==1.35.62
jmespath==1.0.1
python-dateutil==2.9.0.post0
s3transfer==0.10.3
six==1.16.0
urllib3==2.2.3

Fedi Bounouh · Accepted Answer · 2024-11-16 11:34:34Z

Having a MAC OS and trying to create packages for your Lambda end up mostly with these compatibility issues. to make sure that it's not a package issue try to remove the boto3 and botocore from your requirement and use a layer with the correct packages. And if that works, then it's for sure platform issue when installing the packages.

And since lambda uses linux (Amazon Linux), and you're on Mac, it's better to use a Docker image compatible with the Lambda runtime when you want to create a package with

FROM public.ecr.aws/lambda/python:3.11
RUN pip install --target /var/task -r requirements.txt

add --platform=linux/arm64 when you build it, in order to match the lambda runtime

Solution

I installed the botocore[crt] and boto3[crt] using the aws cloudshell (because aws cloudshell platform matches well with the Lambda platform x86_64) and it worked well. here's the steps:

mkdir -p boto3_crt_layer/python/lib/python3.11/site-packages
cd boto3_crt_layer

then installing the dependencies:

pip install boto3[crt] botocore[crt] --target python/lib/python3.11/site-packages

update python3.11 with your python version, and update the python version in the cloudshell as well, the default one is python3.9

then zip it:

zip -r boto3_crt_layer.zip python/

and lastly push it to the lambda layers:

aws lambda publish-layer-version \
  --layer-name boto3-crt-layer \
  --description "Boto3 CRT and Botocore CRT Layer with Correct Path" \
  --compatible-runtimes python3.11 \
  --zip-file fileb://boto3_crt_layer.zip

again, update the compatible-runtimes to match your python version

just upload your code to a new lambda and test it with the uploaded layer.

i tested it using this code:

import json
import boto3
import botocore
import importlib.util

def lambda_handler(event, context):
    # Check if CRT is available
    crt_available = check_crt_availability()

    # Simulate CloudFront Key-Value Store interaction
    try:
        # Create clients (CloudFront Key-Value Store and CloudFront)
        cf_kvs_client = boto3.client("cloudfront-keyvaluestore")
        cf_client = boto3.client("cloudfront")

        # Simulated ARN (replace "blah" with your actual ARN if available later)
        KVS_ARN = "arn:aws:cloudfront::123456789012:key-value-store/fake-store"

        # Simulate fetching values (no actual API call due to placeholder ARN)
        simulated_values = fetch_current_values(cf_kvs_client, KVS_ARN)
        
        return {
            "statusCode": 200,
            "body": json.dumps({
                "crt_available": crt_available,
                "simulated_fetch": simulated_values
            })
        }

    except Exception as e:
        return {
            "statusCode": 500,
            "body": json.dumps({
                "message": "An error occurred",
                "error": str(e)
            })
        }

def fetch_current_values(cf_kvs_client, kvs_arn) -> str:
    # Simulate fetching current values from KVS
    try:
        # Simulate describe_key_value_store (would normally return ETag)
        kvs_etag = "fake-etag"  # Placeholder

        # Simulate list_keys (would normally return key-value items)
        kvs_response = ["key1", "key2"]  # Placeholder for actual KVS keys
        
        return {"etag": kvs_etag, "versions": kvs_response}
    except Exception as e:
        return {"error": str(e)}

def check_crt_availability():
    """Check if CRT module is available and being used by boto3."""
    try:
        # Use importlib to check if awscrt is available
        awscrt_spec = importlib.util.find_spec("awscrt")
        if awscrt_spec:
            return True
    except Exception as e:
        return False
    return False

and it worked with:

{
  "statusCode": 200,
  "body": "{\"crt_available\": true, \"simulated_fetch\": {\"etag\": \"fake-etag\", \"versions\": [\"key1\", \"key2\"]}}"
}

when i remove the layer, the result is :

{
  "statusCode": 200,
  "body": "{\"crt_available\": **false**, \"simulated_fetch\": {\"etag\": \"fake-etag\", \"versions\": [\"key1\", \"key2\"]}}"
}

So that conclude that the issue can be with platform Mac OS incompatibility with the Lambda platform

And as I saw in another question, try to add V4 to the config beforehand:

from botocore.config import Config
my_config = Config(
        signature_version='v4',
    )
cloudfront_keyvaluestore_client = boto3.client('cloudfront-keyvaluestore', config=my_config, region_name='us-east-1')

Potentially I'm missing something but I don't see an option for the AWS-provided boto3 layer. - LambdalnsightsExtension-Arm64 - AWS-AppConfig-Extension-Arm64 - AWS-Parameters-and-Secrets-Lambda-Extension-Arm64 - AWSSDKPandas-Python311-Arm64 - AWSLambdaPowertoolsPythonV2-Arm64 - AWSLambdaPowertoolsPythonV3-python311-x86_64 - AWSLambdaPowertoolsPythonV3-python311-arm64

Collectives™ on Stack Overflow

AWS Lambda Execution Failing with AWS Common Runtime Error [duplicate]

1 Answer 1

2 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Linked

Related