47

I am trying to access S3 and resources on my VPC from AWS Lambda but since I configured my AWS Lambda to access VPC it's timing out when accessing S3. Here's the code

from __future__ import print_function

import boto3
import logging
import json

print('Loading function')

s3 = boto3.resource('s3')

import urllib

def lambda_handler(event, context):
    logging.getLogger().setLevel(logging.INFO)
    # Get the object from the event and show its content type
    bucket = event['Records'][0]['s3']['bucket']['name']
    key = urllib.unquote_plus(event['Records'][0]['s3']['object']['key']).decode('utf8')
    print('Processing object {} from bucket {}. '.format(key, bucket))
    try:
        response = s3.Object(bucket, key)
        content = json.loads(response.get()['Body'].read())
        # with table.batch_writer() as batch:
        for c in content:
            print('     Processing Item : ID' + str(c['id']))
            # ##################
            # Do custom processing here using VPC resources
            # ##################
    except Exception as e:
        print('Error while processing object {} from bucket {}. '.format(key, bucket))
        print(e)
        raise e

I've set my subnets and security groups with appropriate Outbound rules to access internet as shown below but my Lambda simply times out when accessing S3.

enter image description here

enter image description here

Here's a sample of test input as well

# Test Event Configuration
{
  "Records": [
    {
      "awsRegion": "us-east-1",
      "eventName": "ObjectCreated:Put",
      "eventSource": "aws:s3",
      "eventTime": "2016-02-11T19:11:46.058Z",
      "eventVersion": "2.0",
      "requestParameters": {
        "sourceIPAddress": "54.88.229.196"
      },
      "responseElements": {
        "x-amz-id-2": "ljEg+Y/InHDO8xA9c+iz6DTKKenmTaGE9UzHOAabarRmpDF1z0eUJBdpGi37Z2BU9nbTh4p7oZg=",
        "x-amz-request-id": "3D98A2325EC127C6"
      },
      "s3": {
        "bucket": {
          "arn": "arn:aws:s3:::social-gauge-data",
          "name": "social-gauge-data",
          "ownerIdentity": {
            "principalId": "A1NCXDU7DLYS07"
          }
        },
        "configurationId": "b5540417-a0ac-4ed0-9619-8f27ba949694",
        "object": {
          "eTag": "9c5116c70e8b3628380299e39e0e9d33",
          "key": "posts/test/testdata",
          "sequencer": "0056BCDCF1F544BD71",
          "size": 72120
        },
        "s3SchemaVersion": "1.0"
      },
      "userIdentity": {
        "principalId": "AWS:AROAIUFL6WAMNRLUBLL3K:AWSFirehoseDelivery"
      }
    }
  ]
}

3 Answers 3

68

Once you enable VPC support in Lambda your function no longer has access to anything outside your VPC, which includes S3. With S3 specifically you can use VPC Endpoints to solve this. For pretty much anything else outside your VPC, you would need to create a NAT instance or a managed NAT gateway in your VPC to route traffic from your Lambda functions to endpoints outside of your VPC.

I would read the Lambda VPC support announcement, and pay special attention to the "Things to Know" section at the end.

Sign up to request clarification or add additional context in comments.

4 Comments

Thanks Mark. But I think it was more of a problem on the way my Lambda was performing. This link gave me better understanding of what could happen when Lambda gets invoked and I was able to fix it. aws.amazon.com/blogs/compute/container-reuse-in-lambda
my RDS is publicly available and lambda is not in vpc, still timeout. Any thoughts?
@johnny you will need to allow "all traffic" in your RDS inbound security group setting
@Aaron, basically you create VPC endpoint with service as S3 and vpc as the lambda one. This step alone will ensure that you are able to connect to S3 as this creates a private link to S3 from your vpc.
0

You can use a VPC endpoint to connect to AWS Secrets Manager service within a Virtual Private Cloud. A step-by-step description is found on https://aws.amazon.com/blogs/security/how-to-connect-to-aws-secrets-manager-service-within-a-virtual-private-cloud/

In short:

  1. Go to your VPC and select "Endpoint" from the left pane
  2. Select "AWS Services" and find the "com.amazonaws.us-west-2.secretsmanager"
  3. Choose your VPC, subnet(s) and make sure under "Advanced" that "DNS name" is checked
  4. Associate a security group that gives resources in the VPC access to the Secrets Mgr
  5. Click "Create Endpoint"

The VPC will now use private DNS to resolve secretsmanager.{region}.amazonaws.com

Comments

-3

Try adding an endpoint with AWS Secret service

1 Comment

Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.