38

I have an AWS Lambda function, configured with only 128MB of memory, is triggered by SNS (which is itself triggered by S3) and will download the file from S3.

In my function, I have the following:

public class LambdaHandler {

    private final AmazonS3Client s3Client = new AmazonS3Client();

    public void gdeltHandler(SNSEvent event, Context context) {
        System.out.println("Starting");
        System.out.println("Found " + eventFiles.size() + " event files");
    }

I've commented out and excluded from this post all of the logic because I am getting an OutOfMemoryError which I have isolated to the creation of the AmazonS3Client object. When I take that object out, I don't get the error. The exact above code results in the OutOfMemoryError.

I assigned 128MB of memory to the function, is that really not enough to simply grab the credentials and instantiate the AmazonS3Client object?

I've tried giving the AmazonS3Client constructor

new EnvironmentVariableCredentialsProvider()

as well as

new InstanceProfileCredentialsProvider()

with similar results.

Does the creation of the AmazonS3Client object simply require more memory?

Below is the stack trace:

Metaspace: java.lang.OutOfMemoryError java.lang.OutOfMemoryError: Metaspace at com.fasterxml.jackson.databind.deser.BeanDeserializerBuilder.build(BeanDeserializerBuilder.java:347) at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.buildBeanDeserializer(BeanDeserializerFactory.java:242) at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.createBeanDeserializer(BeanDeserializerFactory.java:143) at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:409) at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:358) at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:265) at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:245) at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:143) at com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:439) at com.fasterxml.jackson.databind.ObjectReader._prefetchRootDeserializer(ObjectReader.java:1588) at com.fasterxml.jackson.databind.ObjectReader.(ObjectReader.java:185) at com.fasterxml.jackson.databind.ObjectMapper._newReader(ObjectMapper.java:558) at com.fasterxml.jackson.databind.ObjectMapper.reader(ObjectMapper.java:3108)

When I try providing the InstanceProfileCredentialsProvider or EnvironmentVariableCredentialsProvider, I get the following stack trace:

Exception in thread "main" java.lang.Error: java.lang.OutOfMemoryError: Metaspace at lambdainternal.AWSLambda.(AWSLambda.java:62) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at lambdainternal.LambdaRTEntry.main(LambdaRTEntry.java:94) Caused by: java.lang.OutOfMemoryError: Metaspace at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) at java.net.URLClassLoader.access$100(URLClassLoader.java:73) at java.net.URLClassLoader$1.run(URLClassLoader.java:368) at java.net.URLClassLoader$1.run(URLClassLoader.java:362) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:361) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at lambdainternal.EventHandlerLoader$PojoMethodRequestHandler.makeRequestHandler(EventHandlerLoader.java:421) at lambdainternal.EventHandlerLoader.getTwoLengthHandler(EventHandlerLoader.java:777) at lambdainternal.EventHandlerLoader.getHandlerFromOverload(EventHandlerLoader.java:802) at lambdainternal.EventHandlerLoader.loadEventPojoHandler(EventHandlerLoader.java:888) at lambdainternal.EventHandlerLoader.loadEventHandler(EventHandlerLoader.java:740) at lambdainternal.AWSLambda.findUserMethodsImmediate(AWSLambda.java:126) at lambdainternal.AWSLambda.findUserMethods(AWSLambda.java:71) at lambdainternal.AWSLambda.startRuntime(AWSLambda.java:219) at lambdainternal.AWSLambda.(AWSLambda.java:60) ... 3 more START RequestId: 58837136-483e-11e6-9ed3-39246839616a Version: $LATEST END RequestId: 58837136-483e-11e6-9ed3-39246839616a REPORT RequestId: 58837136-483e-11e6-9ed3-39246839616a Duration: 15002.92 ms Billed Duration: 15000 ms Memory Size: 128 MB Max Memory Used: 50 MB
2016-07-12T14:40:28.048Z 58837136-483e-11e6-9ed3-39246839616a Task timed out after 15.00 seconds

EDIT 1 If I increase the memory allocated to the function to even 192MB, it works just fine, though strangely enough, reports only using 59MB of memory in the cloudwatch logs. Am I simply losing the rest of the memory?

Improve this question
3
  • 1
    Did you find a solution for this? I think the meta memory part is overloaded due to the class loading that jackson causes. The metaspace is a percentage of the total memory, so if you increase the totalt jvm memory the metaspace will get more memory to work with before it throws a OutOfMemoryError. Would be nice if it was possible to just increase the metaspace part of the memory. (-XX:MaxMetaspaceSize=512m) Another solution might be to tweak jackson if it is possible? Explanation of metaspace: plumbr.eu/outofmemoryerror/metaspace Commented Feb 22, 2017 at 9:38
  • No solution that I'm aware of... Commented Feb 22, 2017 at 16:37
  • FWIW, it looks like one can pass the JAVA_TOOL_OPTIONS environment variable to a Java 8 lambda Commented May 14, 2019 at 22:52

4 Answers 4

Reset to default
27

I have been observing this when using AWS Java SDK within the Lambda function. It would seem when creating any of the AWS clients (Sync or Async) you may get out of Metaspace.

I believe this is due to things that the Amazon Client is performing upon instantiation, including AmazonHttpClient creation as well as dynamic loading of request handler chains (part of AmazonEc2Client#init() private method).

It is possible that the reported memory usage is for Heap itself, but may not include Metaspace. There are a few threads on AWS Forums but no responses from AWS on the matter.

Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

Another thing I noticed is that with allocation of 192MB, it works just fine, but while the first Lambda execution takes more than 10 seconds to spin up and instantiate the AmazonS3Client, subsequent executions within a 1 hour timeframe take less than 1 second to instantiate the AmazonS3Client...assuming it caches the authentication? After 1 hour, however, it seems to drop the cache and re-authenticate taking again, greater than 10 seconds.
Yes, there seems to be some keep-alive mechanism in-place. I did not have time to dig further into this but I'm looking into a way to lower the memory footprint. Probably by removing some dependencies that are not actually used even though are not marked as "optional".
I'm experiencing the same issue but with the SNS client. Reported memory usage is well below 128MB but I still get "OutOfMemoryError: Metaspace". Bumping up to 192MB seems to have "solved" the problem but would be good to know what is really going on here.
7

One way to reduce cold start is setting the memory to 1536 mb and the timeout to 15 min. This will give dedicated host to run only your lambda instead of running your lambda on shared host + when a new instance has to be started, it will copy the code from cache on the host rather than copying from S3.

This though will be more expensive and if you don't want to do this, continue reading below.

How can I reduce my cold start times?

  1. Follow the Lambda best practices
    https://docs.aws.amazon.com/lambda/latest/dg/best-practices.html

  2. By choosing a larger memory setting for your function
    Think of the memory as a "power" setting because it also dictates how much CPU your function will receive.

  3. By reducing the size of your function ZIP
    This likely means reducing the number of dependencies you include in your function ZIP. Java JARs can be further reduced in size using ProGuard

  4. [Java Only] Use the bytestream interface instead of the POJO interface.
    The JSON serialization libraries that Lambda uses internally can take some time to start. It will take dev work on your end, but you may be able to improve on this by using the byte stream interface along with a lightweight JSON library. Here are some links that may help: http://docs.aws.amazon.com/lambda/latest/dg/java-handler-io-type-stream.html https://github.com/FasterXML/jackson-jr

  5. [Java Only] Don't use Java 8 feature that replaces anonymous classes (lambdas, method references, constructor references, etc.)
    We've noticed internally that Java 8 Lambda-related bytecode appears to result in sub-optimal startup performance. If your code is using any Java 8 feature that replaces anonymous classes (lambdas, method references, constructor references, etc.) you may get better startup time by moving back to anonymous classes.

  6. By using a different runtime
    Different runtimes have different cold start times, and different runtime performance. While NodeJS might be better for heavy IO work, Go might be better for code that does a lot of concurrent work. Customers have done some basic benchmarks to compare language performance on Lambda, and here is a more generic comparison of different programming languages performance. There is no one-size-fits-all answer, use what makes sense for your requirements.

basic benchmarks:https://read.acloud.guru/comparing-aws-lambda-performance-of-node-js-python-java-c-and-go-29c1163c2581

generic comparison : https://benchmarksgame-team.pages.debian.net/benchmarksgame/which-programs-are-fast.html

Improve this answer

Comments

6

Try to increase the memory allocated to lambda from 128 to 256 MB

Improve this answer

1 Comment

Please note the edit I made to the original question. Increasing the memory fixed it but begged a greater question.
3

I use a tactic that helps for Java-based lambdas. Any class resources that only need a single (reusable) instance can be declared as static class members, and initialized inside a static initializer block. When the lambda creates a new instance of the class to handle an execution, those expensive resources are already initialized. Here is a simple example:

package com.mydomain.myapp.lambda.sqs;

import com.amazonaws.services.lambda.runtime.events.SQSEvent;
import com.amazonaws.services.sns.AmazonSNS;
import com.amazonaws.services.sns.AmazonSNSClientBuilder;
import com.amazonaws.services.sqs.AmazonSQS;
import com.amazonaws.services.sqs.AmazonSQSClientBuilder;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.sql.Connection;
import java.sql.SQLException;
import java.util.Objects;

public class MyLambdaFunctionHandler {

    private static final Logger LOGGER = LoggerFactory.getLogger(MyLambdaFunctionHandler.class);

    // These values come from the 'Environment' property for the lambda, defined in template.yaml
    private static final String ENV_NAME = System.getenv("ENV_NAME");

    // Declare these as static properties so they only need to be created once,
    // rather than on each invocation of the lambda handler method that uses them
    private static final ObjectMapper OBJECT_MAPPER;
    private static final AmazonSNS SNS;
    private static final AmazonSQS SQS;

    static {
        LOGGER.info("static initializer | START");
        Objects.requireNonNull(ENV_NAME, "ENV_NAME cannot be null");
        OBJECT_MAPPER = new ObjectMapper();
        SNS = AmazonSNSClientBuilder.defaultClient();
        SQS = AmazonSQSClientBuilder.defaultClient();
        LOGGER.info("static initializer | END");
    }

    public MyLambdaFunctionHandler() {
        LOGGER.info("constructor invoked");
    }

    public void handlerMethod(SQSEvent event) {
        LOGGER.info("Received SQSEvent with {} messages", event.getRecords().size());
        event.getRecords().forEach(message -> handleOneSQSMessage(message));
    }

    private void handleOneSQSMessage(SQSEvent.SQSMessage message) {
        // your SQS message handling code here...
    }

}

The properties I declared as static will stay in memory until the lambda instance is destroyed by AWS.

This isn't how I would normally write Java code. Lambda-based code is treated differently, so I think it is OK to break some traditional patterns here.

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.