Lambda processing same SNS event multiple times?

Question

I have an AWS Lambda function configured to process SNS events from a single topic. When the function runs it will potentially send out some other notifications and call context.succeed, or context.fail if some error occurs. The problem is the same SNS event seems to be invoking the Lambda multiple times. Looking at the CloudWatch logs I see

START RequestId: cd7afdf8-2816-11e6-bca2-6f2e3027c5e1 Version: $LATEST

Which eventually ends

END RequestId: cd7afdf8-2816-11e6-bca2-6f2e3027c5e1 REPORT RequestId: cd7afdf8-2816-11e6-bca2-6f2e3027c5e1 ...

Immediately followed in the same log by a start for the exact same RequestID

START RequestId: cd7afdf8-2816-11e6-bca2-6f2e3027c5e1 Version: $LATEST

Looking into CloudWatch at the topic sending this SNS Event it seems to only be Publishing and Delivering only one even as I had expected, so it seems to be a Lambda-side problem. Does anyone know of any reason an event might be triggering my lambda multiple times like this?

Edit: I've noticed that this seems to be happening when the lambda receives a failure. I don't see any sort of retry configuration on the lambda and wouldn't expect it to be behaving this way by default.

When you say multiple times, is it 2 times or 3 times?

Prasad Paravatha
– Prasad Paravatha

2018-10-27 23:08:25 +00:00
Commented Oct 27, 2018 at 23:08 — Prasad Paravatha
– Prasad Paravatha, Commented Oct 27, 2018 at 23:08

Shibashis · Accepted Answer · 2016-06-02 04:32:16Z

12

From Amazon lambda FAQs page https://aws.amazon.com/lambda/faqs/

Q: What happens if my Lambda function fails during processing an event?

On failure, Lambda functions being invoked synchronously will respond with an exception. Lambda functions being invoked asynchronously are retried at least 3 times, after which the event may be rejected. Events from Amazon Kinesis streams and Amazon DynamoDB streams are retried until the Lambda function succeeds or the data expires. Kinesis and DynamoDB Streams retain data for 24 hours.

edited Jun 2, 2016 at 4:32

answered Jun 1, 2016 at 21:35

Shibashis

8,4613 gold badges31 silver badges39 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Michael - sqlbot Over a year ago

This answer is technically correct, but doesn't seem directly applicable. Lambda functions can be directly invoked by SNS, not using HTTP.

Squirrel Over a year ago

Thank you for finding this. That does appear to be what is happening. It is unfortunately messing up some reporting that I had in place, so I may end up just having to call the invocation a success even in what I would have considered a failure for the message.

keen Over a year ago

much appreciated tracking this detail down - I have event rules (cron style) being triggered three times... the initial times out, and the next two succeed. so "retried at least 3 times" seems to mean "tried at least 3 times".

Blundell Over a year ago

My lambda runs out of time after executing like 99% of its code so the logs looked legit. Upping the lambda timeout fixed my repetitions :-)

Nourdine Alouane · Accepted Answer · 2017-10-06 14:15:20Z

I had a similar issue with a lambda cron job function. I needed to send a request to a service/api that runs for a couple of seconds before it can return the first byte/response (no response processing needed), so my fix was like the following:

exports.handler = (event, context, callback) => {
// TODO implement

var https = require("https");
token = "xxxxxxxx";

var options = {
    host: "api.xxxxx.com",
    path: "/manage/cronjob/video",
    method: "GET",
    headers: {
        "Content-Type": "application/json",
        "Authorization": "Bearer "+token
    }
};

var req = https.request(options);
req.write("");
req.end();

//add timeout for context.done()
setTimeout(function(){
    context.done();
    callback(null, 'Hello from Lambda');
},1000); //This timeout should be lower than the lambda's timeout };

In this is example, we have an explicit callback to force the lambda exit, before the function gets a timeout error & triggers a retry for another 2 times

Hope it helps

Collectives™ on Stack Overflow

Lambda processing same SNS event multiple times?

2 Answers 2

4 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related