4

I am invoking a lambda from another lambda using python SDK. Both the lambdas belong to the same VPC. The trigger lambda only contains a python script that invokes the second lambda (loader_development) and the code for that is as under:

from __future__ import print_function

import json
import logging
import os
from urllib2 import urlopen,Request,HTTPError
import boto3

logger = logging.getLogger()
logger.setLevel(logging.INFO)
region = os.environ['AWS_REGION']

def lambda_handler(event, context):
    logger.info('starting loading')
    invokeLambda = boto3.client('lambda', region_name = 'us-east-1')  
    request = {'resource':'/bucketstatus/latest','path':'/bucketstatus/latest','httpMethod':'GET', 'requestContext': {'requestId': context.aws_request_id,'identity': {'userArn': context.invoked_function_arn}}}
    invoke_response = invokeLambda.invoke(FunctionName='loader_development',
                                           InvocationType='RequestResponse',
                                           Payload=json.dumps(request))
    print(invoke_response['Payload'].read())

logger.info('Process Complete')

I have run into an issue with the above code. So the trouble that I'm facing now is that this script is triggering the loader_development lambda more than once which is creating redundant entries in my database (for example: There should be only 6 entries in my database but probably due to multiple lambda invocations caused by the script, around 19 entries are getting created).

From what I understood by analyzing the trigger's CloudWatch logs is that it creates a HTTPS connection with the loader lambda and waits for a response. If it doesn't receive a response, it again tries to create a new HTTPS connection. I could see this three times in a single log. I'll paste it here as well.

[DEBUG] 2019-10-25T16:00:05.155Z 0342a98d-1cee-474a-9147-b90c4a804c4c Starting new HTTPS connection (1): lambda.us-east-1.amazonaws.com:443
[DEBUG] 2019-10-25T16:01:05.298Z 0342a98d-1cee-474a-9147-b90c4a804c4c Event needs-retry.lambda.Invoke: calling handler <botocore.retryhandler.RetryHandler object at 0x7f8588337b90>
[DEBUG] 2019-10-25T16:01:05.299Z 0342a98d-1cee-474a-9147-b90c4a804c4c retry needed, retryable exception caught: Read timeout on endpoint URL: "https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/loader_development/invocations"
Traceback (most recent call last):
File "/var/runtime/botocore/retryhandler.py", line 269, in _should_retry
return self._checker(attempt_number, response, caught_exception)
File "/var/runtime/botocore/retryhandler.py", line 317, in __call__
caught_exception)
File "/var/runtime/botocore/retryhandler.py", line 223, in __call__
attempt_number, caught_exception)
File "/var/runtime/botocore/retryhandler.py", line 359, in _check_caught_exception
raise caught_exception
ReadTimeoutError: Read timeout on endpoint URL: "https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/loader_development/invocations"
[DEBUG] 2019-10-25T16:01:05.316Z 0342a98d-1cee-474a-9147-b90c4a804c4c Retry needed, action of: 0.421144501955
[DEBUG] 2019-10-25T16:01:05.316Z 0342a98d-1cee-474a-9147-b90c4a804c4c Response received to retry, sleeping for 0.421144501955 seconds
[DEBUG] 2019-10-25T16:01:05.738Z 0342a98d-1cee-474a-9147-b90c4a804c4c Event request-created.lambda.Invoke: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f8587af6b90>>
[DEBUG] 2019-10-25T16:01:05.738Z 0342a98d-1cee-474a-9147-b90c4a804c4c Event choose-signer.lambda.Invoke: calling handler <function set_operation_specific_signer at 0x7f85887c8398>

The above set of logs appear atleast 3 times under the same log. How do I tell the script to not create a HTTPS connection more than once since this is causing multiple invocations and the loader_development lambda is busy most of the time trying to process older requests and because of that there are times when certain entities are not updated successfully in my database. Please let me know. Thanks!

Update: I added the below solution to my trigger but now "starting loading" is printing thrice so I think this script is getting invoked thrice. Previously only the loader_development lambda was being retried.(FYI: I have a cloudwatch event enabled for my trigger cron(0/30 7-9 * * ? *) so does the cloudwatch event also have a retry mechanism?)

P.S. The below solution works if I don't have a scheduled cloudwatch event to trigger this script.

Improve this question

1 Answer 1

Reset to default
3

The problem is to be blamed on boto3 invoke API inbuilt retry. As per the invoke documentation

When an error occurs, your function may be invoked multiple times. Retry behavior varies by error type, client, event source, and invocation type. For example, if you invoke a function asynchronously and it returns an error, Lambda executes the function up to two more times. For more information, see Retry Behavior .

You can try to implement the following recommendataion from

https://github.com/boto/boto3/issues/1104#issuecomment-385045896

import boto3
import botocore.config

cfg = botocore.config.Config(retries={'max_attempts': 0})
client = boto3.client('lambda', config=cfg)
Improve this answer
Sign up to request clarification or add additional context in comments.

4 Comments

Okay let me try this and get back. Thanks!
Hey! Yes I am 99% sure it did. Just to be extra sure, I have triggered my script again.
Okay this didn't solve my issue. It's still creating 3 HTTPS connections with the lambda. I gave the cfg variable globally and not inside my lambda_handler function. Can that be an issue. Do I need to give cfg inside the function?
https://hackernoon.com/simple-steps-to-avoid-the-retry-behavior-from-aws-lambda-su4w63yx9 gave me the answer to my second question regarding the cloudwatch events.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.