Lambda function not displaying correct output after processing SQS message

Question

Currently, I've got a workflow in AWS that goes as follows:

**Step Function -> SQS Queue -> Lambda function **

The Step Function State Machine, when executed, sends a message to an SQS queue which then kicks off a Lambda function based on an SQS trigger. Here's the simple Lambda (written in Python):

# Library Imports
import boto3
import json
import os

# Variables
sqs   = boto3.resource('sqs')
queue_name = 'ExampleStandardQueue'
queue_url = os.environ['QUEUE_URL']
queue = sqs.get_queue_by_name(QueueName=queue_name)

# Handler
def lambda_handler(event, context):

  # Receive messages from queue, one at a time
  messages = queue.receive_messages()

  for message in messages:
    print('Processed message.')
    print('Message Attributes: {0}'.format(message.attributes))
    print('Message Body: {0}'.format(message.body))

When I remove the SQS trigger and send a message to the queue, then test the Lambda function I get the correct output:

START RequestId: fdf3dffe-0271-4b03-974e-9d8762f2b6b3 Version: $LATEST
Processed message.
Message Attributes: None
Message Body: {"MessageTitle":"Create Group","input":"Started."}
END RequestId: fdf3dffe-0271-4b03-974e-9d8762f2b6b3
REPORT RequestId: fdf3dffe-0271-4b03-974e-9d8762f2b6b3  Duration: 60.38 ms  Billed Duration: 
100 ms  Memory Size: 128 MB Max Memory Used: 77 MB  Init Duration: 397.80 ms

However, when I have the automated workflow where the Lambda is kicked off automatically when the State Machine sends a message to the SQS queue, the logs look like this:

START RequestId: 2fac6aa3-01ba-56dc-b077-3c41fdd731ec Version: $LATEST
END RequestId: 2fac6aa3-01ba-56dc-b077-3c41fdd731ec
REPORT RequestId: 2fac6aa3-01ba-56dc-b077-3c41fdd731ec Duration: 294.12 ms Billed Duration: 300 ms Memory Size: 128 MB Max Memory Used: 78 MB

Essentially, here the code outputs don't display or take affect even though the monitoring on the queue shows a message was processed and the lambda function kicked off. What am I missing here?

At a guess, in the automated workflow the queue message is already read, and is passed in the event argument to your handler — Andrew McGuinness
– Andrew McGuinness, Commented Mar 11, 2020 at 15:25

Mark B · Accepted Answer · 2020-03-11 15:27:13Z

4

When you have an SQS trigger defined, the Lambda service will poll SQS for messages, and send those messages into your function in the event parameter. In this scenario you should never be calling messages = queue.receive_messages(). You shouldn't even be creating a Boto3 SQS client like boto3.resource('sqs') when you have an SQS trigger defined.

The reason you are not seeing any messages when you have the SQS trigger defined is that the Lambda service has already removed the message(s) from the queue, and passed them into your function, so then when your function is trying to poll the queue there are no more messages there for it to receive.

answered Mar 11, 2020 at 15:27

Mark B

203k27 gold badges339 silver badges332 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Franchise Over a year ago

Thank you, @Mark B. That makes perfect sense as to why there's a difference when the execution method is changed. As someone who is newish to Lambda, is there any example or documentation you could point me in the right direction with?

Mark B Over a year ago

@Franchise I would start with the official documentation for Lambda integration with SQS here: docs.aws.amazon.com/lambda/latest/dg/with-sqs.html

Collectives™ on Stack Overflow

Lambda function not displaying correct output after processing SQS message

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related