13

I have a single server running on rackspace which is hosting a single PHP web app.

The PHP web app will accept a form submission that then needs to perform a task based on the form field entries.

The task (let's called it the generate metadata task) requires quite a lot of processing time. I was wondering how to allow the form submission to be a straightforward save to database and immediately show success page to user while running the generate metadata task in the background.

I have installed "aws/aws-sdk-php": "~3.11" using composer into the same web app.

My plan is initially this:

code that handles the form submission

$result = $model->save($_POST);
// this code will send the information to either SQS or SNS
$awsClient->sendsMessage($_POST);
if ($result) {
  $this->redirect('success.html');
}

I have read about the fanout architecture stated by AWS.

My issues with the fanout architecture example (as I understand it) are this:

  1. the server that sends the message to either SQS or SNS will also be the same server that processes the generate metadata task. In fact, it's the same web app.
  2. SQS fulfils the queue part (because I want to execute the tasks in a FIFO and the tasks do take a long time to fulfil). However, it requires my web app to poll SQS continuously. I want a push notification (from AWS to my web app) rather than my web app polling AWS continuously to check for tasks to execute.

I found a possible solution suggested here

The suggested solution is:

  1. send the message to a SNS topic.

  2. The SNS topic will message both a SQS queue and my web app.

  3. My web app, after being triggered, will poll the same SQS queue that has now queued the message continuously until the queue is empty

The drawback I see from this is that my web app will poll the queue before the queue itself has the message.

What is the best way to implement push queues using AWS services?

Improve this question

1 Answer 1

Reset to default
41
+50

my web app will poll the queue before the queue itself has the message

You haven't tried it, then, right? :) I'm afraid you're overthinking this. SQS has long polling, which causes the poll request to be suspended at the SQS side until at least one message is available, at which point that message (up to the max number you requested) will be returned. You can set the long poll wait time from 1 to 20 seconds. If no messages are available within this time frame, the response is returned with no messages.

If you poll the queue in response to the notification from SNS, you will find messages there if you use long polling. It's possible for messages to be delayed, but highly unlikely.

The other problem, though, is your assertion that you don't want the app to be constantly polling SQS. I encounter this objection fairly often, and it's often misplaced. With SQS long polling, "constantly" polling an empty queue means one request every 20 seconds. That's 3 req/minute, 180 req/hour, 4320 req/day, 129600 req/month... which turns out to be less than the 1 million free requests allowed each month.

The problem with your server reacting to notifications rather than polling a queue in the background with an appropriate number of workers is that you'll potentially be easily overwhelmed by a large batch of jobs arriving at about the same time. If you get 10 concurrent requests, can you handle it? 100? 1000? Often, for jobs that are asynchronous like this, it costs less (in terms of resources) to request the job than it does to perform the job (e.g., uploading an image should require much less CPU than resizing that image would require). Unless you coordinate your reaction response, you could overwhelm your system.

Don't fall into a conceptual trap of "polling is bad, push is good" where it doesn't apply. The vast majority of the time, that sentiment is absolutely correct... polling is almost always the wrong solution... but with SQS long polling, what you really have is a push mechanism wrapped up in a way that makes it compatible with HTTP, and much of the inherent evil of polling... disappears. If you're in the middle of a long poll, the queue is empty, and a message arrives, your long poll will return with that message almost immediately. It doesn't sit around waiting the timeout to happen. A background process watching the queue may be a good way to go after all.

Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

Gee thanks. I didn't think of it like that. Awesome answer! I give you extra 50 points for that when the bounty can be opened tomorrow.
Yes...it's overwhelming my system...throttling the dynamo...I wish the person who designed the system I 'am currently working on used polling instead or SQS EBworker marriage

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.