0

When using PHP and a Message Queue (MQ) system like RabbitMQ, Beanstalkd, etc.

Adding jobs to the queue seems simple enough (often a HTTP request).

But to process the queued jobs, most examples involve a PHP script with a while loop, where a connection is established to the MQ server, and this eventually responds with a job (either immediately, if one is already waiting, or the connection will wait until a job is added).

Unfortunately PHP is not ideal for long running jobs, it will often hit different timeouts, and memory management isn't exactly a strong point (it's fairly easy to create code that does not clean up its memory after completing each job).

So I'm wondering if there is a better way to wait for the next job to be made available (potentially waiting many hours), and then it starts the execution of the PHP script as needed (maybe the stdout from the script notes if it was successful or not).

It might be as simple as a bash script?

It could use curl to collect the job (example), and when it has a response, it starts/runs the php script.

I suspect it will need to have a lock, so multiple instances of it cannot be run at the same time, maybe using systemd or a cron job (every 15 min?) to restart it if anything was to fail (or server restart).

I could write this, but would rather use something that's known to work well already.

Improve this question
6
  • 1
    Honestly, we use thousands of PHP scripts to consume messages from hundreds of RabbitMQ queues and don't run into any of the issues you've mentioned. We use supervisord to run them. If you're worried about memory leaks, you can check memory usage after each message, and exit the script. Supervisord simply restarts it again. Commented Dec 15, 2022 at 9:54
  • It's more todo with the timeouts, it's a very low volume system, waiting many hours between jobs (e.g. even the connection to the database will almost certainly have closed). Commented Dec 15, 2022 at 9:58
  • @CraigFrancis what you infer here in real life doesn't happen, or if its happening is just a immediate retry. We use PHP scripts which have uptime more than 100 days, or even more if there is no security upgrade meanwhile. It's not a problem at all what you infer. Commented Dec 31, 2022 at 23:37
  • @Pentium10, my test script had this issue happen a couple of times a day, and it wasn't handled gracefully... maybe it's because I'm using a separate database server, so using a network connection instead of localhost/socket? or you're querying the database often enough to keep the connection alive? Commented Jan 1, 2023 at 12:16
  • establish the connection outside of waiting phase, just when you process Commented Jan 1, 2023 at 19:14

1 Answer 1

Reset to default
0

Check out Roadrunner PHP application server. It supports long runninug PHP scripts for exactly the reasons you outlined.

It has out of the box integration with many existing queueing systems including RabbitMQ and Kafka.

Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

Thanks, that looks interesting... I'll give it a try, and might make a simpler version (been wanting to try writing some Go code, where I'd rather have a simple program do one thing, than have something with loads of features I don't need).
Then consider looking into temporal.io which Roadrunner integrates with as well. It eliminates the need for a queue for the large percentage of use cases.
Thanks @maxim-fateev, that also looks interesting, will add that to my list of things to try.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.