Laravel queue nested processes

Question

I want to use a queue for file uploads. Users can upload files. Each file will have around 500 rows. Now I want to implement this logic:

Maximum of 5 files can be processed at the same time. The remaining files should be in the queue.
Each file should have 5 processes, so 5 rows will be inserted into databases at the same time. Shortly, there are will be a maximum of 25 processes (5 processes in every 5 files).

Now I am adding all files to one queue. Files processing one by one. Shortly first-come, first out. 2nd file needs to wait to finish 1st file.

How can I implement this? Or do you have any other suggestions?

Eric Landheer · Accepted Answer · 2021-07-14 08:37:24Z

2

What exactly is the difference between processing a file, and inserting rows into the DB?

If you want to run multiple workers for the same queue, you can simply start more workers using php artisan queue:work and additionally use flags to specify the queues --queue=process-files for example. See the documentation.

In a production environment, consider to configure a supervisor to run a specific amount of workers on a queue using numprocs directive.

Do I understand correctly you want to run 25 queue workers per user? That does not seem right. Instead, you should consider creating queues for fast/slow jobs.

edited Jul 14, 2021 at 8:37

answered Jul 14, 2021 at 7:36

Eric Landheer

2,2731 gold badge14 silver badges30 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Laravel queue nested processes

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related