Laravel Queue Batches: How to chain multiple batches

Question

I have a series of batches, that need to be run in a chain (in correct order) and I am trying to find a way to link them all back together and ultimately give the user notification that everything is completed.

I currently am using the following:

Bus::batch(new \App\Jobs\TestBatchJob())
    ->allowFailures()
    ->name('test-batch-1')
    ->then(function(\Illuminate\Bus\Batch $batch) {
        Bus::batch(new \App\Jobs\TestBatchJob2())
            ->allowFailures()
            ->name('test-batch-2')
            ->then(function (\Illuminate\Bus\Batch $batch){
                Bus::batch(new \App\Jobs\TestBatchJob2())
                    ->allowFailures()
                    ->name('test-batch-3')
                    ->then(function (\Illuminate\Bus\Batch $batch){
                        Bus::batch(new \App\Jobs\TestBatchJob2())
                            ->allowFailures()
                            ->name('test-batch-4')
                            ->then(function (\Illuminate\Bus\Batch $batch){
                                // Batch Chain Finishes Here.
                                \Illuminate\Support\Facades\Auth::user()->notify(...);
                            })
                            ->dispatch();
                    })
                    ->dispatch();
            })
            ->dispatch();
    })
    ->dispatch();

This is OK... but the major downside is that it makes multiple batches.

I can no longer interact with the batch i.e. $batch->cancel() and any progress indicator on the frontend no longer works.

I need to ensure the first batch of jobs are completed before starting the second but want to tie all batches together somehow. Something like this (doesn't work) maybe explains my idea.

Bus::batch(new \App\Jobs\TestBatchJob())
    ->allowFailures()
    ->name('test-batch-1')
    ->then(function(\Illuminate\Bus\Batch $batch){
        $batch->add(new \App\Jobs\TestBatchJob2());
    })
    ->dispatch();

Update 1 - Bus::Chain()

Thanks to @boolfalse for the idea here. I tried a few things.

Basic Chain - Doesn't work

You can create batches in a chain, but they are still run in parallel. The bus chain method (with batches) is only useful if you have a batch at the very end of a set of chained jobs.

Note that the chain will only catch failure when creating the batch, not running the jobs inside of it.

Bus::chain([
    function(){
        Bus::batch(new TestBatchJob())->name('batch-1')->dispatch();
    },
    function(){
        Bus::batch(new TestBatchJob())->name('batch-2')->dispatch();
    },
    function(){
        Bus::batch(new TestBatchJob())->name('batch-3')->dispatch();
    },
])->dispatch();

Chain with nested batches

One use case for the chain with nested batches is to allow us to run some normal jobs and then work with the processed data, before running a batch.

To run batches in a chain, we must use the then method on the chain.

$model = Model::create();
Bus::chain([
    new TestBatchJob($model), // Run first on its own
    function() use($model){
        $model->doSomething(); // Amend the model prior to next batch
        Bus::batch(new TestBatchJob($model)) // Run second on its own, using up-to-date model.
            ->name('batch-2')
            ->then(function(){
                Bus::batch(new TestBatchJob())->name('batch-3')->dispatch(); // run third on its own
            })
            ->dispatch();
    }
])->dispatch();

Update 2 - Database Solution

For now, I am going back to my original solution of the nested then()s. To link each child job back to the parent (which we can display on the UI and track as one) I will be adding a column to the database job_batches table to link the child batches.

$batch = Bus::batch(new \App\Jobs\TestBatchJob())
    ->name('test-batch-1')
    ->then(function(Batch $parentBatch) {
        $parent =  $parentBatch->id;
        $child = Bus::batch(new \App\Jobs\TestBatchJob2())
            ->name('test-batch-2')
            ->then(function (Batch $batch) use($parent){
                $child = Bus::batch(new \App\Jobs\TestBatchJob2())
                    ->name('test-batch-3')
                    ->then(function(Batch $batch) use($parent){
                        $child = Bus::batch(new \App\Jobs\TestBatchJob2())
                            ->name('test-batch-3')
                            ->dispatch();
                        JobBatch::findByUuid($parent)->addChild($child);
                    })
                    ->dispatch();
                JobBatch::findByUuid($parent)->addChild($child);
            })
            ->dispatch();
        JobBatch::findByUuid($parent)->addChild($child);
    })
    ->dispatch();

return JobBatch::query()->find($batch->id);

Have you considered laravel.com/docs/9.x/queues#chains-within-batches — ejntaylor
– ejntaylor, Commented Aug 23, 2022 at 17:12

boolfalse · Accepted Answer · 2022-02-21 07:49:02Z

I'll share here my code-snippet, so I think it will be useful:

<?php

// LARAVEL 8 QUEUES
// Official Docs: https://laravel.com/docs/8.x/queues
// Author: https://github.com/boolfalse

// include Laravel8-specific classes
use Illuminate\Bus\Batch;
use Illuminate\Support\Facades\Bus;
use \Throwable;

// include custom classes
use \App\Jobs\FirstBatchJob;
use \App\Jobs\SecondBatchJob;


// defining class with body
// ...

// method for adding Batch Jobs
public function runQueueJobs()
{
    // some initial data
    $data = [
        'first_argument' => 111,
        'second_argument' => 222,
    ];

    // chaining synchronous batches
    $batch = Bus::chain([
        new FirstBatchJob($data['first_argument']), // common job in app/Jobs/FirstBatchJob.php
        $second_batch = function () use ($data) {
            $batch_unique_name = $this->getBatchName($data['second_argument']);

            // CHUNKS USAGE EXAMPLE

            // get ready for starting concurrent job instances
            $chunk_intervals = [];
            for ($i = 0; $i < 100; $i++) {
                // common job in app/Jobs/SecondBatchJob.php
                $chunk_intervals[] = new SecondBatchJob($data['second_argument'], [$i * 1000, ($i + 1) * 1000 - 1]);
            }

            // asynchronous/concurrent batch jobs
            Bus::batch($chunk_intervals)
                ->then(function (Batch $concurrent_batch) use ($data) {
                    info("SecondBatchJob jobs have done.");
                })
                ->catch(function (Batch $concurrent_batch, Throwable $e) use ($data) {
                    info("SecondBatchJob error! - " . $e->getMessage());
                })
                ->finally(function (Batch $concurrent_batch) use ($data) {
                    info("SecondBatchJob batch has finished executing.");
                })
                ->name($batch_unique_name)
                ->dispatch();
        },
    ])->dispatch();

    // response back to the client
    return response()->json([
        'success' => true,
        'message' => "Batch Jobs successfully run.",
    ]);
}

// method for getting (a)synchronous/concurrent batch job(s) info
public function batchInfo(BatchInfoRequest $request)
{
    // getting "batch_unique_name" from client
    $batch_unique_name = $request->get('batch_unique_name');

    // getting appropriate "batch_id" from somewhere, where we've stored that "batch_id"<->"batch_unique_name" relation
    $batch_id = $this->getBatchIdFromUniqueName($batch_unique_name);

    // getting appropriate batch_id
    $batch = Bus::findBatch($batch_id);

    // response back to the client
    if ($batch) {
        return response()->json($batch);
    } else {
        return response()->json([
            'success' => false,
            'message' => "Batch not found!",
        ]);
    }
}

Here's an example of sequential (synchronous) and concurrent (asynchronous) queue jobs. As you can see, there's an array in Bus::chain([ ... ]) so there is sequential (ordered) jobs. So in 2-nd part of that array there's a concurrent jobs usage with chunks (in my code there I had some big data, which needed to separate to chunks), which works as async (concurrent).

At the end you can see a method, which returns info about specific batch job at current moment.

NOTE (for people who may not know): this feature is for Laravel v8 and high.

Thanks. The only thing to note here is that when you respond "Batch Jobs successfully run." back to the client, this is actually only the point where the batch has been created. None of the jobs within the batch have run at this point.

Collectives™ on Stack Overflow

Laravel Queue Batches: How to chain multiple batches

Update 1 - Bus::Chain()

Basic Chain - Doesn't work

Chain with nested batches

Update 2 - Database Solution

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

Update 1 - Bus::Chain()

Basic Chain - Doesn't work

Chain with nested batches

Update 2 - Database Solution

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related