2

I'm making a website crawler using Laravel 5.7. The queue is on Redis driver and I'm using laravel horizon to run the job. I want only one job to run or queued at the same time, however it sometimes duplicates a job and ends up running multiple instances of the same job simultaneously. Is there any way to guarantee that only one job is running at a time?

Job class

class MyJob implements ShouldQueue

{ use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

public $timeout = 30000;

public $tries = 1;

public function __construct($jobid = null)
{
    if (!$jobid) {
        $this->ra_jobid = substr(str_shuffle('1234567890abcdefghijklmnopqrstuvwxyz'), 0, 8);
    } else {
        $this->ra_jobid = $jobid;
    }
}

public function handle()
{
    Redis::funnel('RetrieveAucfanData')->limit(1)->then(function () {
        if (Redis::get('stop_job' . $this->ra_jobid)) {
            Redis::del('stop_job' . $this->ra_jobid);
            throw new JobManuallyStoppedException();
        }

        //crawl a website

    }, function () {
        return $this->release(60);
    });
}

public function failed(\Exception $exception)
{
    if ($exception->getMessage() == 'Job Manually Stopped') {
        Log::debug($exception->getMessage());
    } else {
        self::dispatch($this->ra_jobid)
            ->delay(now()->addSeconds(60));
    }

}

}

I want to queue this job again if the job failed unless it's manually stopped. I also want to put the job back to the queue if it hit the limit.

queue.php

<?php

return [

/*
|--------------------------------------------------------------------------
| Default Queue Driver
|--------------------------------------------------------------------------
|
| Laravel's queue API supports an assortment of back-ends via a single
| API, giving you convenient access to each back-end using the same
| syntax for each one. Here you may set the default queue driver.
|
| Supported: "sync", "database", "beanstalkd", "sqs", "redis", "null"
|
*/

'default' => env('QUEUE_DRIVER', 'redis'),

/*
|--------------------------------------------------------------------------
| Queue Connections
|--------------------------------------------------------------------------
|
| Here you may configure the connection information for each server that
| is used by your application. A default configuration has been added
| for each back-end shipped with Laravel. You are free to add more.
|
*/

'connections' => [

    'sync' => [
        'driver' => 'sync',
    ],

    'database' => [
        'driver' => 'database',
        'table' => 'jobs',
        'queue' => 'default',
        'retry_after' => 30,
    ],

    'beanstalkd' => [
        'driver' => 'beanstalkd',
        'host' => 'localhost',
        'queue' => 'default',
        'retry_after' => 30,
    ],

    'sqs' => [
        'driver' => 'sqs',
        'key' => 'your-public-key',
        'secret' => 'your-secret-key',
        'prefix' => 'https://sqs.us-east-1.amazonaws.com/your-account-id',
        'queue' => 'your-queue-name',
        'region' => 'us-east-1',
    ],

    'redis' => [
        'driver' => 'redis',
        'connection' => 'default',
        'queue' => 'default',
        'retry_after' => 300,
        'block_for' => 10,
    ],

],

/*
|--------------------------------------------------------------------------
| Failed Queue Jobs
|--------------------------------------------------------------------------
|
| These options configure the behavior of failed queue job logging so you
| can control which database and table are used to store the jobs that
| have failed. You may change them to any database / table you wish.
|
*/

'failed' => [
    'database' => env('DB_CONNECTION', 'mysql'),
    'table' => 'failed_jobs',
],

];

1

1 Answer 1

3

Your timeout value is more than the retry_after value, due to which two or more workers are processing the same job.

I have added the snapshot of the documentation here.

Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.