python django race condition with celery

Question

Working on a python django project, here is what I want:

User access Page1 with object argument, function longFunction() of the object is triggered and passed to celery so the page can be returned immediately
If user tries to access Page2 with same object argument, I want the page to hang until object function longFunction() triggered by Page1 is terminated.

So I tried by locking mysql db row with objects.select_for_update() but it doesn't work.

Here is a simplified version of my code:

def Page1(request, arg_id):
    obj = Vm.objects.select_for_update().get(id=arg_id)
    obj.longFunction.delay()
    return render_to_response(...)

def Page2(request, arg_id):
    vm = Vm.objects.select_for_update().get(id=arg_id)
    return render_to_response(...)

I want that Page2 hangs at the line vm = Vm.objects.select_for_update().get(id=arg_id) until longFunction() is completed. I'm new to celery and it looks like the mysql connection initiated on Page1 is lost when the Page1 returns, even if longFunction() is not finished.

Is there another way I can achieve that?

Thanks

Mikel Emaldi Manrique · Accepted Answer · 2013-11-20 17:03:45Z

1

Maybe this can be helpul for you:

from celery.result import AsyncResult
from yourapp.celery import app

def Page1(request, arg_id):
    obj = Vm.objects.select_for_update().get(id=arg_id)
    celery_task_id = obj.longFunction.delay()
    return render_to_response(...)

def Page2(request, arg_id, celery_task_id):
    task = AsyncResult(app=app, id=celery_task_id)
    state = task.state
    while state != "SUCCESFUL":
        # wait or do whatever you want
    vm = Vm.objects.select_for_update().get(id=arg_id)
    return render_to_response(...)

More info at http://docs.celeryproject.org/en/latest/reference/celery.states.html

answered Nov 20, 2013 at 17:03

Mikel Emaldi Manrique

6336 silver badges12 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

asksol Over a year ago

Note that it's better to wait in the frontend, you can take a look at jobtastic for example: policystat.github.io/jobtastic which makes this simpler.

joshua · Accepted Answer · 2013-11-20 20:42:01Z

0

The database lock from select_for_update is released when the transaction closes (in page 1). This lock doesn't get carried to the celery task. You can lock in the celery task but that won't solve your problem because page 2 might get loaded before the celery task obtains the lock.

Mikel's answer will work. You could also put a lock in the cache as described in the celery cookbook.

answered Nov 20, 2013 at 20:42

joshua

2,51717 silver badges19 bronze badges

Collectives™ on Stack Overflow

python django race condition with celery

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related