Can't use lock around gather in Python asyncio

Question

For some reason, this code doesn't work on Ubuntu 20, Python 3.8.10, unless the .gather line is commented out. It works on Ubuntu 24, Python 3.12.3 and Windows 11, Python 3.13.9. It doesn't work on Python 3.8/3.9 on the same Ubuntu 24 machine. It seems to work from Python 3.10 onwards, so the issue appears to be one with the Python version.

This code is a minimum working example of the issue that I'm facing. Assume that the locks are actually necessary.

import asyncio
import time

lock = asyncio.Lock()

async def action():
    print("action")
    return

async def task1():
    while True:
        print("1 waiting")
        await lock.acquire()
        print("1 running")
        await asyncio.gather(*[action() for _ in range(1)])
        lock.release()
        print("1 unlocked")

        print("A", time.time())
        await asyncio.sleep(1)
        print("B", time.time())


async def task2():
    while True:
        print("2 waiting")
        await lock.acquire()
        print("2 running")
        lock.release()
        print("2 unlocked")
        await asyncio.sleep(1)


async def main():
    task1_ = asyncio.create_task(task1())
    task2_ = asyncio.create_task(task2())

    while True: await asyncio.sleep(float('inf'))


if __name__ == "__main__":
    asyncio.run(main())

The output on one of the Python versions that works is:

$ python3 test.py
1 waiting
1 running
2 waiting
action
1 unlocked
A 1763606395.8323498
2 running
2 unlocked
B 1763606396.843853
1 waiting
1 running
2 waiting
action
1 unlocked
A 1763606396.8451693
2 running
2 unlocked
B 1763606397.8484037

Notice that task2 gets to run between A and B (while task1 is asleep).

The output of the non-working versions is:

$ python3 test.py
1 waiting
1 running
2 waiting
action
1 unlocked
A 1763606434.4998655
B 1763606435.5010738
1 waiting
1 running
action
1 unlocked
A 1763606435.5020432
B 1763606436.5031264
1 waiting
1 running
action
1 unlocked
A 1763606436.5039277
B 1763606437.5050313

As you can see, task2 doesn't run between A and B.

This issue occurs regardless of using await lock.acquire() or async with lock:. If I comment out the .gather line, it works on all versions. However, I need to run multiple tasks at the same time, so I can't simply comment that out. I haven't been able to find anything in the docs that suggests an issue with gather or locks in older versions of asyncio. So what's going on here, and how do I fix it?

Ilya Egorov · Accepted Answer · 2025-11-20 11:50:09Z

4

The fact that the code works as expected on Python >=3.10 is a side effect of removing the loop parameter.

Prior to Python 3.10, asyncio.Lock was bound to the current event loop at initialization time^[1]. That is, in your case, outside of asyncio.run(). As a result, on Python <3.10, future objects are created on behalf of the wrong event loop^[2], and attempting to use them results in a RuntimeError. Because of this, your second task failed (since the lock was already held by the first task), and the only active task was the first one.

import asyncio

lock = asyncio.Lock()


async def test():
    await lock.acquire()  # actually non-blocking call
    await asyncio.wait_for(lock.acquire(), 1e-3)  # blocking call (deadlock)

    # `RuntimeError` on Python <3.10
    # `TimeoutError` on Python >=3.10


asyncio.run(test())

Starting with Python 3.10, asyncio.Lock binds to the current event loop at the first blocking call^[3], and therefore it creates future objects on behalf of the right event loop. If you want to support Python <3.10, do not create asyncio synchronization primitives at the module level or anywhere else outside the event loop.

The reason why you did not find the real cause of the problem is that you use asyncio.create_task() to create new tasks and do not handle their failures in any way. Exceptions from tasks that no one is waiting for are only logged when they are deleted^[4]^[5]. And since you refer to them in the main() function, they will not be deleted until it finishes its execution (for example, by Control-C).

As for removing the line with asyncio.gather(), this simply means that no context switching occurs between acquiring the lock and releasing it, and thus no task ever sees the lock in the locked state. This is because asyncio implements cooperative multitasking, and you can read more about this either in the documentation or in the answers to a related question.

If you want to work safely with tasks, consider using asyncio.TaskGroup. There is also a backport to Python <3.11.

A quick way to verify the cause is to add the following lines to the beginning of the main() function:

global lock
lock = asyncio.Lock()

With this change, your code works as expected on Python <3.10.

edited yesterday

answered yesterday

Ilya Egorov

5061 gold badge3 silver badges12 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

VPfB yesterday

Nice answer. BTW, asyncio.Queue was doing the same in Python <= 3.9, i.e. creating a loop when instantiated.

WorkerZ yesterday

Thank you. This was a great answer. The only thing I'm a bit confused about is "you use asyncio.create_task() to create new tasks and do not handle their failures in any way". How should I do it properly? Is that what TaskGroup is used for?

Ilya Egorov yesterday

Yes, of course. It raises exceptions for all failed tasks using ExceptionGroup, so you will not lose debugging information when using it. In fact, you should just wait for the tasks you create to complete, but there are pitfalls here, and when using asyncio.gather(), you will only get an exception for the first failed task (all others will be suppressed), so it is better to rely on modern structured concurrency.

Collectives™ on Stack Overflow

Can't use lock around gather in Python asyncio

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related