Python thread safe asyncio

Question

I am trying to run my sync functions using asyncio.run_in_executor

My coroutines shares data and update data. The run in executor starts threads. Do I need to make my objects thread and asyncio safe?

import asyncio
from concurrent.futures import ThreadPoolExecutor

class shared:
  #complex object

def func(a, b):
    # blocking calls
    return a + b

async def main(loop):
  
    result = await loop.run_in_executor(None, func, "Hello,", " world!")
    UpdateSharedObject(result) 

Start main as 5 tasks using create_task

Is are the func calls using a common instance of shared? — Owen
– Owen, Commented Nov 29, 2020 at 0:54
Also, this is unrelated to your question, but I just want to make sure you're aware that using a thread pool will not let you run purely computation tasks in parallel because of the global interpreter lock. Python threads are pretty much only useful for io bound code. — Owen
– Owen, Commented Nov 29, 2020 at 0:58
Yes. They both share same resource. The blocking call is io bound in nature. — InfoLearner
– InfoLearner, Commented Nov 29, 2020 at 1:00
Do you want to access those shared objects from your blocking function? — Dmitry Belaventsev
– Dmitry Belaventsev, Commented Nov 29, 2020 at 7:55

alex_noname · Accepted Answer · 2020-11-29 15:08:00Z

1

If you are accessing (and especially making changes) to the same resource from different threads. Then I highly recommend using synchronization primitives. Because Python does not provide this thread-safety in all cases. Many Python operations are atomic, but not all. Actually, single opcodes are thread-safe.

Examples of thread-safe operations:

L.append(x)
L1.extend(L2)
x = L[i]
x = L.pop()

These are not:

i = i+1
L.append(L[-1])
L[i] = L[j]
D[x] = D[x] + 1

More info here: https://docs.python.org/3/faq/library.html#what-kinds-of-global-value-mutation-are-thread-safe

As for asyncio, I agree with the @user4815162342 in the comments and advise you to redesign the program so that the functions executed in the threads are pure functions and do not change shared objects. Because the use of thread synchronization, together with cooperative multitasking (asyncio), can greatly complicate the code and lead to subtle bugs.

edited Nov 29, 2020 at 15:08

answered Nov 29, 2020 at 14:57

alex_noname

33.2k6 gold badges95 silver badges110 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Ilya Egorov · Accepted Answer · 2025-01-22 08:37:29Z

0

Since you mentioned in your question comment that the blocking call is I/O bound in nature, I can assume that you may need to be in a critical section for a long time. In this case synchronization primitives from the threading module will not suit you: they will block the event loop. Instead, I suggest using primitives from my aiologic package, such as aiologic.Lock:

lock = aiologic.Lock()

with lock:
    # change the state of a complex object (in a worker thread)

async with lock:
    # change the state of a complex object (in an async task)

There are other kinds of synchronization primitives in aiologic. But if all you need is to make your object thread-safe, you only need to know what primitives can be used as a lock.

edited Jan 22 at 8:37

answered Oct 19, 2024 at 18:25

Ilya Egorov

5061 gold badge3 silver badges12 bronze badges

Collectives™ on Stack Overflow

Python thread safe asyncio

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related