How to sync a variable with multiple processes in python?

Question

I wouldd like to use multiprocessing pool to access, read, and modify a variable in python. The processes are completely separate, but have to access the same variable. This would be easily done with pointers in C, but how do I do it in python?

For example, I have done this:

import multiprocessing as mp
import time
a=3

def one(arg):
        time.sleep(2*arg[1])
        print(arg)
        arg[0]+=1
        print(arg[0])
        return arg[0]

if __name__ == '__main__':
        with mp.Pool(processes=2) as pool:
                print(pool.map(one,[[a,1],[a,2]]))

But it returns

[3, 1]

4

[3, 2]

4

[4, 4]

Instead of [4,5].

The multiprocessing package spawns separate processes. It has an API similar to the threading module, but unlike threads, processes have their own address space. — Adrian W
– Adrian W, Commented May 1, 2019 at 11:02
It depends on what you want to do. If you really want separate processes, you need to use the shared memory features as described in this answer. If you use plain threads, then you only need to synchronize access with a Semaphore. There is also a thread pool in the multiprocessing module. See this question — Adrian W
– Adrian W, Commented May 1, 2019 at 14:05

CodeSamurai777 · Accepted Answer · 2019-05-01 13:26:24Z

2

You can share variables between processes. Using shared memory.

from multiprocessing import Process, Value, Array

def f(n, a):
    n.value = 3.1415927
    for i in range(len(a)):
        a[i] = -a[i]

if __name__ == '__main__':
    num = Value('d', 0.0)
    arr = Array('i', range(10))

    p = Process(target=f, args=(num, arr))
    p.start()
    p.join()

    print(num.value)
    print(arr[:])

Value and Array are special objects that allows you to share state between them. The d and i are typcodes of variable type you want to share.

However, this solution doesn't provide any sync mechanism. You need to be careful not to read and write at the same time.

Also this solution can be used only for simple values. If you need to share object you should look at Manager class in multiprocessing module.

An exert from docs

A manager returned by Manager() will support types list, dict, Namespace, Lock, RLock, Semaphore, BoundedSemaphore, Condition, Event, Barrier, Queue, Value and Array

So as you see you can also share RLock to set up a sync mechanism for reading and writing your shared variables.

More info here: https://docs.python.org/3.5/library/multiprocessing.html#sharing-state-between-processes

edited May 1, 2019 at 13:26

answered May 1, 2019 at 12:25

CodeSamurai777

3,3652 gold badges26 silver badges42 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Aphrodite Over a year ago

Yeah, but I'm using it in selectors in sockets that modify one single variable in the RAM, so I don't know whether it would work here.

Collectives™ on Stack Overflow

How to sync a variable with multiple processes in python?

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related