Shared Array not shared correctly in python multiprocessing

Question

I am experimenting multiprocessing in Python and tried to share an Array of strings among two processes. Here is my python code :

from multiprocessing import Process, Array, Value
import ctypes

def f1(a, v):
    for i, l in enumerate(['a', 'b', 'c']):
        a[i] = l*3

    v.value += 1

    print "f1 : ", a[:], v.value

def f2(a,v):

    v.value += 1

    print "f2 : ", a[:], v.value

if __name__ == '__main__':
    val = Value(ctypes.c_int, 0)
    arr = Array(ctypes.c_char_p, 3)

    print "Before :", arr[:], val.value

    p = Process(target=f1, args=(arr, val))
    p2 = Process(target=f2, args=(arr, val))

    p.start()
    p2.start()

    p.join()
    p2.join()

    print "After : ", arr[:], val.value

When I run the script I see that arr is correctly populated and available in f1() but not in f2(). Here is the result:

    % python /tmp/tests.py
    Before : [None, None, None] 0
    f1 :  ['aaa', 'bbb', 'ccc'] 1
    f2 :  ['\x01', '\x11', '\x01'] 2
    After :  ['\x01', '\x01', '\x01'] 2

Did I overlooked something ?

Thanks in advance for your feedback. :)

jfs · Accepted Answer · 2012-08-07 15:20:11Z

My guess is:

arr stores 3 pointers. f1() assigns them to memory addresses that have no meaning outside current process. f2() tries to access the meaningless addresses that contain junk at this point.

Assigning to values that have meaning in all processes seems to help:

from __future__ import print_function
import ctypes
import time
from multiprocessing import Process, Array, Value

values = [(s*4).encode('ascii') for s in 'abc']

def f1(a, v):
    for i, s in enumerate(values):
        a[i] = s

    v.value += 1

    print("f1 : ", a[:], v.value)

def f2(a,v):
    v.value += 1
    print("f2 : ", a[:], v.value)

def main():
    val = Value(ctypes.c_int, 0)
    arr = Array(ctypes.c_char_p, 3)

    print("Before :", arr[:], val.value)

    p = Process(target=f1, args=(arr, val))
    p2 = Process(target=f2, args=(arr, val))

    p.start()
    p2.start()

    p.join()
    p2.join()

    print("After : ", arr[:], val.value)

if __name__ == '__main__':
    main()

Output

Before : [None, None, None] 0
f1 :  ['aaaa', 'bbbb', 'cccc'] 1
f2 :  ['aaaa', 'bbbb', 'cccc'] 2
After :  ['aaaa', 'bbbb', 'cccc'] 2

lbolla · Accepted Answer · 2012-08-07 15:49:32Z

I don't have a solution, but I can add more hints. I've stripped the script to pinpoint the problem. It looks to me that the problem is in the l * 3 operation. I don't know why, but moving the l * 3 operation in the enumerator works as expected:

from multiprocessing import Process, Array                                   
import ctypes                                                                


def f1(a):                                                                   
#    for i, l in enumerate(['a', 'b', 'c']):                                 
#        a[i] = (l * 3)                                                      
    for i, l in enumerate(['a' * 3, 'b' * 3, 'c' * 3]):                      
        a[i] = l                                                             

    print "f1 : ", map(id, a), a[:]                                          


if __name__ == '__main__':                                                   
    arr = Array(ctypes.c_char_p, 3)                                          

    print "Before :", map(id, arr), arr[:]                                   
    p = Process(target=f1, args=(arr, ))                                     
    p.start()                                                                
    p.join()                                                                 
    print "After : ", map(id, arr), arr[:]

Result:

Before : [3077673516L, 3077673516L, 3077673516L] [None, None, None]
f1 :  [3073497784L, 3073497784L, 3073497784L] ['aaa', 'bbb', 'ccc']
After :  [3073497784L, 3073497784L, 3073497784L] ['aaa', 'bbb', 'ccc']

Collectives™ on Stack Overflow

Shared Array not shared correctly in python multiprocessing

2 Answers 2

Output

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Output

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related