I have two objects
shared_array = numpy.zeros(20)
and
shared_matrix = scipy.sparse.csr_matrix((data, (row, col)),(20,20),dtype=numpy.float64)
which I want to be accessible from all processes created with
multiprocessing.Process(target = random_function, args = (shared_matrix, shared_array))
How is this done?
If you just want to access it, you can. You can read data from global variables across all processes created by multiprocessing.
However if you want to write into for example a dictionary (with caution not to overwrite) you can use shared memory objects. multiprocessing has a built in manager from where you can call all the shared memory objects such as lists, dicts.
You pass all your objects in arguments, so it will be available for all of the processes, however if you make changes inside the object it wont we permanent, as it would not be permanent with a simple function.
Simple example for a numpy array:
import numpy as np
import multiprocessing
a = np.zeros(5)
def func(b):
ns.array += b
return
manager = multiprocessing.Manager()
ns = manager.Namespace()
ns.array = np.zeros(5)
pool = multiprocessing.Pool(2)
list(pool.imap_unordered(func, [1,2]))
print(ns.array)
Will output [ 3. 3. 3. 3. 3.]
Here is another very detailed solution: https://stackoverflow.com/a/7908612/4555249
numpy.array and scipy.sparse.csr_matrix ?
multiprocessing. You dont even have to pass them in as arguments.
[3,3,3,3,3] sometimes. Because the array is not locked it can happen that there is a race condition and only one of the two function applications gets commited. The result can as well be [1,1,1,1,1] or [2,2,2,2,2], and maybe even anything in between (although I haven't seen it)