1

I am trying to process some arrays with Pool amd many workers. As I am working with relatively big arrays I use shared memory which is converted to numpy arrays later in the called function:

shared_arr = Array(ctypes.c_ulong, size[0] * size[1])
img = tonumpyarray(shared_arr, size)

shared_arr_l = Array(ctypes.c_ulong, size[0] * size[1])
labimg = tonumpyarray(shared_arr_l, size)

The Pool function is:

with closing(Pool(initializer=init, initargs=(shared_arr, shared_arr_l), processes=n_cores)) as p:
        p.starmap_async(classific, [(p, size) for p in props1])
p.join()

It works for small data but for large data I get an AssertionError for every worker:

Process SpawnPoolWorker-1:
Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\Lib\multiprocessing\process.py", line 258, in _bootstrap
    self.run()
  File "C:\ProgramData\Anaconda3\Lib\multiprocessing\process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "C:\ProgramData\Anaconda3\Lib\multiprocessing\pool.py", line 108, in worker
    task = get()
  File "C:\ProgramData\Anaconda3\Lib\multiprocessing\queues.py", line 335, in get
    res = self._reader.recv_bytes()
  File "C:\ProgramData\Anaconda3\Lib\multiprocessing\connection.py", line 216, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "C:\ProgramData\Anaconda3\Lib\multiprocessing\connection.py", line 318, in _recv_bytes
    return self._get_more_data(ov, maxsize)
  File "C:\ProgramData\Anaconda3\Lib\multiprocessing\connection.py", line 337, in _get_more_data
    assert left > 0
AssertionError

Is it a memory problem? RAM shouldn't be a problem - the computer has 128 GB while the two arrays are less than 2 GB and are stored in a shared memory. I tried to solve it but I am lost. Any help would be greatly appreciated.

1
  • possible duplication of this. please read through Commented Jul 12, 2018 at 13:39

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.