error with module multiprocessing under python3.8

Question

I had a script that was multiprocessing fine until today. To reproduce the problem, I simplified the function that I parallelized with the one shown below:

    from multiprocessing import Process, Queue
    import random

    def rand_num():
        num = random.random()
        print(num)

    if __name__ == "__main__":
        queue = Queue()

        processes = [Process(target=rand_num, args=()) for x in range(4)]

        for p in processes:
            p.start()

        for p in processes:
            p.join()

that renders the exact same error message (repeated 4 times, which I omitted repeating for readability):

    Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/usr/local/Cellar/[email protected]/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
    File "/usr/local/Cellar/[email protected]/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 125, in _main
    prepare(preparation_data)
    File "/usr/local/Cellar/[email protected]/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 236, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
    File "/usr/local/Cellar/[email protected]/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
    File "/usr/local/Cellar/[email protected]/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 262, in run_path
    code, fname = _get_code_from_file(run_name, path_name)
    File "/usr/local/Cellar/[email protected]/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 232, in _get_code_from_file
    with io.open_code(fname) as f:
    FileNotFoundError: [Errno 2] No such file or directory: '/Users/myUserName/<stdin>'

I don't know where to start debugging this error. I'm running python3.8 under mac os Catalina (homebrew install). Please help.

As an aside, you should consider using a Pool, it will simplify the code. — AMC
– AMC, Commented Mar 4, 2020 at 4:35
do you run it from file - python script.py - or from interpreter ? I see File "<string>" and <stdin> which can means you run it from interpreter and maybe it makes problem. — furas
– furas, Commented Mar 4, 2020 at 4:57
Python 3.8 is very new version and it may have bugs and some modules may not works with this version. It is good to wait few month and use older version 3.7. — furas
– furas, Commented Mar 4, 2020 at 7:19
Python 3.8 on MacOS by default now uses "spawn" instead of "fork" as start method for new processes. Try with multiprocessing.set_start_method("fork") in the first line below if __name__ == "__main__":. — Darkonaut
– Darkonaut, Commented Mar 4, 2020 at 10:23
thanks @Darkonaut ! I had a similar issues, where basically my code broke after upgrading to 3.8 ... you should write a medium post ;) -- and add an answer below so we can vote it — jjrr
– jjrr, Commented Aug 24, 2020 at 18:19

Karel S · Accepted Answer · 2021-01-11 11:46:43Z

8

I faced the same problem when upgrading from Python 3.7 to 3.8. In particular now running 3.8.6 on OSX 10.15.6, Python installed by pyenv.

The advice from Darkonaut helped to resolve the issue but it's not so visible so let me rephrase it here:

Python 3.8 on MacOS by default now uses spawn instead of fork as start method for new processes. Try with

multiprocessing.set_start_method("fork")

Apparently the behaviour of the spawn is wrong as following simple example reveals:

import multiprocessing

def parallel_function(x):
    print("Function called with", x)

def test_pool():
    print("Running test_pool")
    with multiprocessing.Pool(4) as pool:
        pool.map(parallel_function, range(10))

print("Starting the test")
test_pool()

This produces following output:

Starting the test
Running test_pool
Starting the test
Running test_pool
Starting the test
Running test_pool
Starting the test
Running test_pool
Starting the test
Running test_pool
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/karel/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/Users/karel/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/spawn.py", line 125, in _main
    prepare(preparation_data)
  File "/Users/karel/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/spawn.py", line 236, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "/Users/karel/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/spawn.py", line 287, in _fixup_main_from_path

So the Pool doesn't create workers properly but instead attempts to run the whole script in each spawned process.

answered Jan 11, 2021 at 11:46

Karel S

891 silver badge3 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

laker93 Over a year ago

Note that using fork instead of spawn can be problematic: github.com/python/cpython/issues/84559. There is a change planned to make spawn the default of unix systems. Your example breaks because you do not protect test_pool() from being run recurssively by calling it in a if __name__ == '__main__' block.

laker93 · Accepted Answer · 2023-08-22 12:44:28Z

Using spawn, the default for Mac and Windows but not unix, is actually advisable; there are known issues with fork: https://github.com/python/cpython/issues/84559

When using spawn, make sure to protect against recursive imports by wrapping the calling code in a if __name__ == '__main__' block.

import multiprocessing
import random
from concurrent.futures import ProcessPoolExecutor


def rand_num(i):
    num = random.random()
    print(f"worker {i} produced random number {num}")

def test_pool():
    print("Running test_pool")
    with ProcessPoolExecutor(mp_context=multiprocessing.get_context("spawn")) as pool:
        pool.map(rand_num, range(4))

if __name__ == '__main__':
    print("Starting the test")
    test_pool()

Collectives™ on Stack Overflow

error with module multiprocessing under python3.8

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related