Multiprocessing in Python

Question

I am trying to apply multiprocessing in the simplest way in Python 3 but it does not work on my laptop. I am using Windows.

from multiprocessing import Process

# a dummy function
def f(x):
    print(x)

if __name__ == '__main__':
    p = Process(target=f, args=('some text',))
    p.start()
    p.join()

print('Done')

It did not end as it was expected. Instead, I got this error:

Traceback (most recent call last):
File "C:\Users\Mahdi\anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 3437, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-2-46a0e604292b>", line 1, in <module>
runfile('C:/Users/Mahdi/Mahdi Code/test.py', wdir='C:/Users/Mahdi/Mahdi Code')
File "C:\Program Files\JetBrains\PyCharm 2021.2.2\plugins\python\helpers\pydev\_pydev_bundle\pydev_umd.py", line 198, in runfile
pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
File "C:\Program Files\JetBrains\PyCharm 2021.2.2\plugins\python\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "C:/Users/Mahdi/Mahdi Code/test.py", line 25, in <module>
p.start()
File "C:\Users\Mahdi\anaconda3\lib\multiprocessing\process.py", line 121, in start
self._popen = self._Popen(self)
File "C:\Users\Mahdi\anaconda3\lib\multiprocessing\context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Users\Mahdi\anaconda3\lib\multiprocessing\context.py", line 327, in _Popen
return Popen(process_obj)
File "C:\Users\Mahdi\anaconda3\lib\multiprocessing\popen_spawn_win32.py", line 93, in __init__
reduction.dump(process_obj, to_child)
File "C:\Users\Mahdi\anaconda3\lib\multiprocessing\reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function f at 0x00000218267C3D30>: attribute lookup f on __main__ failed
Done
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\Mahdi\anaconda3\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "C:\Users\Mahdi\anaconda3\lib\multiprocessing\spawn.py", line 126, in _main
self = reduction.pickle.load(from_parent)
EOFError: Ran out of input

Does anyone know about this?

Don't run it in IPython/interactive shell.

matszwecja
– matszwecja

2023-01-03 08:16:26 +00:00
Commented Jan 3, 2023 at 8:16 — matszwecja
– matszwecja, Commented Jan 3, 2023 at 8:16

Ahmed AEK · Accepted Answer · 2023-01-03 08:17:47Z

5

some interactive IDEs don't support multiprocessing such as jupyter lab, there are tricks around this involving non-standard multiprocessing modules, but the most straight-forward solution is to not use an interactive environment for multiprocessed code, and instead run python in script mode using VsCode or Pycharm or through the terminal. (Spyder also works but you have to run the code as a script)

answered Jan 3, 2023 at 8:17

Ahmed AEK

23.2k3 gold badges19 silver badges50 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Mahdi Haqverdi Over a year ago

Thanks, I am using Pycharm. Can this be something in the settings?

Ahmed AEK Over a year ago

@MahdiHaqverdi don't run it using "Run file in python console" and instead run it using "Run somefile" command. (the one with the green play button image)

Shubhank Gupta · Accepted Answer · 2023-01-03 08:23:11Z

0

You can run the same code via the Terminal or Pycharm. I haven't checked other IDE's.

It ran fine on the terminal and the output is

some text
Done

answered Jan 3, 2023 at 8:23

Shubhank Gupta

8452 gold badges14 silver badges35 bronze badges

Collectives™ on Stack Overflow

Multiprocessing in Python

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related