how to use multiprocessing.Pool in python

Question

I need to run the same function based on the same data a lot of times. For this I am using multiprocessing.Pool in order to speedup the computation.

from multiprocessing import Pool
import numpy as np
x=np.array([1,2,3,4,5])

def func(x): #this should be a function that takes 3 minutes 
    m=mean(x)
    return(m)

p=Pool(100)
mapper=p.map(multiple_cv,[x]*500)

The program works well but at the end I have 100 python processes opened and all my system starts to go very slow.

How can I solve this? Am

I using Pool in the wrong way? Should I use another function?

EDIT: using p = Pool(multiprocessing.cpu_count()) will my PC use 100% of it's power? Or there is something else I should use?

Try to experiment a bit with the pool size. My guess would be that there is some optimum, if your pool size is your CPU count times 2 (exhausting all possibly hyper-threaded cores but not bury cycles in scheduling). — miku
– miku, Commented May 1, 2014 at 16:42
You should close and join your pool at the end, see pymotw.com/2/multiprocessing/communication.html#process-pools — Lukas Graf
– Lukas Graf, Commented May 1, 2014 at 16:56

NanoBennett · Accepted Answer · 2014-05-01 16:56:11Z

2

In addition to limiting yourself to

p = Pool(multiprocessing.cpu_count())

I believe you want to do the following when you're finished as well...

p.close()

This should close out the process after it's completed.

answered May 1, 2014 at 16:56

NanoBennett

1,9001 gold badge13 silver badges13 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

dano · Accepted Answer · 2014-05-01 16:48:50Z

1

As a general rule, you don't want too many more pools than you have CPU cores, because your computer won't be able to parallelize the work beyond the number of cores available to actually do the processing. It doesn't matter if you've got 100 processes when your CPU can only process four thing simultaneously. A common practice is to do this

p = Pool(multiprocessing.cpu_count())

answered May 1, 2014 at 16:48

dano

95.5k20 gold badges234 silver badges231 bronze badges

2 Comments

Donbeo Over a year ago

at the end of the task should'n I still close all the process?

dano Over a year ago

Yes, you should do that, too.

Collectives™ on Stack Overflow

how to use multiprocessing.Pool in python

2 Answers 2

Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related