python2.5 multiprocessing Pool

Question

I have python2.5 and multiprocessoring (get from http://code.google.com/p/python-multiprocessing/)

This simple code (get from docs), works very strange from time to time, sometimes it ok, but sometimes it throw timeout ex or hang my Windows (Vista), only reset helps :) Why this can happen?

from multiprocessing import Pool

def f(x):
    print "fc",x
    return x*x
pool = Pool(processes=4)  
if __name__ == '__main__':
    result = pool.apply_async(f, (10,))     # evaluate "f(10)" asynchronously
    print result.get(timeout=3)           # prints "100" unless your computer is *very* slow

nikow · Accepted Answer · 2010-09-09 08:35:03Z

4

This is just a wild guess, but have you tried to move the Pool creation into the if block? I suspect that otherwise it might spawn an unlimited number of new processes, causing the freeze.

answered Sep 9, 2010 at 8:35

nikow

21.6k7 gold badges54 silver badges71 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

Evg Over a year ago

i was afraid to start this again :) but it works! thanx) really pull was creating over and over and this hangs.

Eric O. Lebigot Over a year ago

@nikow: Why would processes be spawned indefinitely? I can see the 4 initial children each creating a new pool of 4 children upon importing the program, but I don't see how they would run the main code and call f for each of their children…

nikow Over a year ago

@EOL: I suspect that in every child the module code is executed. Every time a new Pool is created, spawning 4 new children. I had this happen on my machine, and saw a very large number of Python processes before the machine froze.

Eric O. Lebigot Over a year ago

@nikow: yes, but why would the 4 new children (for each new pool) create a "very large number of processes"? I would expect the 4 normal children, plus 4*4 new processes, not more. I don't see this as a reason to freeze… I'm still puzzled. :)

nikow Over a year ago

@EOL: It's recursive. Each child accidentally creates a new pool with 4 new children. Note that the process in the first original children is no different from the one in their children. That's how multiprocessing is implemented.

|

Collectives™ on Stack Overflow

python2.5 multiprocessing Pool

1 Answer 1

7 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

7 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related