Python - multi-process code returns results to a queue successfully, but the code stops

Question

Wonder if you can help? I'm new to Python, and am trying to put together a simple piece of code that calculates prime numbers in parallel (across multiple CPU cores)... I know it's lame, but it seemed like a reasonable piece of code to learn about Python threads etc.

I have successfully managed to run the code using Threads, but am now investigating Process(es).

The code below runs fine up to the point where i'm gathering the results. I've tried to debug using eclipse & pdev debugger and have found the While loop that i'm using to remove items from outputQueue seems to stick after 3 iterations, so the code never actually reaches the stage where it outputs the results.

Any ideas / advice / help would be greatly appreciated.

Many thanks

Craig ------ code

from multiprocessing import Process, Queue

# return true if number is prime, else false
def calcPrime(number):    
    divisor=2
    if number%2==0: 
        return False

    numberIsPrime=True
    while divisor*divisor <= number:   
        if number % divisor == 0:
            numberIsPrime = False
            break
        divisor = divisor + 1
    return numberIsPrime

# generate list of primes
def generatePrimes(minimum, maximum):
    return [num for num in range(minimum,maximum) if calcPrime(num)==True]

def workerThread(output, mn, mx):
    primelist=generatePrimes(mn,mx)
    output.put(primelist)

def main():
    outputQueue=Queue()

    t=[]
    t.append(Process(target=workerThread, args=(outputQueue, 1,25000)))
    t.append(Process(target=workerThread, args=(outputQueue, 25001, 50000)))
    t.append(Process(target=workerThread, args=(outputQueue, 50001, 75000)))
    t.append(Process(target=workerThread, args=(outputQueue, 75001, 100000)))

    #start all threads
    for idx in range(len(t)):
        t[idx].daemon=True
        t[idx].start()

    #wait for all process threads to complete
    for p in t:
        p.join()
    print("Processes finished")

    # gather all results
    l=[]
    while True:
        try:  
            l+=outputQueue.get()  # Code seems to stick here after about 3-4 iterations
        except:
            break

    #print out our lovely primes
    for idx in range(len(l)):
        print (str(l[idx]))    

# standard code        
if __name__ == '__main__':
    main()

if calcPrime(num)==True The comparison is redundant. Simply use if calcPrime(num). Also, when comparing for True,False or None you should the is operator: if calcPrime(num) is True. Whenever you simply want to know the "truthy" value of something simply use not to check for "false" and the object itself for "true". — Bakuriu
– Bakuriu, Commented Aug 31, 2013 at 18:20

Bakuriu · Accepted Answer · 2013-08-31 18:14:53Z

1

This code is an infinite loop:

l=[]
while True:
    try:  
        l+=outputQueue.get()  # Code seems to stick here after about 3-4 iterations
    except:
        break

The calls to get() are blocking, i.e. it will wait until you send in something. In your case, when the processes end the loop does another call to get() that never returns.

Since you know the number of processes you can simply do that number of get()s:

l = sum((outputQueue.get() for _ in range(t)), [])

If a process can push a variable number of results, then you can send a sentinel value when the worker finishes(e.g. None). The process that collects the output can count how many sentinel it has received, and eventually stop querying the queue.

answered Aug 31, 2013 at 18:14

Bakuriu

103k23 gold badges208 silver badges236 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python - multi-process code returns results to a queue successfully, but the code stops

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related