Error in use of multiprocessing module with generator using multiple yields

Question

Can some please assist me with the my code below, I'm trying to implement a generator with multiple yields using multiprocessing. I would normally use a next followed by a send for the second and third calls to the generator but I'm not sure how this is done using the multiprocessing module.

from multiprocessing import Pool


def sq_divide(x, y):
    yield x**y
    send_receive = (yield x) #"send_receive" is used as confirmation that text "Done!!" was received

    if send_receive=="Done!!":
        yield x / y


if __name__ == "__main__":
    p = Pool(4)

    #first generator call
    n = p.map(sq_divide, (range(1, 10), range(1, 10)))
    print(n)

    #second generator call
    n = p.map(next, n)

    #last generator call
    n = p.map(n.send, "Done!!")
    print(n)

My expect output is:

first print:
[1, 4, 9, 16, 25, 36, 49, 64, 81]

second print:
[1, 2, 3, 4, 5, 6, 7, 8, 9]

third print:
[1, 1, 1, 1, 1, 1, 1, 1, 1]

I dont think that what you are trying to do can be done like that. Using multiprocessing create a copy of the generator for each process, so it is not really shared. — Netwave
– Netwave, Commented Mar 28, 2019 at 10:38
Is there a way it can be implemented that is different from the way I implemented it but yields the same output? @Netwave — Kusi
– Kusi, Commented Mar 28, 2019 at 10:47
Well, there are many ways, for example splitting the funtion in two. You need to spificy a bit more. Why do you need it like this? — Netwave
– Netwave, Commented Mar 28, 2019 at 10:54
Alright, the code I have written is a condensed version of another. Basically, I need the generator to yield values at different stages. Those values are then used and then the generator must continue from where it left off to the next yield value. However I need all calls to the generator processed quickly cause I have a gif running (using Tkinter) and I don't want it to stop/lag while the generator is processing. — Kusi
– Kusi, Commented Mar 28, 2019 at 11:02
Maybe it simpler if you use a thread only for the gif? I think your approach to the problem may be the problem itself. — Netwave
– Netwave, Commented Mar 28, 2019 at 11:07

Serge Ballesta · Accepted Answer · 2019-03-28 13:18:44Z

1

You are riding the wrong horse here. Pool workers are supposed to receive some input data to do a job, return their output and process a new task. Here the workers return generator objects which are not pickable hence the error. But more fundamentaly, a generator has to live until it has been exhausted, so it makes no sense in a pool worker process.

Options:

use one single background thread/process to manage all the generators while the main one will handle the UI part
dispatch the generators on a bunch of thread/processes (not from a Pool) with a long lifetime (until all generators have been exhausted)

The choice really depends on the possible bottlenecks suffered by the generators

answered Mar 28, 2019 at 13:18

Serge Ballesta

150k13 gold badges137 silver badges267 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Error in use of multiprocessing module with generator using multiple yields

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related