Calling class method in python thread queue

Question

Could someone please shed some light on why this threaded code to call a classes' method never completes?

from Queue import Queue
from threading import Thread

class SimpleThing(object):
    def __init__(self, name):
        self.name = name

    def print_name(self):
        print self.name

class ThingSpawner(object):
    def __init__(self, name_list):
        self.things  = [SimpleThing(name) for name in name_list]
        self.thread_queue = Queue()

    def run(self):
        for thing in self.things:
            t = Thread(target=thing.print_name, name=thing.name)
            t.daemon = True
            t.start()
            self.thread_queue.put(t)
        self.thread_queue.join()

thing_list = ['cat', 'dog', 'llama', 'bat']

sp = ThingSpawner(thing_list)
sp.run()

The code will clearly run the print_name method, but does not join() and exit.

Also, what is the neatest way to modify this code so that the join() completes? The motivation is to use an existing python control class for a bit of hardware, and allows you to call a (very slow) method of the control class in parallel. Thanks!

Ofer Sadan · Accepted Answer · 2017-08-01 08:16:53Z

When you are doing

self.thread_queue.put(t)

You are putting some threads into the Queue, obviously. However, i'm not really sure why. You never use that queue again for anything, and it's completely unnecessary. To make matters worse, you then call

self.thread_queue.join()

Which basically waits forever for the queue to empty, which never happens, because you never empty it or do anything with it.

If I copy paste all your code, but without any Queue at all, everything is fine...

from threading import Thread

class SimpleThing(object):
    def __init__(self, name):
        self.name = name

    def print_name(self):
        print self.name

class ThingSpawner(object):
    def __init__(self, name_list):
        self.things  = [SimpleThing(name) for name in name_list]

    def run(self):
        for thing in self.things:
            t = Thread(target=thing.print_name, name=thing.name)
            t.daemon = True
            t.start()

thing_list = ['cat', 'dog', 'llama', 'bat']

sp = ThingSpawner(thing_list)
sp.run()

However that's not what you want! Because your threads are daemons they will exit when the main program exits, even if they are not done yet (if I add some delay like sleep(1) before printing the name for example). You should call join() on the threads, not the queue, if you want to wait for them to finish. So we'll return the threads first:

    def run(self):
        all_threads = []
        for thing in self.things:
            t = Thread(target=thing.print_name, name=thing.name)
            t.daemon = True
            t.start()
            all_threads.append(t)
        return all_threads

And when we run we'll do this:

threads = sp.run()
for t in threads:
    t.join()

telegraphic · Accepted Answer · 2017-08-01 12:42:41Z

Thanks Ofer for the clear answer, which I've just accepted -- I am indeed not using the queue properly! Having reacquainted myself with queues now you've pointed out my error, for prosperity, here's an alternative approach using a queue:

from Queue import Queue
from threading import Thread

class SimpleThing(object):
    def __init__(self, name, q):
        self.name = name

    def print_name(self, q):
        print self.name
        q.get()
        q.task_done()

class ThingSpawner(object):
    def __init__(self, name_list):
        self.thread_queue = Queue()
        self.things  = [SimpleThing(name, self.thread_queue) for name in name_list]


    def run(self):
        for thing in self.things:
            t = Thread(target=thing.print_name, name=thing.name, args=(self.thread_queue,))
            t.daemon = True
            t.start()
            self.thread_queue.put(t)
        self.thread_queue.join()

thing_list = ['cat', 'dog', 'llama', 'bat']

sp = ThingSpawner(thing_list)
sp.run()

This tutorial on threading and queues was useful once I understood my mistake.

Collectives™ on Stack Overflow

Calling class method in python thread queue

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related