Python multi threading with queue

Question

Here is my code;

import threading
from Queue import Queue



words = open("words.txt")


lock = threading.Lock()
threads = 10


q = Queue(maxsize=0)


def myfunction(q):
    lock.acquire()
    print q.get()
    lock.release()
    q.task_done()


for x in range(threads):
    m = threading.Thread(target=myfunction, args=(q,))
    m.setDaemon(True)
    m.start()

for word in words:
    q.put(word.strip())




q.join()

raw_input()

This will output:

word1
word2
word3
word4
word5
word6
word7
word8
word9
word10

Then it will stop. There are many more words in the file, how can I have it keep continuing? From my understanding, q.join() should wait until the queue is empty to add more.

I thought about putting it in a loop like this:

for word in words:
    q.put(word.strip())
    for x in range(threads):
        m = threading.Thread(target=myfunction, args=(q,))
        m.setDaemon(True)
        m.start()

But I sometimes get an error saying "cannot start new thread".

wim · Accepted Answer · 2014-07-27 13:53:46Z

2

Your threads just finish after they have processed one queue item.

Put the worker code (i.e. the body of myfunction) into a while: True loop so that it keeps on getting more items from the queue, rather than returning once q.task_done() was called.

answered Jul 27, 2014 at 13:53

wim

368k113 gold badges681 silver badges816 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python multi threading with queue

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related