Memory keeps getting up during Threading with Python

Question

I have a problem with Threading in Python. The Memory keeps getting up (pythonw.exe).. it starts with +- 20.000 kB, but keeps rising untill the programm is finished. Anyone knows how to fix this?

class Threads(threading.Thread):
    def run(self):
        try:
            HTML = urllib2.urlopen(//URL//).read()
        except urllib2.HTTPError: pass
        except: pass

def __Scan__():
    Count = 0
    while Count <10000:
        Count = Count + 1
        try:
            Thread = Threads()
            Thread.name = Count
            Thread.start()
        except:
            Count = Count - 1

Each Thread will open an URL, and then I store the number of the Thread in a list. But I don't think thats the cause of the rising memory? Anyone can help?

Thx

what are you trying to? better look at the multiprocessing module of Python — user2665694
– user2665694, Commented Jan 1, 2013 at 18:01
And you are starting tons of new Threads without handling their proper terminatiom — user2665694
– user2665694, Commented Jan 1, 2013 at 18:02

jeffknupp · Accepted Answer · 2013-01-01 18:02:42Z

1

The increase in memory is likely from the stack space created for each new thread. Threads in Python (and most other languages) have a certain amount of resource overhead. Each time you create one, a little bit of memory is allocated for it.

answered Jan 1, 2013 at 18:02

jeffknupp

6,3643 gold badges30 silver badges29 bronze badges

Sign up to request clarification or add additional context in comments.

11 Comments

favoretti Over a year ago

Plus, he is creating 10000 threads, how would that go without memory usage increasing?

Coryza Over a year ago

I supposed that while creating new threads, the old ones would be done. So I thought the Memory should stay atleast proximately at the same level.

jeffknupp Over a year ago

You would need to call join on each thread for the resources to be reclaimed.

Coryza Over a year ago

That works... but it slows this process a lot... It first did 1000 URLS in 2 sec's, with the join() method it takes over 1 minute..

user2665694 Over a year ago

Look into multiprocessing.Pool()

|

Collectives™ on Stack Overflow

Memory keeps getting up during Threading with Python

1 Answer 1

11 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

11 Comments

Your Answer

Sign up or log in

Post as a guest

Related