Python Multi Threads

Question

I need to extract all the urls from an ip list, i wrote this python script, but i have issue extracting the same ip multiple times (more threads are created with the same ip). Could anyone Improve on my solution using multithreading ?

Sorry for my english Thanks all

import urllib2, os, re, sys, os, time, httplib, thread, argparse, random

try:
    ListaIP = open(sys.argv[1], "r").readlines()
except(IOError): 
    print "Error: Check your IP list path\n"
    sys.exit(1)



def getIP():
    if len(ListaIP) != 0:
        value = random.sample(ListaIP,  1)
        ListaIP.remove(value[0])
        return value
    else:
        print "\nListaIPs sa terminat\n"
        sys.exit(1)

def extractURL(ip):
    print ip + '\n'
    page = urllib2.urlopen('http://sameip.org/ip/' + ip)
    html = page.read()
    links = re.findall(r'href=[\'"]?([^\'" >]+)', html)
    outfile = open('2.log', 'a')
    outfile.write("\n".join(links))
    outfile.close()

def start():
    while True:
        if len(ListaIP) != 0:
            test = getIP()
            IP = ''.join(test).replace('\n', '')
            extractURL(IP)
        else:
            break


for x in range(0, 10):
    thread.start_new_thread( start, () )

while 1:
    pass

It works ok to import os once; no need to import it twice. — James Waldby - jwpat7
– James Waldby - jwpat7, Commented Dec 17, 2012 at 17:04

Sanjay Manohar · Accepted Answer · 2012-12-17 16:01:22Z

5

use a threading.Lock. The lock should be global, and create at the beginning when you create the IP list.

lock.acquire at the start of getIP()

and release it before you leave the method.

What you are seeing is, thread 1 executes value=random.sample, and then thread 2 also executes value=random.sample before thread 1 gets to the remove. So the item is still in the list at the time thread 2 gets there. Therefore both threads have a chance of getting the same IP.

answered Dec 17, 2012 at 16:01

Sanjay Manohar

7,0564 gold badges37 silver badges61 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Jonas Schäfer Over a year ago

cooler method to use locks: with lock: statements (using context managers )

AutoSoft Over a year ago

def getIP(): lock.acquire() if len(ListaIP) != 0: value = random.sample(ListaIP, 1) ListaIP.remove(value[0]) return value else: print "\nListaIPs sa terminat\n" sys.exit(1) lock.release() global name 'lock' is not defined Not working ... I give me this error.

Collectives™ on Stack Overflow

Python Multi Threads

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related