how can i use multi-threading for web-crawling in selenium web-driver python

Question

i have scenario where i go to a webpage and open each link in a new window and check for specific documents but its taking enormous time to go through each link so is there a way to increase performance using multi-threading

known problem

selenium is not thread safe

but i can create multiple instances of driver for each thread which takes care of this problem

current code which i am using

    for tag in self.driver.find_elements_by_xpath('//body//a[@href]'):
        current_window = self.driver.current_window_handle
        if href:
                self.driver.execute_script('window.open(arguments[0]);', href)
                time.sleep(10)
                new_window = [window for window in self.driver.window_handles if window != current_window][0]
                self.driver.switch_to.window(new_window)
                # Execute required operations
                func_url=self.driver.current_url
                self.driver.close()
                self.driver.switch_to.window(current_window)
                if not func_url:
                    continue
                if re.search('\.(?:pdf|png|jpg|doc|ppt|zip)',func_url):
                        cat=fd.findCat(func_url)
                        fd.findDate(func_url)
                time.sleep(10)

but i can create multiple instances of driver for each thread which takes care of this problem so what is the question? — Flurin
– Flurin, Commented Apr 24, 2017 at 11:59
time consumption is enormous i want to reduce time so if i create instances it will add more time and memory so i am asking for any alternative way to do multi-threading — Pavan Kumar T S
– Pavan Kumar T S, Commented Apr 24, 2017 at 12:07

Nicolas Alejandro Vaquero · Accepted Answer · 2017-04-25 19:01:08Z

1

If you want to win time, first of all you have to use WebDriverWait:

This is an EXAMPLE

def exec_sync(driver, script, waitTime, scriptName):
    global contador
    print scriptName
    driver.execute_script(script)
    WebDriverWait(driver, waitTime).until(
        EC.presence_of_element_located(
            (By.ID, 'uniqueIdentifier' + str(contador))
        )
        #driver.find_element_by_id(cadena)
    )
    contador += 1
    print 'Success'

Second, WebDriver is not for multithreading as you have said, so you HAVE TO use SEMAPHORE

https://docs.python.org/2/library/threading.html#semaphore-objects https://www.tutorialspoint.com/python/python_multithreading.htm

It's the only way for multithreading

MT + Wait.until is what you need to save time

answered Apr 25, 2017 at 19:01

Nicolas Alejandro Vaquero

263 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

how can i use multi-threading for web-crawling in selenium web-driver python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related