How can I run multiple webdriver processes concurrently in Selenium using Python?

Question

I have a list of thousands of URLs. I want to use Python/Selenuim to:

load each URL,
select one element
close the page

To make it run faster, I want to run lots of these processes in parallel, but I can only work out how to do it one at a time.

from selenium import webdriver
driver = webdriver.Chrome()

url_list = [
            'https://www.instagram.com/p/Bj7NmpqBuSw/?tagged=style', 
            'https://www.instagram.com/p/Bj7Nic3Au85/?tagged=style'
            ]

for url in url_list:
    driver.get(url)
    driver.find_elements_by_class_name("class-name-for-profile-link")
    driver.close()

I tried using lots of browser tabs

driver.switch_to.window(driver.window_handles[1])

but the handles are a bit tricky to manage.

How can I run this process in parallel?

you can also use threads, python's threading module is simple and inbuilt — Stack
– Stack, Commented Jun 12, 2018 at 14:00

hdhdhdhdhdh · Accepted Answer · 2018-11-19 08:59:05Z

4

tl;dr I created this gist to give an easy example of how to run simple Selenium tasks in parallel. You can adapt it to your own purposes.

The issue with parallelising Selenium scripts is that Selenium workers are themselves processes. The above script uses two FIFO queues, one that stores the IDs of idle Selenium workers and one that stores data to pass to the workers. Background master threads listen across both these queues and assign incoming data to idle workers, taking the ID of selenium workers off the worker queue while the worker does its work.

All you would need to do to adapt the code to your purposes is to change the code in the function selenium_task. Hope this helps!

answered Nov 19, 2018 at 8:59

hdhdhdhdhdh

997 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Andrei · Accepted Answer · 2018-06-12 14:23:48Z

2

You can use this to loop parallel. Sample usage:

from joblib import Parallel, delayed

def do_stuff(url):
    phantom = webdriver.PhantomJS('/path/to/phantomjs') # you can use any driver
    phantom.get(url)
    # do your stuff
    phantom.close()

Parallel(n_jobs=-1)(delayed(do_stuff)(url) for url in urls) #execute parallel for all urls

edited Jun 12, 2018 at 14:23

answered Jun 12, 2018 at 13:57

Andrei

5,6675 gold badges27 silver badges51 bronze badges

Collectives™ on Stack Overflow

How can I run multiple webdriver processes concurrently in Selenium using Python?

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related