Run Selenium scrapers in parallel from script / Python

Question

I just learned Selenium to scrape some data that couldn't be reached with Scrapy. I have made different scripts from different bots, and they run as expected individually.

The two browsers are opened one after another, while I want to have them running at the same time. Is there a simple way to do that? Thank you for your help!

EDIT :

I have been trying this with multiprocessing, as indicated. But unfortunately it runs only one script at the time, while two selenium windows are opened. One stay inactive. Here is my code if you want to have a look :

u = UBot()
google = GoogleBot()

list_test = [[u.main(), google.main()]]

processes = []
for test in list_test:
    p = multiprocessing.Process()
    processes.append(p)
    p.start()

EDIT 2:

I could solve it and posted code below !

Possible duplicate of How to use several instance of selenium [python] — saud
– saud, Commented Dec 20, 2018 at 15:20
I watched this topic but my concern is not really about speeding the execution for now, just being able to launch several selenium bots in parallel from a main script — SidGabriel
– SidGabriel, Commented Dec 20, 2018 at 15:22
My bad. I have retracted the flag. Meanwhile, have you looked into stackoverflow.com/questions/30213888/… ? It contains a multiprocessing based approach. — saud
– saud, Commented Dec 20, 2018 at 15:25
you will need an async/threaded solution because calls to the webdriver will block until it's finished. — nicholishen
– nicholishen, Commented Dec 20, 2018 at 15:25

nicholishen · Accepted Answer · 2018-12-20 15:47:39Z

I made a simple lib called selsunpool that wraps concurrent.futures you might want to try. It creates a local pool of selenium workers which stay alive and can be reused any number of times for concurrent jobs. It's not well documented at the moment, but it's simple to use. Here is an example.

Step 1: A function is made with the selenium job decorator. The decorator param defines the name of the kwarg that the webdriver is attached to (coming back from the pool executor).

from selsunpool import selenium_job, SeleniumPoolExecutor


@selenium_job(webdriver_param_name='mydriver')
def get_url(url, mydriver):
    mydriver.get(url)
    return mydriver.title

Step 2: Use the pool executor in the same way you'd use the ThreadpoolExecutor. Note: job results are retrieved via a property which is a generator which yields results as they are finished.

with SeleniumPoolExecutor(num_workers=2, close_on_exit=True) as pool:
    sites = ['https://google.com', 'https://msn.com']
    pool.map(get_url, sites)
    print(list(pool.job_results))

SidGabriel · Accepted Answer · 2018-12-21 09:40:43Z

0

I could solve my problem based on your recommandations about Multiprocessing so thank you all :) I post my code in case other beginner would need something like this, although there is probably others (and better) way to do.

from google_bot import GoogleBot
from u_bot import UBot
from multiprocessing import Pool 

def google_process():

    google = GoogleBot()
    google.main()
    return

def u_process():

    u = UBot()
    u.main()
    return

def main():

    pool = Pool(processes=2)
    google = pool.apply_async(google_process)
    u = pool.apply_async(u_process)

    pool.close()
    pool.join()

answered Dec 21, 2018 at 9:40

SidGabriel

1951 gold badge3 silver badges12 bronze badges

Collectives™ on Stack Overflow

Run Selenium scrapers in parallel from script / Python

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related