Python / Scrapy / Selenium / PhantomJs - performance

Question

I am making a web crawler/scraper using Python and Scrapy. Because some websites load their content dynamically, i´m also using Selenium in combination with PhantomJs. Now when i started using this i thought the performance would be acceptable, but turns out it´s quite slow. Now i´m not sure if that is because of some loophole in my code, or because the frameworks/programs i'm using are not optimised enough. So i´m asking you guy´s about suggestions about what i could do to improve the performance.
The code i wrote takes approx. 35 sec to start and end. It´s executing about 11 GET requests and 3 Post requests.

import scrapy
from scrapy.http.request import Request
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import WebDriverWait
import time


class TechcrunchSpider(scrapy.Spider):
    name = "techcrunch_spider_performance"
    allowed_domains = ['techcrunch.com']
    start_urls = ['https://techcrunch.com/search/heartbleed']



    def __init__(self):
        self.driver = webdriver.PhantomJS()
        self.driver.set_window_size(1120, 550)
        #self.driver = webdriver.Chrome("C:\Users\Daniel\Desktop\Sonstiges\chromedriver.exe")
        self.driver.wait = WebDriverWait(self.driver, 5)    #wartet bis zu 5 sekunden

    def parse(self, response):
        start = time.time()     #ZEITMESSUNG
        self.driver.get(response.url)

        #wartet bis zu 5 sekunden(oben definiert) auf den eintritt der condition, danach schmeist er den TimeoutException error
        try:    

            self.driver.wait.until(EC.presence_of_element_located(
                (By.CLASS_NAME, "block-content")))
            print("Found : block-content")

        except TimeoutException:
            self.driver.close()
            print(" block-content NOT FOUND IN TECHCRUNCH !!!")


        #Crawle durch Javascript erstellte Inhalte mit Selenium

        ahref = self.driver.find_elements(By.XPATH,'//h2[@class="post-title st-result-title"]/a')

        hreflist = []
        #Alle Links zu den jeweiligen Artikeln sammeln
        for elem in ahref :
            hreflist.append(elem.get_attribute("href"))


        for elem in hreflist :
            print(elem)



        print("im closing myself")
        self.driver.close()
        end = time.time()
        print("Time elapsed : ")
        finaltime = end-start
        print(finaltime)

I am using Windows 8 64bit , intel i7-3630QM CPU @ 2,4GHZ , Nvidia Geforce GT 650M, 8GB Ram.
PS: sorry for German comments

You could try generating the AJAX-requests through your spider, thus eliminating the need for Selenium and not needing to wait 5 seconds for the page to be loaded. Check this frequent post. — rongon
– rongon, Commented Jun 13, 2017 at 8:14
Read the answer of this question stackoverflow.com/questions/39036137/… — parik
– parik, Commented Jun 13, 2017 at 8:41

Om Prakash · Accepted Answer · 2017-12-11 09:36:28Z

2

I was also facing this same issue getting only 2 url processed per minute.

I cache web page by doing this.

......
options = ['--disk-cache=true']
self.driver = webdriver.PhantomJS(service_args=options)
......

This shoot up the url processing from 2 to 11 per minute in case. This may very from web page to web page.

In case, you want to disable image loading to speed up page loading in selenium, add --load-images=false to options above.

Hope it helps.

edited Dec 11, 2017 at 9:36

answered Dec 11, 2017 at 9:03

Om Prakash

2,9014 gold badges32 silver badges50 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

graph · Accepted Answer · 2017-06-15 04:27:32Z

1

Try using Splash to process pages with Javascript instead.

answered Jun 15, 2017 at 4:27

graph

774 bronze badges

Collectives™ on Stack Overflow

Python / Scrapy / Selenium / PhantomJs - performance

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related