Headless chrome with selenium, can only find ways to scroll non-headless

Question

There is a lot to find about this subject, but can't figure this out. I need to scroll to the end of the page of a (not so long) infinity scroll. I have 2 options that work with chrome non-headless but doesn't seem to work headless.

The first one which i liked the most, works beautiful and found here on SA:

driver = webdriver.Chrome('c:/cd.exe', chrome_options=chrome_options)
driver.get('http://www.website.com')

while True:
    count = len(driver.find_elements_by_xpath('//div[@itemprop="itemListElement"]'))
    print(count)
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    try:
        WebDriverWait(driver, 50).until(EC.visibility_of_element_located((By.XPATH,
                                                                          "//div[@itemprop='itemListElement'][%s]" % str(count + 1))))
    except TimeoutException:
        break

Second more hack job after realizing i can't get away with above in headless mode:

driver = webdriver.Chrome('c:/cd.exe', chrome_options=chrome_options)
driver.get('https://www.website.com')

while True:

    count = len(driver.find_elements_by_xpath('//div[@itemprop="itemListElement"]'))
    actions = ActionChains(driver)
    actions.send_keys(Keys.PAGE_DOWN)
    actions.perform()
    actions.send_keys(Keys.PAGE_DOWN)
    actions.perform()


    # focus_element_scroll = driver.find_elements_by_xpath('//section[@class="occasion-content"]')
    # driver.find_elements_by_xpath('//div[@itemprop="itemListElement"]')[-1].send_keys(Keys.PAGE_DOWN)
    # driver.find_elements_by_xpath('//div[@itemprop="itemListElement"]')[-1].send_keys(Keys.PAGE_DOWN)
    # self.driver.find_element_by_css_selector("ul.list-with-results").send_keys(Keys.ARROW_DOWN)
    print(count)
    # driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

    try:  
        WebDriverWait(driver, 50).until(EC.visibility_of_element_located((By.XPATH,
                                                                          "//div[@itemprop='itemListElement'][%s]" % str(count + 1))))
    except TimeoutException:
        break

So both work in chrome but don't in headless mode, i need to push them to a ubuntu vps where they need to be headless, i know of the xvfb option but i am glad i could remove that and use native chrome since the droplets don't have much memmory.

Edit: Just tried this approach with focus on an element in the footer, also works in non-headless but doesn't in headless:

ActionChains(driver).move_to_element(focus[0]).perform()

Someone got a different approach?

Edit Just want to know if it is possible to scroll with chrome in headless mode!

userRR · Accepted Answer · 2018-01-17 14:37:37Z

12

Found the answer after 2 days trying different combinations of versions of selenium, chrome and chromedriver i alsmost gave up and wanted to go with the xvfb.

Already tried to maximze the window in the chrome arguments, that didn't help. But this time i tried setting a manual window size. That helped.

    chrome_options.add_argument("window-size=1920,1080")

Posting here so that the next one wont take as long as me.

answered Jan 17, 2018 at 14:37

userRR

3391 gold badge3 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

silverFoxA Over a year ago

This doesn't work for me, does it work with latest version?

undetected Selenium · Accepted Answer · 2019-06-07 10:58:14Z

2

To scroll to the end of the page of a (not so long) infinity scroll through Default Chrome Browser and Headless Chrome Browser you can use the following code block :

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException

options = Options()
options.add_argument("--headless")
options.add_argument("start-maximized")
options.add_argument("disable-infobars")
options.add_argument("--disable-extensions")
options.add_argument("--no-sandbox")
options.add_argument("--disable-dev-shm-usage")
driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
driver.get('http://www.website.com')

while (driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")):
    try:
        WebDriverWait(driver, 50).until(EC.visibility_of_element_located((By.XPATH, "//div[@itemprop='itemListElement']" )))
        # do your other actions within the Viewport
    except TimeoutException:
        break
print("Reached to the bottom of the page")

edited Jun 7, 2019 at 10:58

answered Jan 15, 2018 at 8:54

undetected Selenium

194k44 gold badges304 silver badges387 bronze badges

6 Comments

userRR Over a year ago

thanks for the awnser but this is how i have set it up, it works, bot only in non headless mode, same as the 2 other solutions i presented. Did you try this code?

undetected Selenium Over a year ago

I didn't get your comment. Doesn't my answer executes for you in headless mode? The code is pretty well tested at my end.

userRR Over a year ago

The code works well without the --headless parameter. In windows and on my ubuntu vps it doesn't work in headless mode... There takes no scrolling place.

undetected Selenium Over a year ago

@user9108711 Update the question with the error stack trace please on Windows.

userRR Over a year ago

There is no error, the WebDriverWait kicks in and the element never becomes visible because there is no scrolling. Again, i can see it clearly scrolling nice without the headless argument. The page i am trying to fetch: http:// bit.ly/2mxdV1Z

|

imp · Accepted Answer · 2019-06-05 13:55:15Z

0

I just ran into this problem on windows. Using chrome 74 and i fixed the issue by having the below chromeOptions. My headless mode works again :) Thanks at DebanjanB

chromeOptions.addArguments("--headless")
chromeOptions.addArguments("--no-sandbox")
chromeOptions.addArguments("--disable-dev-shm-usage")
chromeOptions.addArguments("--window-size=1920x1080")
chromeOptions.addArguments("start-maximised")

answered Jun 5, 2019 at 13:55

imp

4366 silver badges21 bronze badges

Collectives™ on Stack Overflow

Headless chrome with selenium, can only find ways to scroll non-headless

3 Answers 3

1 Comment

6 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

6 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related