Python - Click to scroll to bottom using Selenium

Question

I'm using Selenium in Python to extract a list of startups from AngelList. In order to collect all startups, I have to click the button 'More' at the end of the page until I reach the end of the list.

The problem is I cannot figure out how to keep clicking until it reaches the end of the page.

driver = webdriver.Chrome('C:\\Users\\Documents\\chromedriver.exe')
driver.get("https://angel.co/companies?company_types[]=Startup")
driver.find_element_by_class_name("""more""").click()

This results in one click of "More". Each click loads 20 more startups.

I have tried this in order to keep clicking:

i = 0
while i < 20:
    driver.find_element_by_class_name("""more""").click()
    i += 1

and it results in this error:

selenium.common.exceptions.StaleElementReferenceException: Message: stale 
element reference: element is not attached to the page document

Any help is much appreciated.

ewwink · Accepted Answer · 2018-12-25 00:43:09Z

2

when it reaches the end of the page, the element <div class="more">More</div> will be removed from the DOM.

To click and load more content, wait and check if the button or div.more has text More, here the example using WebDriverWait and filtered results URL

from selenium.webdriver.support.ui import WebDriverWait

driver.get('https://angel.co/companies?company_types[]=Startup&markets[]=Education&raised[min]=2830196&raised[max]=100000000&stage[]=Series+B&stage[]=Series+A')

while True:
    try:
        moreButton = WebDriverWait(driver, 10).until(
            lambda d: d.find_element_by_xpath('//div[@class="more" and text() = "More"]')
        )
        moreButton.click()
    except:
        print("scroll finished")
        break

edited Dec 25, 2018 at 0:43

answered Dec 25, 2018 at 0:37

ewwink

19.3k2 gold badges49 silver badges56 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

user10830454 Over a year ago

Thank you very much! This did work better than my other tries, I'm just not sure it captured all 2000~ companies I was looking for, but thank you!

Collectives™ on Stack Overflow

Python - Click to scroll to bottom using Selenium

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related