Selenium - Python Set timeout on finding elements

Question

I am gathering links from a website. I iterate over the pages that it has, and on each page I retrieve the links with:

links = driver.find_elements_by_xpath('//*[contains(@class, "m_rs_list_item_main")]/div[1]/div[1]/a')

Now... sometimes the website fails and does not show the links that it should. For instance, it normally says:

link1

link2

...

link N

page M

And suddenly there is a page, let's say M+1 that doesn't show any links at all. Then the code gets stuck at the above line (links = ...) "looking for" the links. I count the links with a counter in order to see how many links in each page I have:

if numlinks_inrun == 0:
    print('nolinks')

Now, I never get the message 'nolinks' printed. When I press CTRL+C to abort the program in the terminal, I get in the terminal the traceback:

links = driver.find_elements_by_xpath('//*[contains(@class, "m_rs_list_item_main")]/div[1]/div[1]/a')
  File "/home/vladimir/anaconda3/lib/python3.5/site-packages/selenium/webdriver/remote/webdriver.py", line 305, in find_elements_by_xpath
    return self.find_elements(by=By.XPATH, value=xpath)

This is why I know that the program gets stuck at this point. Does anyone know how to set a timeout so that selenium does not search forever those unexisting links?

@VladimirVargas I am getting a message as This request was blocked by the security rules when I try to access the website. Do we have any alternative? Thanks — undetected Selenium
– undetected Selenium, Commented Jun 2, 2017 at 14:27
It looks like a particular page or some page taking long time to loading it. You can set page load timeout. I don't think it is because of find element which is implicitly wait as by default, if it is not finding any element, it will not wait until you set implicitly timeout. — Murthi
– Murthi, Commented Jun 2, 2017 at 22:36

Wunderbread · Accepted Answer · 2018-08-31 21:59:35Z

3

This appears to be an issue with the element not actually loading in time for selenium to locate it. You may have to consider adding an explicit wait which you can use to set the amount of seconds prior to selenium locating the specified page element. That's why you're not seeing the "nolinks" output because it errors out.

Context: https://selenium-python.readthedocs.io/waits.html#explicit-waits

edited Aug 31, 2018 at 21:59

answered Jun 2, 2017 at 17:03

Wunderbread

1,0702 gold badges20 silver badges40 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Selenium - Python Set timeout on finding elements

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related