Python Selenium unable to click() button

Question

I'm trying to automate the scraping of links from here:

https://thegoodpubguide.co.uk/pubs/?paged=1&order_by=category&search=pubs&pub_name=&postal_code=&region=london

Once I have the first page, I want to click the right chevron at the bottom, in order to move to the second, the third and so on. Scraping the links in between.

Unfortunately nothing I try will allow me to send chrome to the next page.

    from selenium import webdriver
    from selenium.webdriver.common.keys import Keys
    import time
    from datetime import datetime
    import csv
    from selenium.webdriver.common.action_chains import ActionChains

    #User login info
    pagenum = 1

    #Creates link to Chrome Driver and shortens this to 'browser'
    path_to_chromedriver = '/Users/abc/Downloads/chromedriver 2' # change path as needed
    driver = webdriver.Chrome(executable_path = path_to_chromedriver)

    #Navigates Chrome to the specified page
    url = 'https://thegoodpubguide.co.uk/pubs/?paged=1&order_by=category&search=pubs&pub_name=&postal_code=&region=london'


    #Clicks Login
    def findlinks(address):

        global pagenum

        list = []

        driver.get(address)
        #wait
        while pagenum <= 2:

            for i in range(20): # Scrapes available links
                xref = '//*[@id="search-results"]/div[1]/div[' + str(i+1) + ']/div/div/div[2]/div[1]/p/a'
                link = driver.find_element_by_xpath(xref).get_attribute('href')
                print(link)
                list.append(link)

            with open("links.csv", "a") as fp: # Saves list to file 
                wr = csv.writer(fp, dialect='excel')
                wr.writerow(list)

            print(pagenum)

            pagenum = pagenum + 1

            element = driver.find_element_by_xpath('//*[@id="search-results"]/div[2]/div/div/ul/li[8]/a')
            element.click()


    findlinks(url)

Is something blocking the button that i'm not seeing?

The error printed in my terminal:

selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[@id="search-results"]/div[2]/div/div/ul/li[8]/a"}

cruisepandey · Accepted Answer · 2018-05-15 18:05:12Z

1

try this :

element = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a[class='next-page btn']"))  
element.click()

answered May 15, 2018 at 18:05

cruisepandey

29.5k6 gold badges23 silver badges43 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

lordf Over a year ago

After adding the imports needed, i'm still getting: selenium.common.exceptions.WebDriverException: Message: unknown error: Element is not clickable at point (565, 4990)

cruisepandey Over a year ago

Yes this error is fine ! You need to scroll down to click on the next arrow. Do the scroll down and then use the code provided by me.

cruisepandey Over a year ago

I have checked css selector provided by me is generic. and would work if scroll down is performed.

lordf Over a year ago

Got it! works great. I simply included: driver.execute_script("window.scrollTo(0, 5000);") so that i could automate the scrolling. It did throw up selenium.common.exceptions.WebDriverException: Message: unknown error: call function result missing 'value' , but all i needed to do was download an updated chrome driver and all works fine.

natn2323 · Accepted Answer · 2018-05-15 16:27:37Z

0

EDIT :

The xpath that you're specifying for the chevron is variable between pages, and is not exactly correct. Note the li[6] and li[8] and li[9].

On page 1: the xpath is //*[@id="search-results"]/div[2]/div/div/ul/li[6]/a/i

On page 2: the xpath is //*[@id="search-results"]/div[2]/div/div/ul/li[8]/a/i

On page 3: the xpath is //*[@id="search-results"]/div[2]/div/div/ul/li[9]/a/i

You'll have to come up with some way of determining what xpath to use. Here's a hint: it seems that the last li under the //*[@id="search-results"]/div[2]/div/div/ul/ designates the chevron.

ORIGINAL POST :

You may want to try waiting for the page to load before you try to find and click the chevron. I usually just do a time.sleep(...) when I'm testing my automation script, but for (possibly) more sophisticated functions, try Waits. See the documentation here.

edited May 15, 2018 at 16:27

answered May 15, 2018 at 16:12

natn2323

2,1112 gold badges17 silver badges34 bronze badges

3 Comments

lordf Over a year ago

Doesn't seem to make a difference. The issue seems to be that it can't locate the button in the first place. I've added a wait in regardless tho. Thanks.

Alloy Over a year ago

Tacking on my .02 here, implicit waits in selenium have saved me before.

lordf Over a year ago

@musikreck thanks for the edit and i'll definitely take a look. However I think i'm still missing something as I cant get the button clicked for p1.

Collectives™ on Stack Overflow

Python Selenium unable to click() button

2 Answers 2

4 Comments

EDIT :

ORIGINAL POST :

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

EDIT :

ORIGINAL POST :

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related