Selenium Web Scraping: Find element by text not working in script

Question

I am working on a script to gather information off Newegg to look at changes over time in graphics card prices. Currently, my script will open a Newegg search on RTX 3080's through Chromedriver and then click on the link for Desktop Graphics Cards to narrow down my search. The part that I am struggling with is developing a for item in range loop that will let me iterate through all 8 search result pages. I know that I could do this by simply changing the page number in the URL, but as this is an exercise that I'm trying to use to learn Relative Xpath better, I want to do it using the Pagination buttons at the bottom of the page. I know that each button should contain inner text of "1,2,3,4 etc." but whenever I use text() = {item} in my for loop, it doesn't click the button. The script runs and doesn't return any exceptions, but doesn't do what I want it too. Below I have attached the HTML for the page as well as my current script. Any suggestions or hints are appreciated.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import NoSuchElementException
import pandas as pd
import time

options = Options()

PATH = 'C://Program Files (x86)//chromedriver.exe'

driver = webdriver.Chrome(PATH)

url = 'https://www.newegg.com/p/pl?d=RTX+3080'

driver.maximize_window()
driver.get(url)

card_path = '/html/body/div[8]/div[3]/section/div/div/div[1]/div/dl[1]/dd/ul[2]/li/a'
desktop_graphics_cards = driver.find_element(By.XPATH, card_path)
desktop_graphics_cards.click()
time.sleep(5)

graphics_card = []
shipping_cost = []
price = []
total_cost = []

for item in range(9):
    try:
        #next_page_click = driver.find_element(By.XPATH("//button[text() = '{item + 1}']"))
        print(next_page_click)
        next_page_click.click()
    except:
        pass

Prophet · Accepted Answer · 2022-01-09 21:49:09Z

1

The pagination buttons are out of the initially visible area.
In order to click these elements you will have to scroll the page until the element appears.
Also, you will need to click next page buttons starting from 2 up to 9 (including) while you trying to do this with numbers from 1 up to 9.
I think this should work better:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import NoSuchElementException
import pandas as pd
import time

options = Options()

PATH = 'C://Program Files (x86)//chromedriver.exe'

driver = webdriver.Chrome(PATH)

url = 'https://www.newegg.com/p/pl?d=RTX+3080'
actions = ActionChains(driver)

driver.maximize_window()
driver.get(url)

card_path = '/html/body/div[8]/div[3]/section/div/div/div[1]/div/dl[1]/dd/ul[2]/li/a'
desktop_graphics_cards = driver.find_element(By.XPATH, card_path)
desktop_graphics_cards.click()
time.sleep(5)

graphics_card = []
shipping_cost = []
price = []
total_cost = []

for item in range(2,10):
    try:
        next_page_click = driver.find_element(By.XPATH(f"//button[text() = '{item}']"))
        actions.move_to_element(next_page_click).perform()
        time.sleep(2)
        #print(next_page_click) - printing a web element itself will not give you usable information
        next_page_click.click()
        #let the next page loaded, it takes some time
        time.sleep(5)
    except:
        pass

edited Jan 9, 2022 at 21:49

answered Jan 9, 2022 at 20:05

Prophet

33.5k28 gold badges58 silver badges90 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Wasian Over a year ago

Thank you for the suggestion @Prophet ! Unfortunately I still get the same result and the script never moves beyond the first page. Do you have any other suggestions or if I need to provide some other information that might make what I'm doing wrong more clear? From what I can tell, the script for some reason doesn't scroll down to the buttons at the bottom.

Prophet Over a year ago

Remove the except to see what error is thrown

Wasian Over a year ago

It looks like it returns a Type Error : next_page_click = driver.find_element(By.XPATH("//button[text() = '{item}']")) TypeError: 'str' object is not callable. I guess I need to use a different XPATH then? I'm not sure which one would get me the clickable button though

Prophet Over a year ago

We forgot the "f-strings" f... :) let me know if now it working

Wasian Over a year ago

Works perfectly, thank you!

|

Collectives™ on Stack Overflow

Selenium Web Scraping: Find element by text not working in script

1 Answer 1

6 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Related