How to extract webelement with Selenium Webdriver in Python

Question

Hello I am trying to extract the date for each row elements of the main table. The date element is in a tag named :

<td style="display: none;">

Here is my script :

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

options = Options()
options.headless = True
options.add_argument("window-size=1400,800")
options.add_argument("--no-sandbox")
options.add_argument("--disable-gpu")
options.add_argument("start-maximized")
options.add_argument("enable-automation")
options.add_argument("--disable-infobars")
options.add_argument("--disable-dev-shm-usage")

#We extract sport event data at this webpage
url = "https://www.coteur.com/cotes-foot.php"

driver = webdriver.Chrome(options=options)
driver.get(url)

date = [my_elem.text for my_elem in WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.XPATH, '//td[contains(@style, "display: none;")]')))]

print(date)

I have this output :

Traceback (most recent call last):
  File "./soccer_scraper_historic.py", line 36, in <module>
    date = [my_elem.text for my_elem in WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.XPATH, '//td[contains(@style, "display: none;")]')))]
  File "/usr/local/lib/python3.8/dist-packages/selenium/webdriver/support/wait.py", line 80, in until
    raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:

Maybe you can help to solve this issue. Thanks

Sers · Accepted Answer · 2021-02-13 12:44:22Z

1

With visibility_of_all_elements_located you're waiting for elements with display: none; - not visible to be visible. Also better to get columns by position rather than display: none; attribute because it's not unique. You cannot get text from not visible elements, so you can use attribute textContent.

driver = webdriver.Chrome(options=options)
wait = WebDriverWait(driver, 10)
driver.get(url)

dates = [d.get_attribute("textContent") for d in
         wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "#mediaTable tr[role=row] td:nth-child(4)")))]

answered Feb 13, 2021 at 12:44

Sers

12.3k2 gold badges14 silver badges33 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to extract webelement with Selenium Webdriver in Python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related