Python Selenium: WebElement not working at all after Login in website because DOM is updated

Question

I'm very new to Selenium and scraping in general and I need some help please. I'm trying to automate some of my work through a website that provides data and queries using Selenium.

The problem is that after the Login, the DOM gets refreshed and apparently that disconects my WebElement, as I no longer am able of doing any find_element, get or any type of function after the login is done. I am even trying to do it manually so that any refresh, wait time load, etc. is not needed because it is under human supervision :(

The same issues happens if instead of doing driver.click() I use driver.submit()

The code is as following:

# import web driver
from selenium import webdriver
import time as time

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# specifies the path to the chromedriver.exe
driver = webdriver.Chrome('/Users/user/bin/chromedriver')

# driver.get method() will navigate to a page given by the URL address
driver.get('https://app.dnbhoovers.com/')

# locate email form by_class_name
username = driver.find_element_by_xpath('//*[@id="username"]')
# send_keys() to simulate key strokes
username.send_keys(username)

#Sleep some time
time.sleep(0.5)

# locate submit button by_class_name
log_in_button = driver.find_element_by_class_name('continue-btn')

# .click() to mimic button click
log_in_button.click()


#Wait for some seconds as password loads after clicking the contiunue bottom
#try:
driver = WebDriverWait(driver, 5).until(
        EC.presence_of_element_located((By.ID, "password"))
    )
time.sleep(3)


# locate password form by_class_name
password = driver.find_element_by_xpath('//*[@id="password"]')

time.sleep(0.7)
# send_keys() to simulate key strokes
password.send_keys(password)

#Sleep some time
time.sleep(0.7)

# .click() to mimic button click and Sign in
log_in_button.click()

After the login is where everything goes down, and if I try to type the following functions I get these errors (even after waiting for the page to load):

search = driver.find_element_by_xpath("//input[@class='search-query']")
StaleElementReferenceException: stale element reference: element is not attached to the page document

Or if I even try this simple task I get the following:

driver.get("http://www.google.com")
AttributeError: 'WebElement' object has no attribute 'get'

And this goes on, if I try to refresh or go down with the keyboard I get:

webdriver.ActionChains(driver).send_keys(Keys.DOWN).perform()
AttributeError: 'WebElement' object has no attribute 'w3c'

Even if I add this lines after the .click() command to tell the driver to wait for the search input to appear, I get errors

# ... previous code...#
log_in_button.click()

#New lines of code
WebDriverWait(driver, 20).until(
    EC.presence_of_element_located((By.XPATH, "//input[@class='search-query']"))
    )

StaleElementReferenceException: stale element reference: element is not attached to the page document

Refreshing the browser will not make any difference either.

If I inspect the element of the webiste, after the login the changes completely and I see some ""<meta name="robots" content="noindex, nofollow>"" also.

I'm really stuck right now so any help is deeply appreciated!

Thanks for your time and for reading this long post! You can have a potato :)

You may need to switch to iframe. Without html code it is hard to tell. — vitaliis
– vitaliis, Commented Apr 11, 2021 at 21:38

C. Peck · Accepted Answer · 2021-04-11 22:48:49Z

1

I think the problem is this

driver = WebDriverWait(driver, 5).until(
        EC.presence_of_element_located((By.ID, "password"))
    )

WebDriverWait() returns a WebElement and you don't want to define driver as such.

driver.get("http://www.google.com")
AttributeError: 'WebElement' object has no attribute 'get'

This is telling you that driver is a WebElement, which you don't want.

answered Apr 11, 2021 at 22:48

C. Peck

3,7193 gold badges21 silver badges38 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Felipe Rodriguez Walker Over a year ago

Thanks! But it doesn't work either. If I delete "driver = " to that line I still don't get the StaleElement error.

vitaliis · Accepted Answer · 2021-04-12 13:41:13Z

0

Try to wait till it is clickable:

elem = wait.until(EC.element_to_be_clickable((By.XPATH, "//input[@class='search-query']")))
elem.click()

Try 2: If there is a running JQuery try running this script before click:

wait.until(lambda driver: driver.execute_script('return jQuery.active') == 0)
wait.until(lambda driver: driver.execute_script('return document.readyState') == 'complete')

edited Apr 12, 2021 at 13:41

answered Apr 12, 2021 at 12:22

vitaliis

4,1975 gold badges26 silver badges48 bronze badges

6 Comments

Felipe Rodriguez Walker Over a year ago

Thanks for your answer!! But unfortunately, the error is still the same. There is apparenly no webdriver as I get a TimeoutException. The wait.until code didn't work so I tried with "elem = WebDriverWait(driver, 10).until(" and I still didn't get any result. I am also trying to switch iframe but I get AttributeError: 'WebElement' object has no attribute 'switch_to'

vitaliis Over a year ago

Does new page has a different domain?

vitaliis Over a year ago

I mean is the first part of page url different?

Felipe Rodriguez Walker Over a year ago

The URL is the same but the DOM changes completely

vitaliis Over a year ago

I added one more option, but without page source it is hard to tell

|

Collectives™ on Stack Overflow

Python Selenium: WebElement not working at all after Login in website because DOM is updated

2 Answers 2

1 Comment

6 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

6 Comments

Your Answer

Sign up or log in

Post as a guest

Related