'Suspected' Odd behavior from output of Selenium using Python

Question

Good Day, I am running the following snippet and find the following behavior which I am not sure if it is correct or not:

for url in links:
        driver.get(url)
        date = driver.find_elements_by_xpath("""//*[contains(@id, 'node')]/div[1]/div[1]/div[2]/div/span""")
        secref1 = driver.find_elements_by_xpath("""/html/body/div[3]/div/section/div[2]/div/section/div/section/div/article/div[1]/div[3]/div[2]/div""")
        secref2 = driver.find_elements_by_xpath("""/html/body/div[3]/div/section/div[2]/div/section/div/section/div/article/div[1]/div[4]/div[2]/div""")

        if not secref2:
            secref2.append("Null")
        else:
            secref2 = secref2

        num_page_items = len(date)

        for i in range(num_page_items):
            print secref2

driver.close()

I expect "secref2" to be missing from the webpage, hence the IF/ELSE.

my output is as follows when running the script:

DevTools listening on ws://127.0.0.1:64592/devtools/browser/da7ab0e6-e0e9-4edb-963a-913b38c6f4dd
['Null']
[<selenium.webdriver.remote.webelement.WebElement (session="a7bc63bef087357d1510c3b28ec8db87", element="0.14518628426304736-4")>]
[<selenium.webdriver.remote.webelement.WebElement (session="a7bc63bef087357d1510c3b28ec8db87", element="0.6063690703515521-4")>]
[<selenium.webdriver.remote.webelement.WebElement (session="a7bc63bef087357d1510c3b28ec8db87", element="0.16122194044687665-7")>]
[<selenium.webdriver.remote.webelement.WebElement (session="a7bc63bef087357d1510c3b28ec8db87", element="0.7547639796767653-4")>]
[<selenium.webdriver.remote.webelement.WebElement (session="a7bc63bef087357d1510c3b28ec8db87", element="0.768240568661338-16")>]
[<selenium.webdriver.remote.webelement.WebElement (session="a7bc63bef087357d1510c3b28ec8db87", element="0.3077014556092601-4")>]
[<selenium.webdriver.remote.webelement.WebElement (session="a7bc63bef087357d1510c3b28ec8db87", element="0.9689075758046188-4")>]
[<selenium.webdriver.remote.webelement.WebElement (session="a7bc63bef087357d1510c3b28ec8db87", element="0.09545508090332766-4")>]
[<selenium.webdriver.remote.webelement.WebElement (session="a7bc63bef087357d1510c3b28ec8db87", element="0.068763767350847-4")>]

I see the first "Null" however subsequent entries look to be some sort of output.

If I try:

        for i in range(num_page_items):
        print secref2[i].text

I get the following error:

DevTools listening on ws://127.0.0.1:64788/devtools/browser/df696310-30cf-4833-89fa-fac28e6b3bb0
Traceback (most recent call last):
  File "test.py", line 54, in <module>
    print secref2[i].text
AttributeError: 'str' object has no attribute 'text'

Any help with this would be appreciated.

Lucas Tierney · Accepted Answer · 2018-11-28 05:25:55Z

1

You're iterating twice. So in the first URL, you get Null. In subsequent URLs you get a list of webelements. You can't print secref2[i].text because the first time you hit it, it is "Null" and "Null" is a str.

Did you mean to assign something else to the variable here? I don't know why you'd assign the variable to itself.

else:
    secref2 = secref2

edited Nov 28, 2018 at 5:25

answered Nov 28, 2018 at 5:10

Lucas Tierney

2,57317 silver badges13 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

qbbq Over a year ago

thanks Lucas - not sure if was correct, but my reasoning was the following. If secref2 is not available, make the variable "Null" otherwise make the variable whatever selenium was able to get. would you suggest "pass" ?

qbbq Over a year ago

something I didn't add to this is the fact that driver.get(url), sequentially opens 10 URL's and extracts the pieces of information from the Xpath's I specify. On on of the 10 URL's I expect "secref2" to not be available. hence I want "Null" as the value in that instance

Lucas Tierney Over a year ago

But it IS doing that. You're assigning secref2 = driver.find_elements_by_xpath which will give you a list of webelements or an empty list. secref2 = secref has no effect. Your output shows that it's working as you intend, on the first URL it gives you an empty list, so you add "Null", on the next 9 URLs it finds one webelement and prints the list containing that one element. If you want the text from the webelement found, use secref2 = secref2[0].text

qbbq Over a year ago

I moved the if/else to the portion the script where I will be writing to CSV. this is what I have:

with open('results.csv', 'a') as f:             for i in range(num_page_items):                     if not secref2:                         secref2.append("Null")                     else:                         secref2 = secref2[i].text                     f.write(date[i].text + "#" + secref1[i].text + "#" + secref2[i].text +"\n")

however I get the error: AttributeError: 'str' object has no attribute 'text' which is correct - however how do I get around it ?

chitown88 Over a year ago

as stated above, when secref2[i] is "Null", it is a string. so doing secref2[i].text is basically saying take that object (a str) and apply .text . But there is no such attribute for a str. So at some point, some combination of your date[i], secref2[i], or secref1[i] is a str and you're trying to apply the .text attribute to it so do something like if type(secref1[i]) != str: secref1[i] = secref1[i].text etc. then you could do your else: secref2 = secref2[i] f.write(date[i] + "#" + secref1[i] + "#" + secref2[i] +"\n") and drop the .text since you made sure they're all strings

Collectives™ on Stack Overflow

'Suspected' Odd behavior from output of Selenium using Python

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related