looping through a dropdown menu using Selenium and Python

Question

I'm trying to loop through a dropdown menu on at this url: https://www.accuform.com/safety-sign/danger-danger-authorized-personnel-only-MADM006

So, for example, the first dropdown menu - under options - lists out different materials and I want to select each one in turn and then gather some other information from the webpage before moving on to the next material. Here is my current code:

driver = webdriver.Firefox()
driver.get('https://www.accuform.com/safety-sign/danger-danger-authorized-personnel-only-MADM006')

time.sleep(3)

driver.find_element_by_id('x-mark-icon').click()

select = Select(driver.find_element_by_name('Wiqj7mb4rsAq9LB'))
options = select.options
optionsList = []

driver.find_elements_by_class_name('select-wrapper')[0].click()

element = driver.find_element_by_xpath("//select[@name='Wiqj7mb4rsAq9LB']")
actions = ActionChains(driver)
actions.move_to_element(element).perform()

# driver.execute_script("arguments[0].scrollIntoView();", element)


for option in options: #iterate over the options, place attribute value in list
    optionsList.append(option.get_attribute("value"))

for optionValue in optionsList:
    print("starting loop on option %s" % optionValue)
    # select = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//select[@name='Wiqj7mb4rsAq9LB']")))
    # select = Select(select)
    select.select_by_value(optionValue)

I started with just the loop, but got this error:

ElementNotInteractableException: Message: Element <option> could not be scrolled into view

I then added the webdriverwait and get a TimeoutException error.

I then realized I should probably click on the wrapper in which the dropdown is held, so I added the click, which does pup up the menu, but I still got the TimeoutException.

So I thought, maybe I should move to the element, which I tried with the action chain lines and I got this error

WebDriverException: Message: TypeError: rect is undefined

I tried to avoid that error by using this code instead:

    # driver.execute_script("arguments[0].scrollIntoView();", element)

Which just resulted in the timeoutexception again.

I pretty new to Python and Selenium and have basically just been modifying code from SO answers to similar questions, but nothing has worked.

I'm using python 3.6 and the current versions of Selenium and firefox webdriver.

If anything is unclear or if you need more info just let me know.

Thanks so much!

EDIT: Based on the answer and comments by Kajal Kunda, I've updated my code to the following:

`material_dropdown = driver.find_element_by_xpath("//input[@class='select- 
dropdown']")

driver.execute_script("arguments[0].click();", material_dropdown)

materials=driver.find_elements_by_css_selector("div.select-wrapper 
ul.dropdown-content li")



for material in materials:

    # material_dropdown = 
    driver.find_element_by_xpath("//input[@class='select-dropdown']")

    # driver.execute_script("arguments[0].click();", material_dropdown)

    # materials=driver.find_elements_by_css_selector("div.select-wrapper ul.dropdown-content li")

    material_ele=material.find_element_by_tag_name('span')

if material_ele.text!='':

    material_ele.click()

    time.sleep(5)

    price = driver.find_element_by_class_name("dataPriceDisplay")

    print(price.text)`

The result is that it successfully prints the price for the first type of material, but then it returns: StaleElementReferenceException: Message: The element reference of <li class=""> is stale;...

I've tried variations of having the hashed out lines in and outside of the loop, but always get a version of the StaleElementReferenceException error.

Any suggestions?

Thanks!

Were you able to solve this problem?

Moises Felipe
– Moises Felipe

2020-05-04 22:30:39 +00:00
Commented May 4, 2020 at 22:30 — Moises Felipe
– Moises Felipe, Commented May 4, 2020 at 22:30

QHarr · Accepted Answer · 2019-02-27 21:48:49Z

3

You could do the whole thing with requests. Grab the drop down list from the options listed in drop down then concatenate the value attributes into requests url that retrieves json containing all the info on the page. Same principle applies for adding in other dropdown values. The ids for each drop down selection are the value attributes of the options in the drop down and appear in the url I show separated by // for each drop down selection.

import requests
from bs4 import BeautifulSoup as bs

url = 'https://www.accuform.com/product/getSku/danger-danger-authorized-personnel-only-MADM006/1/false/null//{}//WHFIw3xXmQx8zlz//6wr93DdrFo5JV//WdnO0RpwKpc4fGF'
startURL = 'https://www.accuform.com/safety-sign/danger-danger-authorized-personnel-only-MADM006'

res = requests.get(startURL)
soup = bs(res.content, 'lxml')
materials = [item['value'] for item in soup.select('#Wiqj7mb4rsAq9LB option')]
sizes = [item['value'] for item in soup.select('#WvXESrTyQjM3Ciw option')]
languages = [item['value'] for item in soup.select('#WUYWGMePtpmpmhy option')]
units = [item['value'] for item in soup.select('#W91eqaJ0WPXwe9b option')]

for material in materials:
    data = requests.get(url.format(material)).json()
    soup = bs(data['dataMaterialBullets'], 'lxml')
    lines = [item.text for item in soup.select('li')]
    print(lines)
    print(data['dataPriceDisplay'])     
    # etc......

Sample of JSON:

edited Feb 27, 2019 at 21:48

answered Feb 27, 2019 at 21:34

QHarr

84.5k14 gold badges58 silver badges105 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Jarod Jacobs Over a year ago

Thanks QHarr, while this seems like an interesting approach, I'm not really interested in moving to a different package. I've been learning Selenium for a bit now and don't want to start over with beautifulsoup. Even though it may be better, I'm in the middle of this project and it doesn't seem like a good time to start over.

QHarr Over a year ago

No worries. I’ll see what answers are here by tomorrow and perhaps I’ll add a selenium answer.

KunduK · Accepted Answer · 2019-02-28 18:09:48Z

1

Try the below code.It should work.

    driver = webdriver.Firefox()
    driver.get('https://www.accuform.com/safety-sign/danger-danger-authorized-personnel-only-MADM006')

    time.sleep(3)
    driver.find_element_by_id('x-mark-icon').click()

    material_dropdown = driver.find_element_by_xpath("//input[@class='select-dropdown']")
    driver.execute_script("arguments[0].click();", material_dropdown)

    #Code for material dropdown
    materials=driver.find_elements_by_css_selector("div.select-wrapper ul.dropdown-content li")


    material_optionsList = []
    for material in materials:
        material_ele=material.find_element_by_tag_name('span')
        if material_ele.text!='':
          material_optionsList.append(material_ele.text)

    print(material_optionsList)

    driver.execute_script("arguments[0].click();", material_dropdown)


    size_dropdown = driver.find_element_by_xpath("(//input[@class='select-dropdown'])[2]")
    driver.execute_script("arguments[0].click();", size_dropdown)

    #Code for size dropdown
    Sizes=driver.find_elements_by_css_selector("div.select-wrapper ul.dropdown-content li")
    size_optionsList = []
    for size in Sizes:
        size_ele=size.find_element_by_tag_name('span')
        if size_ele.text!='':
            size_optionsList.append(size_ele.text)



driver.execute_script("arguments[0].click();", size_dropdown)

Output :

[u'Adhesive Vinyl', u'Plastic', u'Adhesive Dura-Vinyl', u'Aluminum', u'Dura-Plastic\u2122', u'Aluma-Lite\u2122', u'Dura-Fiberglass\u2122', u'Accu-Shield\u2122']

Hope you will do the remaining.Let me know if it works for you.

EDIT Code for loop through and get the price value of materials.

for material in range(len(materials)):
    material_ele=materials[material]

    if material_ele.text!='':
       #material_optionsList.append(material_ele.text)
       #material_ele.click()
       driver.execute_script("arguments[0].click();", material_ele)
       time.sleep(2)
       price = driver.find_element_by_id("priceDisplay")
       print( price.text)
       time.sleep(2)
       material_dropdown = driver.find_element_by_xpath("//input[@class='select-dropdown']")
       driver.execute_script("arguments[0].click();", material_dropdown)
       materials = driver.find_elements_by_css_selector("div.select-wrapper ul.dropdown-content li")
       material+=2

Output :

$8.31
$9.06
$13.22
$15.91
$15.91

edited Feb 28, 2019 at 18:09

answered Feb 27, 2019 at 19:59

KunduK

33.4k5 gold badges19 silver badges42 bronze badges

14 Comments

JeffC Over a year ago

Rather than just rewriting the OP's entire script and dropping it in an answer, it would be nice for the OP and future readers if you would explain what you changed and why so others can learn from it.

KunduK Over a year ago

@JeffC - You are right.I should mentioned what i have done.I have just changed the locator and mouse hover on the element but I am not sure whether it would work every time.

JeffC Over a year ago

You need to put that description in the answer itself. Not everyone reads all comments, nor should they have to to fully understand your answer.

Jarod Jacobs Over a year ago

Thanks Kajal, but I'm trying to loop through the materials, size, etc. dropdowns, not the shop by category one. I tried to modify your code to work with one of those, but I keep getting a nosuchelementexception with this line: "element = driver.find_element_by_xpath("//span[class='caret']")"

KunduK Over a year ago

Your xpath is wrong.When you pass any attribute you should use @ . element = driver.find_element_by_xpath("//span[@class='caret']") Now try.

|

Collectives™ on Stack Overflow

looping through a dropdown menu using Selenium and Python

2 Answers 2

2 Comments

14 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

14 Comments

Your Answer

Sign up or log in

Post as a guest

Related