Select from dropdown using Selenium and Python

Question

With the help of Selenium and Python. I wanted to crawl a webpage which has a nested drop-down menu. I am posting only the nested section below:

<div class="dropDown active" data-dropdown-block="FOOTBALL_COMPSEASON" data-dropdown-default="All Seasons">
    <div class="label" id="dd-FOOTBALL_COMPSEASON">Filter by Season</div> 
    <div class="current" data-dropdown-current="FOOTBALL_COMPSEASON" role="button" tabindex="0" aria-expanded="false" aria-labelledby="dd-FOOTBALL_COMPSEASON" data-listen-keypress="true" data-listen-click="true">
        2018/19
    </div>
    <ul class="dropdownList" data-dropdown-list="FOOTBALL_COMPSEASON" role="listbox" aria-labelledby="dd-FOOTBALL_COMPSEASON" data-listen-keypress="true" data-listen-click="true">
        <li role="option" tabindex="0" data-option-name="All Seasons" data-option-id="-1" data-option-index="-1">
             All Seasons
        </li> 
        <li role="option" tabindex="0" data-option-name="2018/19" data-option-id="210" data-option-index="0">
            2018/19
         </li>
         <li role="option" tabindex="0" data-option-name="2017/18" data-option-id="79" data-option-index="1">
              2017/18
         </li>
         <li role="option" tabindex="0" data-option-name="2016/17" data-option-id="54" data-option-index="2">
             2016/17
         </li>
    </ul>
</div>

Here is the screenshot of how it looks:

So, I wanted to make the crawler click the drop-down and select 2017/18.

I first tried this:

driver.get(_url)
select_element = driver.find_elements_by_class_name("dropdownList")[1]

As the class dropdownList is used multiple times in the HTML and my desired element is in the second position, i.e. <ul class="dropdownList".... is the second time the class dropdown is used, so I used [1] to get the second child.

But then I get this error:

File "shots_2017_18.py", line 15, in shots_2017_18 select_element = driver.find_elements_by_class_name("dropdownList")1 IndexError: list index out of range

What should I change or do so that the crawler can select the 2017/18 item from the dropdown list and can crawl?

Are you able to click on drop down? I see it is not made using select and span tag though. — cruisepandey
– cruisepandey, Commented Jul 21, 2018 at 10:16
well actually that's what I'm trying to do. But resulting above error and also if you check the answer below that also generating error that I have posted below. — sphoenix
– sphoenix, Commented Jul 21, 2018 at 10:18

cruisepandey · Accepted Answer · 2018-07-21 12:17:10Z

If you are able to click on drop down , using python and selenium. Then you can try this code :

UPDATE :

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC 
from selenium.webdriver.common.action_chains import ActionChains
import time


driver   = webdriver.Chrome(executable_path = r'C:/Users/user***/Downloads/chromedriver_win32/chromedriver.exe')
driver.maximize_window()

wait = WebDriverWait(driver,40)

driver.get("https://www.premierleague.com/stats/top/players/goals")  

wait.until(EC.visibility_of_element_located((By.ID, 'dd-FOOTBALL_COMPSEASON')))

time.sleep(5)
drop_down_click = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div.current[data-dropdown-current='FOOTBALL_COMPSEASON']")))
drop_down_click.click()

options = driver.find_elements_by_css_selector("ul[data-dropdown-list='FOOTBALL_COMPSEASON'] li")

for option in options:
  if "2017/18" in option.text.strip():
    option.click()

UPDATE1 :

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC 
from selenium.webdriver.common.action_chains import ActionChains
import time

driver   = webdriver.Chrome(executable_path = r'C:/Users/user***/Downloads/chromedriver_win32/chromedriver.exe')
driver.maximize_window()

wait = WebDriverWait(driver,40)

driver.get("https://www.premierleague.com/stats/top/players/total_scoring_att")


cookie_button = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div.btn-primary.cookies-notice-accept")))
ActionChains(driver).move_to_element(cookie_button)
driver.execute_script('arguments[0].click();', cookie_button)
wait.until(EC.visibility_of_element_located((By.ID, 'dd-FOOTBALL_COMPSEASON')))

time.sleep(5)
drop_down_click = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div.current[data-dropdown-current='FOOTBALL_COMPSEASON']")))
drop_down_click.click()

options = driver.find_elements_by_css_selector("ul[data-dropdown-list='FOOTBALL_COMPSEASON'] li")

for option in options:
  if "2017/18" in option.text.strip():
    option.click()

Explanation :

An explicit wait is code you define to wait for a certain condition to occur before proceeding further in the code. The worst case of this is Thread.sleep(), which sets the condition to an exact time period to wait. There are some convenience methods provided that help you write code that will wait only as long as required. WebDriverWait in combination with ExpectedCondition is one way this can be accomplished.

More about explicit wait, can be found here

Arashsyh · Accepted Answer · 2018-07-21 10:40:03Z

0

when you get index out of range in this situation it means elements found are None or just one, since your written code is correct i think you entered wrong URL. but if URL is correct so you can use XPATH to find Your appropriate element. try this code:

select_element = driver.find_element_by_xpath("//li[@data-option-name='2017/18']")

edited Jul 21, 2018 at 10:40

answered Jul 21, 2018 at 10:21

Arashsyh

6571 gold badge10 silver badges16 bronze badges

2 Comments

sphoenix Over a year ago

I am getting the error:

raise exception_class(message, screen, stacktrace) selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//li[@data-option-name='2017/18']"}

Arashsyh Over a year ago

it is not sufficient but try: li_tags = driver.find_elements_by_tag_name("li") for item item in li_tags: if item.text == '2017/18': item.click()

Collectives™ on Stack Overflow

Select from dropdown using Selenium and Python

2 Answers 2

Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related