Cannot scrape input from dynamic dropdown menu due to <span> (Python - Selenium)

Question

this is a somewhat discussed problem, however I haven't find a solution in other posts.

I am creating an automation to submit parcels, using Selenium on Python, for this website: https://www.mondospedizioni.com

To select Country and City, I need to add an input on a text field. The text field is actually a search field, that opens a dropdown menu upon adding the key.

I fail to select the option and close the dropdown menu, thus filling the "Country"/"City" field and moving fwd.

The div I refer to (Country):

<span class="select2-selection__rendered" id="select2-nazione_mittente-container" title="Selezionare Nazione di ritiro">Selezionare Nazione di ritiro</span>

The one for City:

<span class="select2-selection__placeholder">Inserire il CAP e selezionare la località</span>

This Post uses Select, but because of the tags I receive error:

Select only works on <select> elements, not on <span>.

I found another suggestion in this post from Anzel and even this one did not work for me.

My code gets stuck when I try to enter the sender's city, and the error is given to me when the code calls the input field for the receiver's country. My code:

import pandas as pd
import numpy as np
import time
from datetime import datetime, date

from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
from selenium.webdriver.support.select import Select
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys

import itertools

driver = webdriver.Chrome(executable_path...)
url = "https://www.mondospedizioni.com/"
driver.get(url)
driver.set_page_load_timeout(-1) #Makes sure the webpage is fully loaded before scraping. Thus avoiding errors.

#All Page Components
hoCapito =driver.find_element(By.XPATH, '//*[@id="cookie-bar"]/p/a[1]')

countryFrom = driver.find_element(By.XPATH, '//*[@id="select2-nazione_mittente-container"]')
cityFrom = driver.find_element(By.XPATH, '//*[@id="loc_mittente"]')
postFrom = driver.find_element(By.XPATH, '//*[@id="cap_mittente"]')

countryTo = driver.find_element(By.XPATH, '//*[@id="select2-nazione_destinatario-container"]')
cityTo = driver.find_element(By.XPATH, '//*[@id="loc_destinatario"]')
postTo = driver.find_element(By.XPATH, '//*[@id="cap_destinatario"]')

myWeight = driver.find_element(By.XPATH, '//*[@id="peso1"]')
myLength = driver.find_element(By.XPATH, '//*[@id="l1"]')
myWidth = driver.find_element(By.XPATH, '//*[@id="p1"]')
myHeight = driver.find_element(By.XPATH, '//*[@id="h1"]')

calculate = driver.find_element(By.XPATH, '//*[@id="btn-calcola"]')


#MyKeys
driver = webdriver.Chrome(executable_path=r"C:\Users\AndreaPaviglianiti\Jupyter-2022\webdrivers\chromedriver.exe")
url = "https://www.mondospedizioni.com/"
driver.get(url)
driver.set_page_load_timeout(-1) #Makes sure the webpage is fully loaded before scraping. Thus avoiding errors.

hoCapito =driver.find_element(By.XPATH, '//*[@id="cookie-bar"]/p/a[1]')

countryFrom = driver.find_element(By.XPATH, '//*[@id="select2-nazione_mittente-container"]')
cityFrom = driver.find_element(By.XPATH, '//*[@id="select2-loc_cap_mittente-container"]/span')
postFrom = driver.find_element(By.XPATH, '//*[@id="cap_mittente"]')

countryTo = driver.find_element(By.XPATH, '//*[@id="select2-nazione_destinatario-container"]')
cityTo = driver.find_element(By.XPATH, '//*[@id="loc_destinatario"]')
postTo = driver.find_element(By.XPATH, '//*[@id="cap_destinatario"]')

#Accept Cookies
hoCapito.click()


countryFrom.click()
select_countryFrom = driver.find_element(By.CLASS_NAME, 'select2-search__field').send_keys('Italia')
confirm_countryFrom = driver.find_element(By.XPATH, '//*[@id="select2-nazione_mittente-results"]').click()

cityFrom.click()
select_cityFrom = driver.find_element(By.CLASS_NAME, 'select2-search__field').send_keys('Partinico')
confirm_cityFrom = driver.find_element(By.XPATH, '//*[@id="select2-loc_cap_mittente-results"]/li/div').click()


countryTo.click()
select_countryTo = driver.find_element(By.CLASS_NAME, 'select2-search_field').send_keys('Repubblica Slovacca')
confirm_countryTo = driver.find_element(By.XPATH, '//*[@id="select2-nazione_destinatario-result-885w-Repubblica Slovacca"]').click()

cityTo.send_keys('Bratislava')
postFrom.send_keys(82101)

myWeight.send_keys(30)
myLength.send_keys(47)
myWidth.send_keys(25)
myHeight.send_keys(38)

calculate.click()
time.sleep(3)
driver.set_page_load_timeout(-1)

My error (now - based on my latest attempt):

<ipython-input-77-dd29b60fadf7>:2: DeprecationWarning: executable_path has been deprecated, please pass in a Service object
  driver = webdriver.Chrome(executable_path=r"C:\Users\AndreaPaviglianiti\Jupyter-2022\webdrivers\chromedriver.exe")

---------------------------------------------------------------------------
NoSuchElementException                    Traceback (most recent call last)
<ipython-input-77-dd29b60fadf7> in <module>
     25 cityFrom.click()
     26 select_cityFrom = driver.find_element(By.CLASS_NAME, 'select2-search__field').send_keys('Partinico')
---> 27 confirm_cityFrom = driver.find_element(By.XPATH, '//*[@id="select2-loc_cap_mittente-results"]/li/div').click()
     28 #confirm_cityFrom = driver.find_element(By.XPATH, '//*[@id="select2-loc_cap_mittente-results"]/li/div/text()').click()
     29 #confirm_cityFrom = driver.find_element(By.XPATH, '/html/body/span/span/span[2]').click()

~\Miniconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py in find_element(self, by, value)
   1242             value = '[name="%s"]' % value
   1243 
-> 1244         return self.execute(Command.FIND_ELEMENT, {
   1245             'using': by,
   1246             'value': value})['value']

~\Miniconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py in execute(self, driver_command, params)
    422         response = self.command_executor.execute(driver_command, params)
    423         if response:
--> 424             self.error_handler.check_response(response)
    425             response['value'] = self._unwrap_value(
    426                 response.get('value', None))

~\Miniconda3\lib\site-packages\selenium\webdriver\remote\errorhandler.py in check_response(self, response)
    245                 alert_text = value['alert'].get('text')
    246             raise exception_class(message, screen, stacktrace, alert_text)  # type: ignore[call-arg]  # mypy is not smart enough here
--> 247         raise exception_class(message, screen, stacktrace)
    248 
    249     def _value_or_default(self, obj: Mapping[_KT, _VT], key: _KT, default: _VT) -> _VT:

NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[@id="select2-loc_cap_mittente-results"]/li/div"}
  (Session info: chrome=98.0.4758.81)
Stacktrace:
Backtrace:
    Ordinal0 [0x007F7AC3+2587331]
    Ordinal0 [0x0078ADD1+2141649]
    Ordinal0 [0x00683BB8+1063864]
    Ordinal0 [0x006B01CE+1245646]
    Ordinal0 [0x006B03CB+1246155]
    Ordinal0 [0x006DA612+1418770]
    Ordinal0 [0x006C86D4+1345236]
    Ordinal0 [0x006D8A0A+1411594]
    Ordinal0 [0x006C84A6+1344678]
    Ordinal0 [0x006A53F6+1201142]
    Ordinal0 [0x006A62E6+1204966]
    GetHandleVerifier [0x0099DF22+1680738]
    GetHandleVerifier [0x00A50DBC+2413564]
    GetHandleVerifier [0x0088D151+563089]
    GetHandleVerifier [0x0088BF13+558419]
    Ordinal0 [0x0079081E+2164766]
    Ordinal0 [0x00795508+2184456]
    Ordinal0 [0x00795650+2184784]
    Ordinal0 [0x0079F5BC+2225596]
    BaseThreadInitThunk [0x77CDFA29+25]
    RtlGetAppContainerNamedObjectPath [0x77E27A9E+286]
    RtlGetAppContainerNamedObjectPath [0x77E27A6E+238]

EDIT To clarify the matter I am now adding two printscreens that focus on the elements I cannot manage - the Sender's Locality.

It is more advanced to what I am used to (and I am fairly new with Selenium).

Which is this element //*[@id="select2-loc_cap_mittente-results"]/li/div? What are the manual steps to repro this issue? — undetected Selenium
– undetected Selenium, Commented Feb 9, 2022 at 19:17
When entering the city name as key for <input class="select2-search__field"> (the Sender's Locality), it opens a dropdown menu that does not work as the others. For the countries I could retrieve the name even without Arundeep's solution posted below. For the city it's a different story because the city&postcode text is nested in the html. I edited my request so you can see the actual code I am dealing with. — Andrea P.
– Andrea P., Commented Feb 10, 2022 at 9:20

Arundeep Chohan · Accepted Answer · 2022-02-09 23:11:10Z

1

#select2-nazione_mittente-container

Simply click the above element

.select2-search__field

Send keys to the above element

//li[@class='select2-results__option' and contains(@id,'Italia')]

Then proceed to click on the li tag with that value.

It's how to deal with any non Select dropdown element.

answered Feb 9, 2022 at 23:11

Arundeep Chohan

9,9895 gold badges17 silver badges36 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Andrea P. Over a year ago

Hi, that is a problem that I sorted because it is easier to get to the id but the same does not apply for the dropdown menu of the selected city. As described above, my issue occurs between the sender's city and the receiver's country.

Andrea P. · Accepted Answer · 2022-02-11 14:23:29Z

0

It turns out that it is actually that I need to target and that works only if I get to have only one option in the dropdown menu.

I came up with this solution and it works, for I was able to finish to automate the form:

#Sender's City
cityFrom = driver.find_element(By.XPATH, '//*[@id="select2-loc_cap_mittente-container"]').click()
select_cityFrom = driver.find_element(By.XPATH, '/html/body/span/span/span[1]/input').send_keys('Partanna')
time.sleep(1)
confirm_cityFrom = driver.find_element(By.XPATH, '//*[@id="select2-loc_cap_mittente-results"]/li/div').click()

answered Feb 11, 2022 at 14:23

Andrea P.

728 bronze badges

Collectives™ on Stack Overflow

Cannot scrape input from dynamic dropdown menu due to <span> (Python - Selenium)

2 Answers 2

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related