Scraping dynamic dropdown with Selenium in Python

Question

I woud like a bit of help with the following. I am trying to scrape the elements of the tickers' dropdown on this website: https://live.hxro.io/tixwix

My code is as follow using selenium

from selenium import webdriver
from selenium.webdriver.support.select import Select
from selenium.webdriver.chrome.options import Options

url = "https://live.hxro.io/tixwix"
chrome_options = Options()
chrome_options.add_argument("--headless")
driver = webdriver.Chrome(executable_path = r'C:\geckodriver\chromedriver.exe',options = chrome_options)
driver.get(url)

tickers = driver.find_elements_by_class_name('lastprice-toggle')

tickers[0].text

This will only return

'BTC\nLast Price\n$39,255.07'

As it is an Ajax call I am not sure how to retrieve the other tickers in an efficient way. I thought the function find_element's' will return all the elements into a list but I only get the first one tickers[1] is out of bound.

Screenshot of the page source: enter image description here

Thanks

cruisepandey · Accepted Answer · 2021-06-17 10:52:35Z

2

The below code works fine on my local :

Explanation :

You need to click on accept cookies button and shall try with ExplicitWait plus you would need to click on a svg icon which is there in your drop down (XPATH : //span[text()='Last Price']/../following-sibling::*).

In the end span.moon contains all the elements that you are looking for.

Code :

executablePath = r'C:\geckodriver.exe'
options = webdriver.FirefoxOptions()
options.add_argument("--headless")
driver = webdriver.Firefox(executable_path = executablePath, options=options)
driver.maximize_window()
driver.get("https://live.hxro.io/tixwix")
wait = WebDriverWait(driver, 10)
wait.until(EC.element_to_be_clickable((By.ID, "onetrust-accept-btn-handler"))).click()
wait.until(EC.element_to_be_clickable((By.XPATH, "//span[text()='Last Price']/../following-sibling::*"))).click()
sleep(5)
for values in driver.find_elements(By.CSS_SELECTOR, "span.moon"):
    print(values.get_attribute('innerHTML'))

output :

$39,321.02
$39,321.02
$8,728.5
$0.31063
$2,435.337
$24.232
$40.99
$8.9730
$22.685
$37,234.01
&nbsp;HIGHER
<span class="tooltip-text">CLOSE ABOVE</span>
(CLOSE ABOVE)
(TOUCH)
 $100,000
14.20X
 $100,000
1000.05X
 $68,000
152.69X
 $66,000
127.38X
 $64,000
117.34X
 $63,000
112.25X
 $62,000
105.95X
 $61,000
4.490X
 $60,000
92.80X
 $44,000
4.790X

Process finished with exit code 0

answered Jun 17, 2021 at 10:52

cruisepandey

29.5k6 gold badges23 silver badges43 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

Rahan Over a year ago

Thanks for your reply. I get the an error with your code. ElementClickInterceptedException: Message: element click intercepted: Element <svg aria-hidden="true" focusable="false" data-prefix="fas" data-icon="chevron-down" class="svg-inline--fa fa-chevron-down fa-w-14 fa-1x dropdown-icon" role="img" xmlns="w3.org/2000/svg" viewBox="0 0 448 512">...</svg> is not clickable at point (490, 204).

cruisepandey Over a year ago

This is strange.. it works just fine on my machine

cruisepandey Over a year ago

Do you have driver.maximize_window() ?

cruisepandey Over a year ago

I got the above output, should work for you.

Rahan Over a year ago

Thank you, both answers are very helpful actually.

|

Prophet · Accepted Answer · 2021-06-17 16:18:35Z

1

There are several issues here:

You should add

options.add_argument("--start-maximized")

You should add a wait / delay
You should open the drop list
use correct locator to get elements in the drop list
UPD
5)In case of ElementClickInterceptedException click it with JavaScript. Not the best practice, but will work.
I Think the code should look like this:

from selenium import webdriver
from selenium.webdriver.support.select import Select
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

wait = WebDriverWait(driver, 30)

url = "https://live.hxro.io/tixwix"
chrome_options = Options()
chrome_options.add_argument("--headless")
options.add_argument("--start-maximized")
driver = webdriver.Chrome(executable_path = r'C:\geckodriver\chromedriver.exe',options = chrome_options)
driver.get(url)

last_price_toggle = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, '.lastprice-toggle')))
driver.execute_script("arguments[0].click();", last_price_toggle)

wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, '.lastprice-item')))

tickers = driver.find_elements_by_css_selector('div.lastprice-item span.moon')
for ticker in tickers:
    print(ticker.text)

edited Jun 17, 2021 at 16:18

answered Jun 17, 2021 at 10:55

Prophet

33.5k28 gold badges58 silver badges90 bronze badges

2 Comments

Rahan Over a year ago

Like cruisepandey's code I get an error. ElementClickInterceptedException: Message: element click intercepted: Element <div class="lastprice-toggle">...</div> is not clickable at point (399, 203). Other element would receive the click: <div style="position: absolute; display: flex; width: 100vw; height: 100vh; z-index: 101; background-color: rgb(19, 19, 21); text-align: center; align-items: center; font-size: 7vw; line-height: 7vw; color: gray;">...</div> (Session info: headless chrome=91.0.4472.106)

Rahan Over a year ago

Hi Prophet, as per above I have switched to firefox and the code is now working. Sorry for the troubles. Thank you

Collectives™ on Stack Overflow

Scraping dynamic dropdown with Selenium in Python

2 Answers 2

7 Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

7 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related