Python + Selenium: web scraping

Question

I am trying to extract some information from a website using Selenium below is the link to the website: http://www.ultimatetennisstatistics.com/playerProfile?playerId=4742 The information I am trying to get is players statistics which is located at a dropdown button 'statistics' which takes you to another page I have inspected the button and got an XPath and CSS but when I run my program it does not open the player's statistics page instead it just open this link below: http://www.ultimatetennisstatistics.com/playerProfile?playerId=4742

and gives me an error:

NoSuchElementException: no such element: 

Unable to locate element: {"method":"css selector","selector":"#playerPills > li.dropdown.active.open > ul > li.active"}
  (Session info: chrome=67.0.3396.99)
  (Driver info: chromedriver=2.41.578737 (49da6702b16031c40d63e5618de03a32ff6c197e),platform=Windows NT 6.3.9600 x86_64)

Below is my code:

from bs4 import BeautifulSoup
from selenium import webdriver

driver = webdriver.Chrome()
driver.get("http://www.ultimatetennisstatistics.com/playerProfile?playerId=4742")
soup = BeautifulSoup(driver.page_source,"lxml")

bm = driver.find_element_by_css_selector('#playerPills > li.dropdown.active.open > ul > li.active')
bm.click()

Can someone show we how I can get the player's statistics page to open with Selenium and extract the information in the table?

try working with this element <a href="/statsLeaders"><span class="glyphicon glyphicon-stats fa-fw"></span> Statistics Leaders</a> — Ywapom
– Ywapom, Commented Aug 7, 2018 at 23:01
@Ywapom that not the information i am trying to get the infor i am interested in is on the player's profile under statistics — smith
– smith, Commented Aug 7, 2018 at 23:10

Karl · Accepted Answer · 2018-08-07 23:49:38Z

1

If you inspect the html source of the page, you can access the CSS id directly for the button you want to click. Using selenium you can find the button by its id by doing driver.find_element_by_id('statisticsPill') which will allow you to then click on it to show the table.
After that loads you can then parse through the table to get the data you want.

Example:

from selenium import webdriver

driver = webdriver.Chrome()
driver.get("http://www.ultimatetennisstatistics.com/playerProfile?playerId=4742")

try:
    # Fist click on the dropdown
    dropdown = driver.find_element_by_xpath("//a[@id='statisticsPill']/../../..")
    dropdown.click()

    # Then click on the statistics button
    bm = driver.find_element_by_id('statisticsPill')
    bm.click()
except NoSuchElementException as e:
    # Do error handling when cannot find the button

Edit: You have to first click on the dropdown for the button to become visible and then click on it.

edited Aug 7, 2018 at 23:49

answered Aug 7, 2018 at 23:13

Karl

1,7442 gold badges14 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

smith Over a year ago

hey thanks for your help but it is only still opening the players profile for some reason it is not clicking or opening the players statistics page

smith Over a year ago

you know of any other ways i could try to access the information

Karl Over a year ago

@smith I just added a fix to my code which first clicks on the dropdown button and then clicks on the statistics button which worked fine for me when tested.

smith Over a year ago

thanks man it work this is my first time working with selenium great help

smith Over a year ago

hey could you also show me how i can parse the players stats in the table

Collectives™ on Stack Overflow

Python + Selenium: web scraping

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related