Assistance with scraping in selenium

Question

This question was asked in June (Assistance needed scraping a site with Selenium in Python) and I am using the code from then, but running into problems. I assume there has been a change to the PP site. Here is my updated code that seems to be running into an error at clicking the Fantasy Score portion. The goal of the project is just to scrape the fantasy scores from NBA players

from time import sleep
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import chromedriver_binary

# Assign PrizePicks URL

driver = webdriver.Chrome()
PATH = "C:\Program Files (x86)\chromedriver.exe"
############## PRIZEPICKS ################################################
#
url = "https://app.prizepicks.com/"

driver.get(url)

#this is to get rid of the pop-up box that shows after you Selenium opens the prizepicks page
driver.find_element(By.CLASS_NAME,"close").click()

#Selecting NBA
driver.find_element(By.XPATH, "//div[@class='name'][normalize-space()= 'NBA']").click()

#Clicking the Fantasy Score tab
driver.find_element(By.XPATH, "//div[@class='stat stat-active'][normalize-space()='Fantasy Score']").click()
driver.find_element(By.XPATH, "//div[@class='segment-selector-button']").click()
projections = WebDriverWait(driver, 20).until(
 EC.presence_of_all_elements_located((By.CSS_SELECTOR, ".projection")))

#creating a list of the players names and fantasy scores to be collected
nbaPlayers = []

for projection in projections:

    names = projection.find_element_by_xpath('.//div[@class="name"]').text
    points= projection.find_element_by_xpath('.//div[@class="presale-score"]').get_attribute('innerHTML')
    print(names, points)

    players = {
        'Name': names,
        'FantasyPoints': points,
        }

    nbaPlayers.append(players)

#Put the list in a dataframe
df = pd.DataFrame(nbaPlayers)
print(df)

driver.quit()

user17945623 · Accepted Answer · 2022-01-16 18:56:46Z

1

For the newer Selenium version it is advised to use

driver.find_element(By.XPATH, "//*[text()='Fantasy Score']").click()

answered Jan 16, 2022 at 18:56

user17945623

Sign up to request clarification or add additional context in comments.

Comments

ilhanmertalan · Accepted Answer · 2022-01-16 17:08:23Z

0

You can use "find_element_by_xpath" like this.

driver.find_element_by_xpath("//*[text()='Fantasy Score']").click()

answered Jan 16, 2022 at 17:08

ilhanmertalan

536 bronze badges

Collectives™ on Stack Overflow

Assistance with scraping in selenium

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related