1

I wanted to retrieve a variable named "stats" from the site https://csgostats.gg/player/76561198087162515 using Selenium with Python with the following code:

browser = webdriver.Chrome()
browser.get("https://csgostats.gg/player/76561198087162515")
stats = browser.execute_script("return stats")
print(stats)

However, the console returns an error saying "match is not defined". When I use inspect element on the site, I can clearly see the stats variable as shown here: stats variable inside script tag

Does anyone know what I might be doing wrong here? The code I'm using seems to work for variables from other websites so I'm not sure what I'm doing wrong. I'm pretty new to this stuff so any help would be appreciated, thanks!

Improve this question
5
  • If you aren't able to find a solution using selenium, you may want to try using BeautifulSoup to parse the HTML of the page to get the stats variable. See here: stackoverflow.com/a/51780317/7253453 -- you would need to do soup = BeautifulSoup(browser.page_source, 'lxml') to parse the HTML of the webpage. Not as elegant, but a workable solution nonetheless Commented May 14, 2020 at 21:27
  • I see. I tried to use BeautifulSoup at first but I wasn't able to access the website for some reason using requests.get. I kept getting a response 403 from the url I posted above even after setting my User-Agent properly. Also, chrome loads the webpage normally even in icognito mode so I'm confused why I can't get a response 200 for that webpage Commented May 14, 2020 at 21:35
  • Were you using selenium to .get() the page? Or were you using requests. If it is the latter, that's likely the reason, as requests.get() is not javascript enabled, whereas selenium.get() is. You'll want to .get() the page with selenium, then parse the .page_source with BeautifulSoup Commented May 14, 2020 at 21:39
  • Okay looks like I got it to work correctly. The variable I was looking for actually rests in a piece of commented-out code so I found a way to find all comments. From there I'll just use some way to parse the variable. Thanks a ton! Commented May 14, 2020 at 23:46
  • No problem, happy to help! Commented May 15, 2020 at 0:11
Related questions
2
Scraping javascript with Python and Selenium Webdriver
1
Issue scraping javascript generated content with Selenium and python
2
Python Selenium Scraping Javascript - Element not found
Load 7 more related questions Show fewer related questions

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.