Reading elements from html selenium

Question

I am trying to record every item on the tf2 market place using selenium. I am trying to record the name of each item in a file on sale. This is the link to the page. I think it is this tag I just dont know how to reference and record the name in a text file with each name on a new line.

<span id="result_0_name" class="market_listing_item_name" style="color; #7D6D00;">

Edit 1:

I have used the solution by alecxe and it works for the first page I'm now trying to run it to select the next button then run again. But to no avail this is what I am trying.

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time

from selenium import webdriver
url="http://steamcommunity.com/market/search?appid=440#p1_popular_desc"
driver = webdriver.Firefox()
driver.get(url)

x=1
while x==1:
    WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.market_listing_row")))
    time.sleep(5)
    results = [item.text for item in driver.find_elements_by_css_selector("div.market_listing_row .market_listing_item_name")]
    time.sleep(5)
    driver.find_element_by_id('searchResults_btn_next').click()
    with open("output.dat", "a") as f:
        for item in results:
            f.write(item + "\n")

This produces this error

Traceback (most recent call last):
  File "name.py", line 14, in <module>
    results = [item.text for item in driver.find_elements_by_css_selector("div.market_listing_row .market_listing_item_name")]
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webelement.py", line 61, in text
    return self._execute(Command.GET_ELEMENT_TEXT)['value']
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webelement.py", line 402, in _execute
    return self._parent.execute(command, params)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 175, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 166, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.StaleElementReferenceException: Message: Element is no longer attached to the DOM
Stacktrace:
    at fxdriver.cache.getElementAt (resource://fxdriver/modules/web-element-cache.js:8956)
    at Utils.getElementAt (file:///tmp/tmpUpLsV7/extensions/[email protected]/components/command-processor.js:8546)
    at WebElement.getElementText (file:///tmp/tmpUpLsV7/extensions/[email protected]/components/command-processor.js:11704)
    at DelayedCommand.prototype.executeInternal_/h (file:///tmp/tmpUpLsV7/extensions/[email protected]/components/command-processor.js:12274)
    at DelayedCommand.prototype.executeInternal_ (file:///tmp/tmpUpLsV7/extensions/[email protected]/components/command-processor.js:12279)
    at DelayedCommand.prototype.execute/< (file:///tmp/tmpUpLsV7/extensions/[email protected]/components/command-processor.js:12221)

Any help would be greatly appreciated even if it is links to guides

alecxe · Accepted Answer · 2015-07-06 14:07:37Z

1

You can get the names from elements with market_listing_item_name class name located in div elements having market_listing_row class:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

from selenium import webdriver

url = "http://steamcommunity.com/market/search?appid=440"
driver = webdriver.Chrome()
driver.get(url)

# wait for results
WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.market_listing_row")))

results = [item.text for item in driver.find_elements_by_css_selector("div.market_listing_row .market_listing_item_name")]

driver.quit()

# dump results to a file
with open("output.dat", "wb") as f:
    for item in results:
        f.write(item + "\n")

Here is the contents of the output.dat file after running the script:

Mann Co. Supply Crate Key
The Powerhouse Weapons Case
The Concealed Killer Weapons Case
Earbuds
Bill's Hat
Gun Mettle Campaign Pass
Tour of Duty Ticket
Genuine AWPer Hand
Specialized Killstreak Kit
Gun Mettle Key

answered Jul 6, 2015 at 14:07

alecxe

476k127 gold badges1.1k silver badges1.2k bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Daniel Prinsloo Over a year ago

ok how do identify what the css selectors are like how did you determine that it was those i know it is basic but im new to css and selenium

alecxe Over a year ago

@DanielPrinsloo used browser developer tools and "inspect element" feature, also applied some CSS knowledge.

Daniel Prinsloo Over a year ago

Do you know any good online guides to learning how CSS works?

Daniel Prinsloo Over a year ago

please take a look at the edit as i have tried to further it

Daniel Prinsloo Over a year ago

Never mind i just needed to add a sleep to let the page fully load ive edited

Collectives™ on Stack Overflow

Reading elements from html selenium

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related