selenium-webdriver: How to use for loop to find_elements

Question

I want to get all the links and start_time and end_time one a page and then send to function(parse_detail) to scrap another information But I don't know how to use selenium to for loop

Here is my code AND there is error:

for site in sites:
exceptions.TypeError: 'WebElement' object is not iterable

Please teach me how to use for loop like scrapy in selenium. Thank you!

class ProductSpider(Spider):
    name = "city20140808"
    start_urls = ['http://wwwtt.tw/11']

    def __init__(self):
        self.driver = webdriver.Firefox()
        dispatcher.connect(self.spider_closed, signals.spider_closed)

    def parse(self, response):
        self.driver.get(response.url)

        item  = CitytalkItem()
        sites = self.driver.find_element_by_css_selector("div.body ")
        for site in sites:
            linkiwant = site.find_element_by_css_selector(".heading a")
            start = site.find_element_by_css_selector("div.content p.m span.date")
            end = site.find_element_by_css_selector("div.content p.m span.date")

            item['link'] = linkiwant.get_attribute("href") 
            item['start_date']  = start.text
            item['end_date']  = end.text
            yield Request(url=item['link'], meta={'items':items}, callback=self.parse_detail)  
     def parse_detail(self,response):
        item = response.meta['items']
        ........
        yield item

alecxe · Accepted Answer · 2014-08-09 00:16:05Z

5

Instead of find_element_by_css_selector(), which returns a single element, you need to use find_elements_by_css_selector(), which returns a list of elements.

answered Aug 9, 2014 at 0:16

alecxe

476k127 gold badges1.1k silver badges1.2k bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

selenium-webdriver: How to use for loop to find_elements

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related