selenium python return all outputs

Question

The purpose of this code is to scrape a bunch of URLs then extract the title from every web page. Then use the outputs in another function.

Here is the code:

from selenium import webdriver


class DataEngine:
    def __init__(self):
        self.urls = open(r"C:\Users\Sayed\Desktop\script\links.txt").readlines()
        self.driver = webdriver.Chrome(r"D:\Projects\Tutorial\Driver\chromedriver.exe")

    def title(self):
        for url in self.urls:
            self.driver.get(url)
            title = self.driver.find_element_by_xpath('//*[@id="leftColumn"]/h1').text
            return title

    def rename(self):
        names = self.title()
        for name in names:
            print(name)


x = DataEngine()
x.rename()

Here is what I expected:

Title (1)

Title (2)

Title (3)

Title (4)

Here is the output:

T

i

t

l

e

(

1

)

The return statement added inside the for loop, just extracts the first title and exists the loop. — gurpreet singh chahal
– gurpreet singh chahal, Commented Sep 27, 2018 at 12:39

Nordle · Accepted Answer · 2018-09-27 12:37:47Z

2

Build a list of the results for each URL, currently you only returning one (the first) result which is why it is printing like that:

from selenium import webdriver

class DataEngine:
    def __init__(self):
        self.urls = open(r"C:\Users\Sayed\Desktop\script\links.txt").readlines()
        self.driver = webdriver.Chrome(r"D:\Projects\Tutorial\Driver\chromedriver.exe")

    def title(self):
        titles = []
        for url in self.urls:
            self.driver.get(url)
            title = self.driver.find_element_by_xpath('//*[@id="leftColumn"]/h1').text
            titles.append(title)
        return titles

    def rename(self):
        names = self.title()
        for name in names:
            print(name)


x = DataEngine()
x.rename()

answered Sep 27, 2018 at 12:37

Nordle

2,9913 gold badges18 silver badges37 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

selenium python return all outputs

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related