webscrape table with text in html using WebDriverWait

Question

I try to webscrape all the Company Names with 500 or more employees of the following website:

https://de.statista.com/companydb/suche?idCountry=276&idBranch=0&revenueFrom=-1000000000000000000&revenueTo=1000000000000000000&employeesFrom=500&employeesTo=100000000&sortMethod=revenueDesc&p=1

I wrote a code to scrape the Company Names of the the first site and the script will then click on the "Next Site Button" and scrape again the names. The names will be saved into a list, and this will happen until the list has a certain number of names in it. Then it will transfer the list into a dataframe and export it into an xslfile. Unfortunately it does not do this at the moment. Here is the Code

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
import pandas as pd
import time
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

company_list = []

driver = webdriver.Chrome('/Users/rieder/Anaconda3/chromedriver_win32/chromedriver.exe')

driver.get('https://de.statista.com/companydb/suche?idCountry=276&idBranch=0&revenueFrom=-1000000000000000000&revenueTo=1000000000000000000&employeesFrom=500&employeesTo=100000000&sortMethod=revenueDesc&p=1')

driver.find_element_by_id("cookiesNotificationConfirm").click();

while len(company_list) < 20:

    company_name = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//table[@class='zebraTable zebraTable--companies']//following::tr[2]/td[@class='zebraTable__td zebraTable__td--companyName']/a"))).get_attribute("innerHTML")
    
    for p in range(len(company_name)):
        company_list.append(company_name)
        
    driver.find_element_by_xpath("//*[@id='content']/section[3]/div/div/form/div/div[2]/div[2]/div[2]/div/button[2]").click();
              
    print(company_list)

    df = pd.DataFrame(company_list,columns =['Unternehmensname']) 

    df.to_excel("output.xlsx")  
            
    time.sleep(5)

And my Output looks like this:

['\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ', '\n                        Progress-Werk Oberkirch AG\n                    ']

I think its because the .get_attribute() only gets one attribute, but i dont know how to get all the attributes at this points.

Do you want all attributes or all companies listed?

Mate Mrše
– Mate Mrše

2020-08-31 10:59:58 +00:00
Commented Aug 31, 2020 at 10:59 — Mate Mrše
– Mate Mrše, Commented Aug 31, 2020 at 10:59
I want all company names listed

Yankzz
– Yankzz

2020-09-01 08:38:09 +00:00
Commented Sep 1, 2020 at 8:38 — Yankzz
– Yankzz, Commented Sep 1, 2020 at 8:38

rahul rai · Accepted Answer · 2020-09-01 09:26:20Z

1

Yes using .get_attribute() you can only get one attribute at a time. To get all attributes you can below javascript code:

driver.execute_script('var items = {}; for (index = 0; index < arguments[0].attributes.length; ++index) { items[arguments[0].attributes[index].name] = arguments[0].attributes[index].value }; return items;', ele)

Here ele is your webelement.

To Print all the company name you can use below approach:

company_names = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//td[@class='zebraTable__td zebraTable__td--companyName']")))
for cn in company_names:
    print(cn.text)

Note : It will print all the company names on first page. If you want to get names from all the page then you need to click on next page icon on each page and click above code in a loop.

edited Sep 1, 2020 at 9:26

answered Aug 31, 2020 at 10:56

rahul rai

2,3361 gold badge9 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Yankzz Over a year ago

I implemented this code: company_name = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//table[@class='zebraTable zebraTable--companies']//following::tr[2]/td[@class='zebraTable__td zebraTable__td--companyName']/a"))).get_attribute("innerHTML") company_name2 = driver.execute_script('var items = {}; for (index = 0; index < arguments[0].attributes.length; ++index) { items[arguments[0].attributes[index].name] = arguments[0].attributes[index].value }; return items;', company_name)

rahul rai Over a year ago

The reason you got error because in company_name you have already stored string by using .get_attribute("innerHTML"). You can run Java script methods meant for webelements on string :P. Also if you want to get all attribute for company_name element remove .get_attribute("innerHTML"). Like in below comment.

rahul rai Over a year ago

company_name = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//table[@class='zebraTable zebraTable--companies']//following::tr[2]/td[@class='zebraTable__td zebraTable__td--companyName']/a"))) company_name2 = driver.execute_script('var items = {}; for (index = 0; index < arguments[0].attributes.length; ++index) { items[arguments[0].attributes[index].name] = arguments[0].attributes[index].value }; return items;', company_name)

rahul rai Over a year ago

However As I can see you want to print all the company name so you need to take different approach.

rahul rai Over a year ago

I have given code at botton to scape name of company list from first page, let me know if you need help to get data from all page.

|

Mate Mrše · Accepted Answer · 2020-09-01 09:00:07Z

1

You can use find_elements_by_css_selector method to find multiple web elements (all shown company names.

I won't write everything, but the start of the while loop should look something like this:

companies = driver.find_elements_by_css_selector(".zebraTable__td--companyName")

Then you should loop through companies list and get attributes for each list member.

answered Sep 1, 2020 at 9:00

Mate Mrše

8,52413 gold badges52 silver badges95 bronze badges

Comments

Yankzz · Accepted Answer · 2020-09-24 09:02:59Z

It worked with the following code, which generates a excel list with first 100 company names:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
import pandas as pd
import time
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

company_list = []

driver = webdriver.Chrome('/Users/rieder/Anaconda3/chromedriver_win32/chromedriver.exe')

driver.get('https://de.statista.com/companydb/suche?idCountry=276&idBranch=0&revenueFrom=-1000000000000000000&revenueTo=1000000000000000000&employeesFrom=500&employeesTo=100000000&sortMethod=revenueDesc&p=1')

driver.find_element_by_id("cookiesNotificationConfirm").click();

while len(company_list) < 100:
    
    company_names = WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//td[@class='zebraTable__td zebraTable__td--companyName']")))
    
    for cn in company_names:
        company_list.append(cn.text)
        
    driver.find_element_by_xpath("//*[@id='content']/section[3]/div/div/form/div/div[2]/div[2]/div[2]/div/button[2]").click();
              
   
    df = pd.DataFrame(company_list, columns =['Unternehmensname'])
    
    df.to_excel("output.xlsx")  
            
    time.sleep(5)

thanks a lot guys

Collectives™ on Stack Overflow

webscrape table with text in html using WebDriverWait

3 Answers 3

6 Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

6 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related