web scraping with selenium returns empty list

Question

I worked on little bit web scraping before but I have not idea of javascript. I want to scrape "Company Name" and "description of the company" from https://www.ces.tech/Show-Floor/Exhibitor-Directory.aspx. I am using selenium for scraping but I don't want to use browser in background. I write some code here:

from selenium.webdriver.common.by import By
from selenium import webdriver
import os
op = webdriver.ChromeOptions()
op.add_argument('headless')
driver = webdriver.Chrome(options=op)
driver.get('https://www.ces.tech/Show-Floor/Exhibitor-Directory.aspx')
company = []
items = driver.find_elements(By.CLASS_NAME, "exhibitorCardModal")
for item in items:
    comp=item.find_elements(By.CLASS_NAME, "company-name")
    desc = item.find_elements(By.CLASS_NAME, "description")
    result_dict = {
        "company":comp.text,
        "description":desc.text
    }
    company.append(result_dict)

But got empty list. Can someone tell me what is wrong here. I also try to use there api https://www.ces.tech/api/Exhibitors?searchTerm=&sortBy=alpha&alpha=&state=&country=&venue=&exhibitorType=&pageNo=1&pageSize=30 but got this error :

{"error":{"code":"ApiVersionUnspecified","message":"An API version is required, but was not specified."}}

Thanks for your effort. But got an error "TimeoutException: Message:" On this line "wait.until(EC.presence_of_element_located((By.CLASS_NAME, "exhibitorCardModal")))" — Sarla Devi
– Sarla Devi, Commented Dec 7, 2021 at 21:26
@SarlaDevi items is a list of web elements, not texts. you can not print it. Is that list is empty? — Prophet
– Prophet, Commented Dec 8, 2021 at 6:58

Prophet · Accepted Answer · 2021-12-07 21:32:42Z

You also have to add wait / delay before accessing the elements to let the page completely loaded before you trying to access them.
You should use find_element instead of find_elements for the loop internal commands:

comp=item.find_elements(By.CLASS_NAME, "company-name")
desc = item.find_elements(By.CLASS_NAME, "description")

So your code should be something like this:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium import webdriver
import os
import time

op = webdriver.ChromeOptions()
op.add_argument('headless')
driver = webdriver.Chrome(options=op)
wait = WebDriverWait(driver, 20)

driver.get('https://www.ces.tech/Show-Floor/Exhibitor-Directory.aspx')

wait.until(EC.presence_of_element_located((By.CLASS_NAME, "exhibitorCardModal")))
time.sleep(0.5)
company = []
items = driver.find_elements(By.CLASS_NAME, "exhibitorCardModal")
for item in items:
    comp=item.find_element(By.CLASS_NAME, "company-name")
    desc = item.find_element(By.CLASS_NAME, "description")
    result_dict = {
        "company":comp.text,
        "description"::desc.text
    }
    company.append(result_dict)

Collectives™ on Stack Overflow

web scraping with selenium returns empty list

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related