Scraping website with selenium By css selector method gives errors

Question

I had a notebook for scraping Indiegogo website that was working perfectly, not I got errors as I see css selector method is now deprecated. I checked the website and nothing has changed, and I try to update my methods but still it does not work:

import sys
import logging
from selenium.webdriver.remote.remote_connection import LOGGER
LOGGER.setLevel(logging.WARNING)
sys.path.insert(0,'/usr/lib/chromium-browser/chromedriver')
from selenium import webdriver
from tqdm import tqdm_notebook as tqdm
import pandas
import json
import pprint

chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
chrome_options.add_argument("user-agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36")

wd = webdriver.Chrome('chromedriver',chrome_options=chrome_options)
wd.get("https://www.indiegogo.com/explore/all?project_type=campaign&project_timing=all&sort=trending")

list_titles = wd.find_element(By.CSS_SELECTOR, "div.discoverableCard-title")
print(len(list_titles))

this is what is suggested, by running this in Colab says:

name 'By' is not defined

Jaky Ruby · Accepted Answer · 2022-11-09 19:58:44Z

I think you are missing this import:

from selenium.webdriver.common.by import By

Beside that, the locator looks not correct, with some small changes your code should be like this and it should do what you wanted:

import sys
import logging
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.remote.remote_connection import LOGGER
LOGGER.setLevel(logging.WARNING)
sys.path.insert(0,'/usr/lib/chromium-browser/chromedriver')


chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--headless')
chrome_options.add_argument('--disable-dev-shm-usage')
chrome_options.add_argument("user-agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36")

wd = webdriver.Chrome('chromedriver',chrome_options=chrome_options)
wd.maximize_window()

wd.get("https://www.indiegogo.com/explore/all?project_type=campaign&project_timing=all&sort=trending")
list_titles = WebDriverWait(wd, 20).until(EC.presence_of_all_elements_located((By.XPATH, "//div[@class='discoverableCard-title ng-binding discoverableCard-lineClamp2']")))
print(len(list_titles))

Collectives™ on Stack Overflow

Scraping website with selenium By css selector method gives errors

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related