How to scrape data from website using selenium

Question

I am try to scrape data but they will provide me an error this is website link https://www.dastelefonbuch.de/Suche/Zahnarzt

from selenium import webdriver
PATH="C:\Program Files (x86)\chromedriver.exe"
url='https://www.dastelefonbuch.de/Suche/Zahnarzt'
driver =webdriver.Chrome(PATH)
driver.get(url)
vid=driver.find_element_by_xpath("//div[@class='vcard']")
for item in vid:
    title=item.find_element_by_xpath("//div[@class='name']").text
    phone=item.find_element_by_xpath("//span[@class='nr']").text
    website=item.find_element_by_xpath("//i[@class='icon icon_url']").text
    print(title,phone,website)

undetected Selenium · Accepted Answer · 2021-11-18 12:06:27Z

1

Instead of vid=driver.find_element_by_xpath("//div[@class='vcard']") which returns a WebElement you need vid=driver.find_element_by_xpath("//div[@class='vcard']") which would return a list to iterate as follows:

from selenium.webdriver.common.by import By

vid = driver.find_elements(By.XPATH, "//div[@class='vcard']")
for item in vid:
    title=item.find_element_by_xpath("//div[@class='name']").text
    phone=item.find_element_by_xpath("//span[@class='nr']").text
    website=item.find_element_by_xpath("//i[@class='icon icon_url']").text
    print(title,phone,website)

edited Nov 18, 2021 at 12:06

answered Nov 18, 2021 at 11:57

undetected Selenium

194k44 gold badges304 silver badges387 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Amen Aziz Over a year ago

they give these error NameError: name 'find_elements_by_xpath' is not defined

undetected Selenium Over a year ago

Checkout the answer update and let me know the status

Amen Aziz Over a year ago

they give us wong output repeatedly print one name many time

Amen Aziz Over a year ago

and also not provide website link

undetected Selenium Over a year ago

@AmenAziz This answer was based on find_element_by_xpath() and find_elements_by_xpath() stuff, which I strongly believe have got you started in the right direction. repeatedly print one name sounds like a different issue all together and you may like to raise a new question.

Wonka · Accepted Answer · 2021-11-18 12:37:20Z

0

When you extract xpath on an item you have to change a bit xpath, saying lookup on current node with ".//"

Change

title=item.find_element_by_xpath("//div[@class='name']").text

to

title=item.find_element_by_xpath(".//div[@class='name']").text

Checkout and feedback if it works

answered Nov 18, 2021 at 12:37

Wonka

1,9012 gold badges15 silver badges23 bronze badges

2 Comments

Amen Aziz Over a year ago

it give me the correct but why put dot can you tell me

Wonka Over a year ago

is cause // means global (all html), and .// means from the node (web element, your div). Also just one / means direct childrens. Here you can get more info about it and more complexity Xpaths en.wikipedia.org/wiki/XPath

Collectives™ on Stack Overflow

How to scrape data from website using selenium

2 Answers 2

5 Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related