I'm trying to get the text within the element id='resultStats' from the Google Search Result Page for the following query https://www.google.com/search?q=site:https://theshipibomarket.com/
I get an output with the code I've got but it's not the text within the element. The output is:
[<selenium.webdriver.firefox.webelement.FirefoxWebElement (session="5a1e4063-dcb8-48b2-93f6-1c60bb7e9e05", element="63dabd48-bd5f-4380-9598-173b91e72367")>]
When I use the .text function on the results element I get the follow error:
AttributeError: 'list' object has no attribute 'text'
Here's my code:
# import libraries
import urllib.request
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
import time
options = Options()
options.headless = True
query = "site:https://theshipibomarket.com/"
urlpage = "https://www.google.com/search?q="+query
print(urlpage)
# run firefox webdriver from executable path of your choice
driver = webdriver.Firefox(options=options)
# get web page
driver.get(urlpage)
# execute script to scroll down the page
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);var lenOfPage=document.body.scrollHeight;return lenOfPage;")
# sleep for 30s
time.sleep(30)
# driver.quit()
# find elements by xpath
results = driver.find_elements_by_xpath("//*[@id='resultStats']")
#print('Number of results', len(results))
print("The number of pages Google have index {}".format(results.text))
I suspect that because it's javascript that this is causing the issue, as the output is a list. I've not got much experience scraping Google or doing a lot of scraping in general so apologise if this is a simple misunderstanding on my behalf.
