Python: Selenium getting empty results

Question

I am following this video to get myself familiar with selenium. My code is

from selenium import webdriver
from selenium.webdriver.common.keys import Keys

from pyvirtualdisplay import Display
import os

chromedriver = "/usr/bin/chromedriver"
os.environ['webdriver.chrome.driver'] = chromedriver

display = Display(visible=0, size=(800,600))
display.start()

br = webdriver.Chrome(chromedriver)
br.get("http://www.google.com")

Now to print the results

q = br.find_element_by_name('q')
q.send_keys('python')
q.send_keys(Keys.RETURN)
print br.title

results = br.find_elements_by_class_name('g')
print results

for result in results:
    print result.text
    print "-"*140

The output I am getting is just python and when I try to print results it is [].

When I try the below code in chrome's javascript console it works fine.

res = document.getElementsByClassName('g')[0]
<li class="g">…</li>
res.textContent
"           Python Programming Language – Official Websitewww.python.org/Cached - SimilarShareShared on Google+. View the post.You +1'd this publicly. UndoHome page for Python, an interpreted, interactive, object-oriented, extensible programming language. It provides an extraordinary combination of clarity and ...CPython - Documentation - IDEs - GuiProgramming"

So, any idea why am I not getting any results with selenium+python.

alex_jordan · Accepted Answer · 2012-11-12 14:44:17Z

2

Adding time.sleep(3) after q.send_keys(Keys.RETURN) seems to solve the problem. That's because when you press Keys.RETURN, ajax starts working and when you try to collect the result, they aren't on page yet. Selenium, AFAI has no stright way to determine whether the scripts like this have finished execution.

As I think, it would be more reliable to do

br.get("http://www.google.com/search?q=python")
results = br.find_elements_by_class_name('g')

answered Nov 12, 2012 at 14:44

alex_jordan

8771 gold badge6 silver badges11 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

RocketDonkey Over a year ago

+1. There is also the WebDriverWait method, where you can specify exactly which element to wait for. In this case, div ID center_col contains the results, and replacing the sleep with ` _ = WebDriverWait(br, 10).until(lambda x: br.find_element_by_id('center_col'))` (after you import via from selenium.webdriver.support.ui import WebDriverWait). In any case, I vote for you!

Bob Evans Over a year ago

i agree with RocketDonkey. I think it's a bad idea to say that adding a time.sleep is the answer. Please use WebDriverWait, or write your own polling ajax method. There should be no case where a static sleep solves your problem

Collectives™ on Stack Overflow

Python: Selenium getting empty results

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related