Scroll in Selenium Webdriver (Python)

Question

Prerequisites.
You need an account at Instagram to use this script.
Setup a test environment:

Log in, open the needed list(works correctly):

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from time import sleep

driver = webdriver.Chrome(
# driver = webdriver.Firefox(    
# driver = webdriver.PhantomJS(
    service_args=['--ignore-ssl-errors=true', '--ssl-protocol=any'])

driver.get("https://instagram.com/accounts/login")
username = driver.find_element_by_name("username")
password = driver.find_element_by_name("password")

username1 = 'instagram' # change it!
password1 = 'instagrampassword1' # change it!

username.send_keys(username1)
password.send_keys(password1)

submit_button = driver.find_element_by_css_selector(
    '#react-root > div > article > div > div:nth-child(1) > div > form > span > button')
submit_button.click()

sleep(2)

link = 'https://www.instagram.com/youtube/'
driver.get(link)

driver.implicitly_wait(2)
driver.find_elements_by_class_name("_218yx")[2].click()

Wrong scroll. How to fix this block?

How to focus and scroll correctly on this page?

My attempts:

driver.find_element_by_class_name("_cx1ua").send_keys(Keys.NULL) # focus
      #The element has been deleted entirely or
      #The element is no longer attached to the DOM.

driver.find_element_by_class_name("_q44m8").send_keys(Keys.NULL)
# cannot focus element

driver.find_element_by_class_name("_qjr85").send_keys(Keys.NULL) 
# cannot focus element


for i in range(5):
    driver.find_element_by_class_name("_cx1ua").send_keys(Keys.END)

=============================================================
to @Moshisho :

We need to focus on some element to activate it.
The question is what the element we need to choose to focus and how?
This is not a "body":
something like that, but not this:

background = driver.find_element_by_css_selector("body")
# background = driver.find_element_by_css_selector("div._2uju6")

for i in range(5):
    background.send_keys(Keys.SPACE)
    time.sleep(1)

Without it this command do not work.

to @Naveen :

print(driver.find_element_by_css_selector("div._a1rcs").location_once_scrolled_into_view) # {'x': 0, 'y': 0}
print(driver.find_element_by_class_name("_cx1ua").location_once_scrolled_into_view) # {'x': 376, 'y': 229}
print(driver.find_element_by_class_name("_q44m8").location_once_scrolled_into_view) # {'x': 376, 'y': 180}
print(driver.find_element_by_class_name("_qjr85").location_once_scrolled_into_view) # {'x': 376, 'y': 180}

And what's next?

driver.execute_script("window.scrollTo(0, 3000);") # do not working

Your question should be closed as off-topic. Give some details and description to make your issue more clear... not everyone has account on Instagram :) — Andersson
– Andersson, Commented Dec 25, 2016 at 17:58
Your attempts are for focusing or scrolling a specific element? — Moshisho
– Moshisho, Commented Dec 25, 2016 at 18:40
Yes, I want to scroll all the list of followers, to see > 10 followers per one visit. — lvcpp
– lvcpp, Commented Dec 25, 2016 at 18:42

lvcpp · Accepted Answer · 2016-12-26 15:39:09Z

5

Try the following code:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from time import sleep
from selenium.webdriver.support.ui import Select

driver = webdriver.Chrome(
# driver = webdriver.Firefox(    
# driver = webdriver.PhantomJS(
    service_args=['--ignore-ssl-errors=true', '--ssl-protocol=any'])

driver.maximize_window()

driver.get("https://instagram.com/accounts/login")
username = driver.find_element_by_name("username")
password = driver.find_element_by_name("password")

username1 = 'instagramlogin1'    # change it!
password1 = 'instagrampassword1' # change it!

username.send_keys(username1)
password.send_keys(password1)

submit_button = driver.find_element_by_css_selector(
    '#react-root > div > article > div > div:nth-child(1) > div > form > span > button')
submit_button.click()

sleep(2)

link = 'https://www.instagram.com/youtube/'
driver.get(link)

driver.implicitly_wait(2)
following = driver.find_element_by_xpath("//a[@href='/youtube/following/']/span")
total_following = int(following.text)
print "total no. of users following: ", total_following
# click on 239 following, displays 10 users
following.click()

loaded_following = driver.find_elements_by_xpath("//ul[@class='_539vh _4j13h']/li")
loaded_till_now = len(loaded_following)

while(loaded_till_now<total_following):
    print "following users loaded till now: ", loaded_till_now
    print loaded_following[loaded_till_now-1]
    loaded_following[loaded_till_now-1].location_once_scrolled_into_view
    # driver.execute_script("arguments[0].focus();", loaded_following[loaded_till_now-1])
    driver.find_element_by_tag_name('body').send_keys(Keys.END) # triggers AJAX request to load more users. observed that loading 10 users at a time.
    sleep(1) # tried wihtout sleep but throws StaleElementReferenceException. As it takes time to get the resposne and update the DOM
    loaded_following = driver.find_elements_by_xpath("//ul[@class='_539vh _4j13h']/li")
    loaded_till_now = len(loaded_following)

# All 239 users are loaded. 
driver.quit()

Observed that browser is sending AJAX request to load more users. this action is triggered when you scroll using mouse or enter Space or End keys

edited Dec 26, 2016 at 15:39

lvcpp

1791 gold badge3 silver badges16 bronze badges

answered Dec 26, 2016 at 6:29

Naveen Kumar R B

6,4385 gold badges36 silver badges68 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

lvcpp Over a year ago

thanks, interesting property, but how I can use it to do the needed things? see the explained message to you above.

Naveen Kumar R B Over a year ago

I am not sure which element you are talking about (the class names are dynamic, so not working for me). Can you just share the screenshot of the screen, highlighting the element and what you want to do with it? either click or send keys?

lvcpp Over a year ago

At this case the class names are static, only looks like dynamic. You can check it with different browsers.

lvcpp Over a year ago

without any manipulation self.driver.find_elements_by_class_name("_6jvgy") I'll get only 10 elements,

lvcpp Over a year ago

but I want to scroll down, ideally to the end of list, to get 239 elements(for youtube).

|

Moshisho · Accepted Answer · 2016-12-26 07:24:18Z

4

In order to scroll in the window, you need to execute JavaScript, try this:

driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

EDIT: in order to focus an element (it needs to be able to get the focus e.g. an anchor, input, button etc...) you also need to use JavaScript executor:

elementToFocus = driver.find_element_by_id("yourID")
driver.execute_script("arguments[0].focus();", elementToFocus)

edited Dec 26, 2016 at 7:24

answered Dec 25, 2016 at 18:46

Moshisho

3,0411 gold badge25 silver badges43 bronze badges

2 Comments

lvcpp Over a year ago

thanks, I saw this code snippet in a few places earlier. We need to focus on some element to activate it. The question is what the element we need to choose to focus and how? This is not a "body": background = browser.find_element_by_css_selector("body") # background = driver.find_element_by_css_selector("div._2uju6") for i in range(5): background.send_keys(Keys.SPACE) time.sleep(1) Without it this command do not work.

lvcpp Over a year ago

there no any element with id.<br/> I've tested the second line, but it dont work yet.<br/> So I've selected by class, but maybe need to select 2nd or 3rd element of the class<br/> (I know how to do it, but there a lot of elements, and it looks like the majority of them should be visible and acceptable).

Sida Zhou · Accepted Answer · 2021-03-24 04:03:20Z

I'm working with a dynamic React app, I need to scroll to the pages bottom to make react render all the data.

For unknown reasons, solutions based on JS execute_script didn't work. However I got send_keys solution working:

# scroll to bottom to load all
WebDriverWait(driver, 5).until(
    EC.presence_of_element_located((By.XPATH, "//body"))
)

attempt_num = 2
while attempt_num > 0:
    try:
        elem = driver.find_element_by_xpath("//body")
        elem.click()
        elem.send_keys(Keys.END)
    except StaleElementReferenceException as e:
        print(e)
    attempt_num = attempt_num - 1

The click() on body and the retry for StaleElementReferenceException are crucial. I haven't found a more elegant way than to retry.

See top answer of How to avoid "StaleElementReferenceException" in Selenium?

Collectives™ on Stack Overflow

Scroll in Selenium Webdriver (Python)

3 Answers 3

6 Comments

2 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

6 Comments

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related