How to find position of an element using pyautogui and selenium automatically

Question

I'm writing an script and I need to find cursor position automatically for an element. The script will run on different computers, so I want it be done automatically.

Suppose I want to click on search box in the head of stackoverflow.com to find its position using selenium and pyautogui.

How to do that so that mouse is in that position and script is exited?

Edit 1

Suppose I want it to click on this element:

input which name="q" and class="s-input s-input__search js-search-field "

Edit 2:

My current code with the help of Prophet and undetected Selenium, but the output I get is not correct:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains
import pyautogui
from time import sleep
from datetime import datetime
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as ec
from selenium.webdriver.support.wait import WebDriverWait

path = r'path'
user_data = r'user'
options = webdriver.ChromeOptions()
options.add_argument(f'--user-data-dir={user_data}')
options.add_argument('--profile-directory=Default')
driver = webdriver.Chrome(executable_path=path, options=options)
actions = ActionChains(driver)
xpath = '//*[@id="search"]/div/input'

url = 'https://stackoverflow.com/'
driver.get(url)
position = WebDriverWait(driver, 20).until(ec.visibility_of_element_located((By.XPATH, xpath))).location_once_scrolled_into_view
# both tried the above position var and the below
# position = WebDriverWait(driver, 20).until(ec.visibility_of_element_located((By.XPATH, xpath))).location
# element = driver.find_element(By.XPATH, xpath)
# actions.move_to_element(element).perform()
# position = pyautogui.position()
file = open('position.txt', 'a')
file.write(f'{position}, {datetime.now()}\n')
driver.close()

# to test the exact or relative position to compare with above code
# while True:
#     sleep(1)
#     print(pyautogui.position())

Edit 3:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains
import pyautogui
from time import sleep
from datetime import datetime
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait

path = r'D:\\Downloads\\chromedriver_win32\\chromedriver.exe'
user_data = r'C:\\Users\\Saeed\\AppData\\Local\\Google\\Chrome\\User Data\\Default'
options = webdriver.ChromeOptions()
options.add_argument(f'--user-data-dir={user_data}')
options.add_argument('--profile-directory=Default')
driver = webdriver.Chrome(executable_path=path, options=options)
actions = ActionChains(driver)
xpath = '//*[@id="search"]/div/input'  # `search` box at header
url = 'https://stackoverflow.com//'
driver.get(url)
driver.maximize_window()
sleep(10)
position = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, xpath))).location
print(position)
file = open('position.txt', 'a')
file.write(f'{position}, {datetime.now()}\n')
# pyautogui.click(793, 9)
driver.close()

# while True:
#     sleep(1)
#     print(pyautogui.position())

The video how my code now works: https://up.mazandhost.com/uploads/1028407950.bandicam-2022-03-27-21-33-27-729.mp4

Solution in my case:

Answer https://stackoverflow.com/a/56693949/5790653 helps me.

Prophet · Accepted Answer · 2022-03-27 15:23:16Z

2

With Selenium you can perform mouse actions with the use of action_chains library.
Once you locate the element with selenium you can move the mouse to that element as following:

from selenium.webdriver.common.action_chains import ActionChains

actions = ActionChains(driver)

element = driver.find_element(By.XPATH, 'the_xpath_locator')
actions.move_to_element(element).perform()

answered Mar 27, 2022 at 15:23

Prophet

33.5k28 gold badges58 silver badges90 bronze badges

Sign up to request clarification or add additional context in comments.

10 Comments

Prophet Over a year ago

Sure. I can't do it right now, but I will take a look in 20-30 minutes

Prophet Over a year ago

Can you please clarify: what exactly are you trying to achieve? The location of mouse cursor or Google search input web element? In case you are talking about the former - I can't see you scrolling the web page, so why should the location of the web element change and also if so why / how it is connected with mouse you mentioned in the original question?

Prophet Over a year ago

OK, seems that is what undetected Selenium initially gave you. His code prints the location of that element. But why should the location of that element change if you don't scroll the entire web page?

Prophet Over a year ago

Thank you for letting me know, that's really interesting. I'm sorry I could not help you more, was very busy this evening.

Prophet Over a year ago

Sure. In case you need more my assistance please let me know

|

undetected Selenium · Accepted Answer · 2022-03-27 16:32:13Z

1

To extract the location and/or position an element you need to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following locator strategies:

Using location attribute:

driver.get('https://www.google.com/')
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.NAME, "q"))).location)

Console Output:
```
{'x': 439, 'y': 184}
```

Using location_once_scrolled_into_view attribute:

driver.get('https://www.google.com/')
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.NAME, "q"))).location_once_scrolled_into_view)

Console Output:
```
{'x': 439, 'y': 184}
```

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

This usecase

As per the HTML details:

input which name="q" and class="s-input s-input__search js-search-field "

To click on the clickable element you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following locator strategies:

Using CSS_SELECTOR:

WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input.s-input.s-input__search.js-search-field[name='q']"))).click()

Using XPATH:

WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[@class='s-input s-input__search js-search-field ' and @name='q']"))).click()

Alternative

As an alternative you can also use the move_to_element() method from ActionChains API as follows:

ActionChains(driver).move_to_element(WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[@class='s-input s-input__search js-search-field ' and @name='q']")))).perform()

edited Mar 27, 2022 at 16:32

answered Mar 27, 2022 at 15:25

undetected Selenium

194k44 gold badges304 silver badges387 bronze badges

12 Comments

undetected Selenium Over a year ago

Ideally your screen should be always maximized options.add_argument("start-maximized") while executing your tests. The example was of the search field on Google Home Page. Checkout the answer update and let me know the status.

undetected Selenium Over a year ago

Fixed the CSS_SELECTOR. Well we are not printing the current mouse position as that would involve different set of Mouse actions APIs. We are printing the element identified by the respective locator strategies

undetected Selenium Over a year ago

Updated my previous comment with link to usage of Mouse actions APIs.

undetected Selenium Over a year ago

@Saeed I would suggest hold on the idea of using the Mouse actions APIs, use the expected_conditions if that works. Else I will update the answer with another approach.

undetected Selenium Over a year ago

Moreover location through Selenium and pyautogui may differ significantly.

|

Umair Ayub · Accepted Answer · 2023-11-20 06:54:23Z

1

Here is how I did it.

import pyautogui
from random import *

elementToClick = driver.find_element(by=By.CSS_SELECTOR,value='#nav-search-submit-button')

panel_height = driver.execute_script('return window.outerHeight - window.innerHeight;')
abs_x = elementToClick.location['x']
y = elementToClick.location['y']
abs_y = y + panel_height

pyautogui.moveTo(abs_x, abs_y, uniform(0.6, 1.7), pyautogui.easeOutQuad)
pyautogui.click(abs_x, abs_y)

answered Nov 20, 2023 at 6:54

Umair Ayub

21.7k14 gold badges82 silver badges154 bronze badges

Collectives™ on Stack Overflow

How to find position of an element using pyautogui and selenium automatically

3 Answers 3

10 Comments

This usecase

Alternative

12 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

10 Comments

This usecase

Alternative

12 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related