Selenium: How to find element in <script src="xx.js"> [python]

Question

I'd like to execute find_element_by_class_name using selenium in google colaboratory.

The following error was displayed.

NoSuchElementException: Message: no such element: Unable to locate element

I found this page was generated by javascript after executing print(driver.page_source).

    <body>
        <script src="../public/vendor.bundle.js"></script>
        <script src="../public/bundle.js"></script>
    </body>

The class name <div class="Graph"> is displayed when I check the chrome developer tool (F12).

How can I get this class using selenium?

edit

html is as below;

<!doctype html>
<html>
    <head>
        <title>xxx</title>

        <link rel="shortcut icon" href="../public/favicon.ico" />
        <link rel="stylesheet" type="text/css" href="../public/bundle.css">

        <script type="text/javascript">

            var contextPath = "/xxx";

            if(location.pathname.match(/(.*)(\/\S*\/\S*)/)){
                contextPath = RegExp.$1;
            }

            window.raise = window.raise || {};
            window.raise.appInfo = {
                "xxx" :  'xxx',
                "xxx" : xxx,
                .
                .
                .
            };
        </script>
    </head>
    <body>
        <div id="app-progress" class="AppInitializing"></div>
        <div id="root">

            <noscript>
                <p class="xxx">xxx</p>
            </noscript>
        </div>

        <script src="../public/vendor.bundle.js"></script>
        <script src="../public/bundle.js"></script>
    </body>
    
    
</html>

edit

"Elements" tab in Chrome developer tool is as below;

.
.
<div class="main-body-wrapper">
    <div class="main-body-area">
        <div class="page-grid single GraphPage">
            <h3 class="page-title">...</h3>
                <div class="graph-container">
                    <div class="Graph">
                        <div class="loading">...</div>
                        <div>...</div>
.
.

Python code is as below;

.
.
driver.find_element_by_id("openGraphPage").click()
time.sleep(3)
driver.get(r'https://example.com/graph')
time.sleep(10)
print(driver.current_url)
print(driver.page_source)

graph = driver.find_element_by_class_name("Graph")
.
.

Although I tried find_element_by_xpath, the result was same.

@PDHide Thank you your comment. Yes, I added time.sleep(3) — SamuraiBlue
– SamuraiBlue, Commented Dec 18, 2020 at 22:05
Please add the html , see whether the element is inside iframe — PDHide
– PDHide, Commented Dec 18, 2020 at 22:10
@PDHide I added the html. I'm afraid this page doesn't use iframe. — SamuraiBlue
– SamuraiBlue, Commented Dec 18, 2020 at 22:28
... What am I missing? The class "Graph" isn't found on the html page... — Shmack
– Shmack, Commented Dec 21, 2020 at 1:44

jackblk · Accepted Answer · 2020-12-22 07:20:10Z

Sorry for late response from last comment, I think I know your problem now after your last comment.

# After this action, it will open a new tab
driver.find_element_by_id("openGraphPage").click()

Let's call your first tab tab0 and the new tab tab1.

After clicking to open new tab, even if you see that current tab is tab1, Selenium driver is still focusing on tab0! This is why when you do print(driver.page_source), it does not show any elements that you see in tab1, thus you cannot find your element.

In order to switch the tab after opening new tab, you need to do this:

# Firstly get all window handles (tab ids)
window_list = driver.window_handles
tab0 = window_list[0]
tab1 = window_list[1]

# Switch to your new tab
driver.switch_to.window(tab1)

# Do your stuff here
graph = driver.find_element_by_class_name("Graph")
# ...

# If you need to come back to your main tab (even if closing tab1, you still need to switch back)
driver.switch_to.window(tab0)

Notes:

I'm not totally sure why it does not work when you do driver.get(r'https://example.com/graph'), it does not find your element. Because even if you do not switch tab, if you do driver.get, the first tab should get your graph normally.

Since I don't have access to the site, my guesses are:

Directly do driver.get(r'https://example.com/graph') will not load your graph. It will need to click the open graph button, it will include some arguments to load your graph in new tab. --> If this is the reason, you have no other way but to open the graph by clicking the button, then switch to new tab and do your stuff there.
After clicking to open it in new tab, you get the page directly in the first tab, but the page does not load normally if not focused (it will happen for lots of javascript sites). That's why you cannot find your element. But if you debug it normally by using Chrome, you focus the page, so it loads. --> If this is the reason, you can just simply switch to this tab, focus it. It will load the graph and you can get your element.

Would love to help you more, but I don't have the access to the site so I just guess left and right.

Tips:

Try not using time.sleep, instead use ExplicitWait. It will stop waiting when it finds your element, so you do not waste time. It will still raise NoSuchElement when the timeout is reached.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

element = WebDriverWait(driver, 10).until(
        EC.visibility_of_element_located((By.CLASS_NAME, "Graph"))
    )

Further documentation: https://selenium-python.readthedocs.io/waits.html#explicit-waits

Thank you for your in-depth explanation, @jackblk. I can take a step forward though I can't do what I want. Because I was redirected to the url of not supporting browser... Selenium may not work as Chrome even if I use chromedriver for selenium... But I think your answer works. Thanks a lot!!
Have you tried Firefox? It's easy to switch your code to firefox. Glad it helped you :).

Collectives™ on Stack Overflow

Selenium: How to find element in <script src="xx.js"> [python]

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related