Extend Selenium Timeouts to avoid urllib3 read timeout (120s)

Question

TL/DR: How can I extend the amount of time that Selenium waits before triggering a timeout? set_page_load_timeout() in isolation does not work as a ReadTimeoutError is still generated by urllib3.

Context: I am using Selenium to define a configuration (sent via a website form / POST request) and download the resulting CSV file. My code works well for small requests, but times out with large datasets that require > 120s to prepare and download.

I have attempted to update the timeout configuration for the Selenium webdriver to no avail:

driver.set_page_load_timeout(300)

The resulting error that I get is still:

urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='localhost', port=55676): Read timed out. (read timeout=120)

How can I go about increasing the urllib3 timeout within Selinium to be able to process/download these larger CSV files? Relevant code is below, but not sure it will be helpful:

from selenium import webdriver
from selenium.webdriver.common.by import By
import time, glob, os, zipfile
from url_destinations import url_destinations

target_data = url_destinations["OTP"]

# Selenium Code to Initiate Download
chrome_options = webdriver.ChromeOptions()

prefs = {"download.default_directory": r"C:\Users\<hidden>\data\downloads"}
chrome_options.add_experimental_option("detach", True)
chrome_options.add_experimental_option("prefs", prefs)

driver = webdriver.Chrome(options=chrome_options)
driver.set_page_load_timeout(300)
driver.get(target_data['URL'])

latest_data = driver.find_element(By.ID, value="lblLatest").text

for val in target_data["Check Options"]:
    selected_item = driver.find_element(By.ID,value=val)
    selected_item.click()

# Wait for Download to Complete
while len(glob.glob(prefs["download.default_directory"]+"\*.tmp")) > 0:
    time.sleep(0.5)

Blaise · Accepted Answer · 2025-06-19 07:42:52Z

3

In Selenium 4.x, the default timeout can be defined using driver.command_executor.set_timeout(1000)

~~or environment variable GLOBAL_DEFAULT_TIMEOUT~~ no longer supported in newer versions.

edited Jun 19 at 7:42

answered Jan 28 at 17:13

Blaise

13.6k9 gold badges79 silver badges103 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Pikamander2 Apr 16 at 10:35

driver.command_executor.set_timeout works for now, but it throws a deprecation warning.

DeprecationWarning: set_timeout() in RemoteConnection is deprecated, set timeout to ClientConfig instance in constructor instead

Corey Goldberg Jun 16 at 13:05

Note that support for GLOBAL_DEFAULT_TIMEOUT environment variable has been removed.

Michael Koehler · Accepted Answer · 2025-01-22 17:42:20Z

2

I've been able to develop a workaround for this issue using the following line of code

self.webdriver.command_executor._client_config._timeout = 10000

With that being said, I recognize that it's not proper to access the '_timeout' variable directly (which is an element of the "ClientConfig" class in Selenium. I would welcome any thoughts on how to correctly implement this fix. Revised code below:

from selenium import webdriver
from selenium.webdriver.common.by import By
import time, glob, os, zipfile
from url_destinations import url_destinations

target_data = url_destinations["OTP"]

# Selenium Code to Initiate Download
chrome_options = webdriver.ChromeOptions()

prefs = {"download.default_directory": r"C:\Users\<hidden>\data\downloads"}
chrome_options.add_experimental_option("detach", True)
chrome_options.add_experimental_option("prefs", prefs)

driver = webdriver.Chrome(options=chrome_options)
self.webdriver.command_executor._client_config._timeout = 10000

latest_data = driver.find_element(By.ID, value="lblLatest").text

for val in target_data["Check Options"]:
    selected_item = driver.find_element(By.ID,value=val)
    selected_item.click()

# Wait for Download to Complete
while len(glob.glob(prefs["download.default_directory"]+"\*.tmp")) > 0:
    time.sleep(0.5)

answered Jan 22 at 17:42

Michael Koehler

631 gold badge2 silver badges5 bronze badges

1 Comment

Corey Goldberg Jun 16 at 13:06

You can now access timeout.

Collectives™ on Stack Overflow

Extend Selenium Timeouts to avoid urllib3 read timeout (120s)

2 Answers 2

2 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related