Headless mode disable python web file downloading

Question

I aim to download web files while in headless mode. My program downloads perfectly when NOT in headless mode, but once I add the constraint not to show MS Edge opening, the downloading is disregarded.

import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import Select

driver = webdriver.Edge()
driver.get("URL")

id_box = driver.find_element(By.ID,"...")
pw_box = driver.find_element(By.ID,"...")
id_box.send_keys("...")
pw_box.send_keys("...")
log_in = driver.find_element(By.ID,"...")
log_in.click()

time.sleep(0.1) # If not included, get error: "Unable to locate element"

drop_period = Select(driver.find_element(By.ID,"..."))
drop_period.select_by_index(1)
drop_consul = Select(driver.find_element(By.ID,"..."))
drop_consul.select_by_visible_text("...")
drop_client = Select(driver.find_element(By.ID,"..."))
drop_client.select_by_index(1)

# Following files do not download with headless inculded:

driver.find_element(By.XPATH, "...").click()
driver.find_element(By.XPATH, "...").click()

kaliiiiiiiii · Accepted Answer · 2023-01-20 10:30:20Z

0

In that case, you might try downloading the file using the direct link (to the file) and python requests.

You'll need to get the url, by parsing the elemt its href:

Downloading and saving a file from url should work as following then:

import requests as req

remote_url = 'http://www.example.com/file.txt'
local_file_name = 'my_file.txt'

data = req.get(remote_url)

# Save file data to local copy
with open(local_file_name, 'wb')as file:
    file.write(data.content)

resource

answered Jan 20, 2023 at 10:30

kaliiiiiiiii

1,1571 gold badge8 silver badges30 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Erik Over a year ago

I can't find a URL for the file download, only to the website. The file download only appears after I have selected drop down lists, and the page automatically update.

kaliiiiiiiii Over a year ago

Can you add the page_source or url to the website, and the desired button element?

Erik Over a year ago

I can share the URL: interactive.advantage.am/Portal/Secure/…

Erik Over a year ago

However, information I download on the webpage is discretionary, I can not provide access to that page.

kaliiiiiiiii Over a year ago

Hmm I think there should be a way to identify the direct link. You might try using the network (browser-developer-tools) tab to get the url to the file-request.

|

Michael Mintz · Accepted Answer · 2023-01-20 17:48:31Z

0

There are different headless modes for Chrome. If you want to download files, use one of the special ones.

For Chrome 109 and above, use:

options.add_argument("--headless=new")

For Chrome 108 and below, use:

options.add_argument("--headless=chrome")

Reference: https://github.com/chromium/chromium/commit/e9c516118e2e1923757ecb13e6d9fff36775d1f4

answered Jan 20, 2023 at 17:48

Michael Mintz

16.1k9 gold badges51 silver badges109 bronze badges

Comments

Badufz · Accepted Answer · 2023-02-17 10:09:40Z

0

Downloading files in headless mode works for me on MicrosoftEdge version 110.0.1587.41 using following options:

    MicrosoftEdge: [{
        "browserName": "MicrosoftEdge",
        "ms:edgeOptions": {
            args: ['--headless=new'],
            prefs: {
                "download.prompt_for_download": false,
                "plugins.always_open_pdf_externally": true,
                'download.default_directory': "dlFolder"
            }
        },
    }]

Nothing worked until I added the option '--headless=new'

N.B: Tested on a Mac environment using webdriverIO

edited Feb 17, 2023 at 10:09

answered Feb 17, 2023 at 10:08

Badufz

11 bronze badge

Comments

Kristina G · Accepted Answer · 2023-03-21 12:16:18Z

The options.add_argument("headless=new") syntax also works for Edge.

I previously used the following syntax to open Edge in headless mode:

from selenium import webdriver
from selenium.webdriver.edge.options import Options

options = Options()
options.add_experimental_option("prefs", {"download.default_directory": my_download_folder, "download.prompt_for_download": False, 'profile.default_content_settings.popups': False})     
options.add_experimental_option("excludeSwitches", ["enable-logging"])
options.add_argument('log-level=3') 
options.headless = True
browser = webdriver.Edge(options=options)
browser.get(url)

The above still works fine (opens the browser in headless mode, clicks links, etc), but doesn't allow file downloads. (You can click on a download link, but nothing happens). New syntax fixes this issue:

from selenium import webdriver
from selenium.webdriver.edge.options import Options

options = Options()
options.add_experimental_option("prefs", {"download.default_directory": my_download_folder, "download.prompt_for_download": False, 'profile.default_content_settings.popups': False})     
options.add_experimental_option("excludeSwitches", ["enable-logging"])
options.add_argument('log-level=3') 
options.add_argument("headless=new")
browser = webdriver.Edge(options=options)
browser.get(url)

Collectives™ on Stack Overflow

Headless mode disable python web file downloading

4 Answers 4

6 Comments

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

6 Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related