Basic http file downloading and saving to disk in python?

For Python3+ URLopener is deprecated. And when used you will get error as below:

url_opener = urllib.URLopener() AttributeError: module 'urllib' has no attribute 'URLopener'

So, try:

import urllib.request 
urllib.request.urlretrieve(url, filename)

As mentioned here:

import urllib
urllib.urlretrieve ("http://randomsite.com/file.gz", "file.gz")

EDIT: If you still want to use requests, take a look at this question or this one.

Four methods using wget, urllib and request.

#!/usr/bin/python
import requests
from io import StringIO
from PIL import Image
import profile as profile
import urllib
import wget


url = 'https://tinypng.com/images/social/website.jpg'

def testRequest():
    image_name = 'test1.jpg'
    with requests.get(url, stream=True) as r:
        r.raise_for_status()
        with open(image_name, 'wb') as f:
            for chunk in r.iter_content():
                f.write(chunk)

def testRequest2():
    image_name = 'test2.jpg'
    r = requests.get(url)
    r.raise_for_status()
    i = Image.open(StringIO(r.content))
    i.save(image_name)

def testUrllib():
    image_name = 'test3.jpg'
    testfile = urllib.URLopener()
    testfile.retrieve(url, image_name)

def testwget():
    image_name = 'test4.jpg'
    wget.download(url, image_name)

if __name__ == '__main__':
    profile.run('testRequest()')
    profile.run('testRequest2()')
    profile.run('testUrllib()')
    profile.run('testwget()')

testRequest - 4469882 function calls (4469842 primitive calls) in 20.236 seconds

testRequest2 - 8580 function calls (8574 primitive calls) in 0.072 seconds

testUrllib - 3810 function calls (3775 primitive calls) in 0.036 seconds

testwget - 3489 function calls in 0.020 seconds

I use wget.

Simple and good library if you want to example?

import wget

file_url = 'http://johndoe.com/download.zip'

file_name = wget.download(file_url)

wget module support python 2 and python 3 versions

Exotic Windows Solution

import subprocess

subprocess.run("powershell Invoke-WebRequest {} -OutFile {}".format(your_url, filename), shell=True)

For text files, you can use:

import requests

url = 'https://WEBSITE.com'
req = requests.get(url)
req.raise_for_status()
path = "C:\\YOUR\\FILE.html"

with open(path, 'wb') as f:
    f.write(req.content)

I started down this path because ESXi's wget is not compiled with SSL and I wanted to download an OVA from a vendor's website directly onto the ESXi host which is on the other side of the world.

I had to disable the firewall(lazy)/enable https out by editing the rules(proper)

created the python script:

import ssl
import shutil
import tempfile
import urllib.request
context = ssl._create_unverified_context()

dlurl='https://somesite/path/whatever'
with urllib.request.urlopen(durl, context=context) as response:
    with open("file.ova", 'wb') as tmp_file:
        shutil.copyfileobj(response, tmp_file)

ESXi libraries are kind of paired down but the open source weasel installer seemed to use urllib for https... so it inspired me to go down this path

For those who want a solution with requests, here you go:

import requests

url = 'https://static.wikia.nocookie.net/dqw4w9wgxcq/images/0/08/Site-background-dark/revision/latest?cb=20220428173233'
path = "myfile.jpg"

with requests.get(url, stream=True) as r:
    r.raise_for_status()
    with open(path, 'wb') as f:
        for chunk in r.iter_content():
            f.write(chunk)

Note that:

• session.get() should also work, if you already have a requests.Session() (which is generally a good idea if you are making many requests.)
• raise_for_status() checks whether the HTTP status code is an error. Without that you might blindly download a json or HTML error page and save it with a .jpg file extension. Without it, you won't notice what's gone wrong until you've wasted time debugging why your .jpg (or .mp3 or whatever format you're expecting) is corrupt. (The other solutions with requests didn't include this, until I edited them to add it.)
• This stream=True approach means that you don't have to load the whole object into memory first. e.g. if you have 8GB of memory, you can still download a 16GB file with this approach
• We open the stream with a with context manager. This helps tidy things up. Otherwise it's possible to get issues with connections not being returned to the pool.

The urllib solution above works fine. Since urllib is pre-installed and requests not, you should probably only use this approach if you already have requests installed and imported.