Downloading a file from the command line using Python

Question

I am looking for a quick way to download a file via HTTP, using a Python one-liner from the command line (similar to the functionality of wget or curl). The idea is to enable a quick copy/paste to download distutils on Windows.

I know of one solution (see my answer below). I'm interested in other solutions that consider the following:

Concise
Most "Pythonic" solution
Compatible with both Python 2 and Python 3
Cross-platform
Can deal with large files efficiently
No dependencies (we're fetching distutils here. It's unlikely we'll have access to requests at this stage)
Correctly handles various HTTP headers, such as Content-Disposition

Distutils and setuptools have merged. To fulfill my original goal: c:\python27\python.exe -c "from urllib import urlretrieve; urlretrieve('https://bitbucket.org/pypa/setuptools/raw/bootstrap/ez_setup.py', 'ez_setup.py')" & c:\python27\python.exe ez_setup.py — dwurf
– dwurf, Commented Aug 13, 2013 at 14:07

Peter Mortensen · Accepted Answer · 2022-05-01 15:17:17Z

13

The simplest solution I could come up with would be:

try:
    from urllib.request import urlretrieve
except ImportError:
    from urllib import urlretrieve

urlretrieve('http://example.org', 'outfile.dat')

urlretrieve takes care of downloading the resource to a local file and can deal with large files.

It, however, ignores Content-Disposition headers. If you want that to be considered, you'd need to use urlopen and parse the response headers yourself. Content-Disposition isn't a HTTP standard header, so I doubt you will find much support for it in the Python HTTP libraries...

edited May 1, 2022 at 15:17

Peter Mortensen

31.4k22 gold badges110 silver badges134 bronze badges

answered Jun 18, 2013 at 8:29

mata

69.3k10 gold badges168 silver badges162 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

dwurf Over a year ago

The main thing this doesn't have is the ability to run it via a CLI one-liner. I'll mark this as correct since it can be split into two statements, one for python2 and one for python3, that can handle large files and be run as a one-liner.

dwurf Over a year ago

Python 2:

python -c "from urllib import urlretrieve; urlretrieve('http://python-distribute.org/distribute_setup.py', 'distribute_setup.py')"

dwurf Over a year ago

Python 3:

python3 -c "from urllib.request import urlretrieve; urlretrieve('http://python-distribute.org/distribute_setup.py', 'distribute_setup.py')"

Scott Prive Over a year ago

Dwurf has the answer correct: the original question wanted this as a one-liner suitable for invoking directly from the command line. (And coincidentally, the search for which brought me here).

iuridiniz Over a year ago

Content-Disposition is a standard now. If recent python releases implemented it, well, I don't know. w3.org/Protocols/rfc2616/rfc2616-sec19.html --> 19.5.1 Content-Disposition

dwurf · Accepted Answer · 2013-06-18 07:28:34Z

6

My solution is:

python -c "import urllib; print urllib.urlopen('http://python-distribute.org/distribute_setup.py').read()" > distribute_setup.py

answered Jun 18, 2013 at 7:28

dwurf

12.9k6 gold badges33 silver badges45 bronze badges

1 Comment

mata Over a year ago

this is really bad for large files, and it only works on python2. not really what the OP wanted.

Tom Pohl · Accepted Answer · 2024-09-11 09:41:19Z

0

Here's my one-liner avoiding the deprecated urlretrieve:

python -c 'from urllib.request import urlopen; from pathlib import Path; Path("outfile.dat").write_bytes(urlopen("http://example.org").read())'

answered Sep 11, 2024 at 9:41

Tom Pohl

3,1943 gold badges31 silver badges36 bronze badges

Collectives™ on Stack Overflow

Downloading a file from the command line using Python

3 Answers 3

5 Comments

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

5 Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related