62

What's the best way to specify a proxy with username and password for an http connection in python?

Improve this question

6 Answers 6

Reset to default
58

This works for me: 

import urllib2

proxy = urllib2.ProxyHandler({'http': 'http://
username:password@proxyurl:proxyport'})
auth = urllib2.HTTPBasicAuthHandler()
opener = urllib2.build_opener(proxy, auth, urllib2.HTTPHandler)
urllib2.install_opener(opener)

conn = urllib2.urlopen('http://python.org')
return_str = conn.read()
Improve this answer
Sign up to request clarification or add additional context in comments.

6 Comments

urllib2.HTTPHandler is added by default (see urllib2 doc). It seems it is redundant to add it when building opener.
And why do you use urllib2.HTTPBasicAuthHandler() if no authentication is involved?
Normally proxies are in an IP format. That's the same as proxyurl, right?
Yes. I guess if you've got an IP Address for your proxy you can put this IP address as proxyurl in there.
I used the same methods but a different order to input the info and it did not work, thanks ! proxy = urllib2.ProxyHandler({'http': '172.24.8.69:8080'}) #proxy = urllib2.ProxyHandler({}) proxy_auth_handler = urllib2.HTTPBasicAuthHandler() proxy_auth_handler.add_password(None, 'ixx.groupe', 'poutrathor', 'PoutrathorPassword') opener = urllib2.build_opener(proxy, proxy_auth_handler) opener = urllib2.build_opener(proxy) urllib2.install_opener(opener) response = urllib2.urlopen('google.com') html = response.read() print html
|
29

Use this:

import requests

proxies = {"http":"http://username:password@proxy_ip:proxy_port"}

r = requests.get("http://www.example.com/", proxies=proxies)

print(r.content)

I think it's much simpler than using urllib. I don't understand why people love using urllib so much.

Improve this answer

5 Comments

This is the best script probably and the best thing about using requests is you don't have to worry about whether it is Python 2 or 3.
@Aminah Nuraini Could you please explain what do you mean by username:password here?
@Oshada, replace it with your proxy's username and password. You usually get it from the provider.
People sometimes have to use what libraries exist in servers. Can't install libraries or limited access.
How do I get these values?
18

Setting an environment var named http_proxy like this: http://username:password@proxy_url:port

Improve this answer

2 Comments

But not really answering the answer.
@Bouncner Answering the Question, not the Answer. And it's not answering neither.
10

The best way of going through a proxy that requires authentication is using urllib2 to build a custom url opener, then using that to make all the requests you want to go through the proxy. Note in particular, you probably don't want to embed the proxy password in the url or the python source code (unless it's just a quick hack).

import urllib2

def get_proxy_opener(proxyurl, proxyuser, proxypass, proxyscheme="http"):
    password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
    password_mgr.add_password(None, proxyurl, proxyuser, proxypass)

    proxy_handler = urllib2.ProxyHandler({proxyscheme: proxyurl})
    proxy_auth_handler = urllib2.ProxyBasicAuthHandler(password_mgr)

    return urllib2.build_opener(proxy_handler, proxy_auth_handler)

if __name__ == "__main__":
    import sys
    if len(sys.argv) > 4:
        url_opener = get_proxy_opener(*sys.argv[1:4])
        for url in sys.argv[4:]:
            print url_opener.open(url).headers
    else:
        print "Usage:", sys.argv[0], "proxy user pass fetchurls..."

In a more complex program, you can seperate these components out as appropriate (for instance, only using one password manager for the lifetime of the application). The python documentation has more examples on how to do complex things with urllib2 that you might also find useful.

Improve this answer

Comments

3

Or if you want to install it, so that it is always used with urllib2.urlopen (so you don't need to keep a reference to the opener around):

import urllib2
url = 'www.proxyurl.com'
username = 'user'
password = 'pass'
password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
# None, with the "WithDefaultRealm" password manager means
# that the user/pass will be used for any realm (where
# there isn't a more specific match).
password_mgr.add_password(None, url, username, password)
auth_handler = urllib2.HTTPBasicAuthHandler(password_mgr)
opener = urllib2.build_opener(auth_handler)
urllib2.install_opener(opener)
print urllib2.urlopen("http://www.example.com/folder/page.html").read()
Improve this answer

Comments

2

Here is the method use urllib 

import urllib.request

# set up authentication info
authinfo = urllib.request.HTTPBasicAuthHandler()
proxy_support = urllib.request.ProxyHandler({"http" : "http://ahad-haam:3128"})

# build a new opener that adds authentication and caching FTP handlers
opener = urllib.request.build_opener(proxy_support, authinfo,
                                     urllib.request.CacheFTPHandler)

# install it
urllib.request.install_opener(opener)

f = urllib.request.urlopen('http://www.python.org/')
"""
Improve this answer

2 Comments

And where is the proxy authentication in your example? Your ProxyHandler doesn't contain proxy auth details.
please refer to this link epydoc.sourceforge.net/stdlib/urllib2-module.html

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.