How do you open a URL with Python without using a browser?

Question

I want to open a URL with Python code but I don't want to use the "webbrowser" module. I tried that already and it worked (It opened the URL in my actual default browser, which is what I DON'T want). So then I tried using urllib (urlopen) and mechanize. Both of them ran fine with my program but neither of them actually sent my request to the website!

Here is part of my code:

finalURL="http://www.locationary.com/access/proxy.jsp?ACTION_TOKEN=proxy_jsp$JspView$SaveAction&inPlaceID=" + str(newPID) + "&xxx_c_1_f_987=" + str(ZA[z])

print finalURL

print ""

br.open(finalURL)

page = urllib2.urlopen(finalURL).read()

When I go into the site, locationary.com, it doesn't show that any changes have been made! When I used "webbrowser" though, it did show changes on the website after I submitted my URL. How can I do the same thing that webbrowser does without actually opening a browser?

I think the website wants a "GET"

Isn't there an '?' in proxy.jspACTION_TOKEN= like proxy.jsp?ACTION_TOKEN= — Paulo Scardine
– Paulo Scardine, Commented Dec 23, 2011 at 16:30
Do you know if the service wants a POST or a GET request? — Wayne Werner
– Wayne Werner, Commented Dec 23, 2011 at 16:30
Not specifically related to your problem, but you might want to note that urllib.urlopen() has been deprecated, and is removed it Python 3.0. Use urllib2.urlopen() — Amadan
– Amadan, Commented Dec 23, 2011 at 16:35
This is a good primer for urllib2: voidspace.org.uk/python/articles/urllib2.shtml#fetching-urls — sgallen
– sgallen, Commented Dec 23, 2011 at 16:38
Is an HTTPError raised? Can you provide an example of a url, minus the variables, that works using the 'webbrowser' module? If I attempt to construct a url based on your submission (locationary.com/access/…) and put it in my actual browser I get: {"manifest":{"errorTimeout":0,"succeed":true,"errorCode":0,"serverVersion":"1.0","type":"locaaccess"},"saveResult":{"message":"Yellowpages.com incomplete. Please enter a value or press \"cancel\".","placeOpenedState":0,"isSucess":false}} — sgallen
– sgallen, Commented Dec 23, 2011 at 17:15

Kirsten Jones · Accepted Answer · 2011-12-23 17:25:01Z

1

I'm not sure what OS you're working on, but if you use something like httpscoop (mac) or fiddler (pc) or wireshark, you should be able to watch the traffic and see what's happening. It may be that the website does a redirect (which your browser is following) or there's some other subsequent activity.

Start an HTTP sniffer, make the request using the web browser and watch the traffic. Once you've done that, try it with the python script and see if the request is being made, and what the difference is in the HTTP traffic. This should help identify where the disconnect is.

answered Dec 23, 2011 at 17:25

Kirsten Jones

2,8061 gold badge19 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Aaron Digulla · Accepted Answer · 2011-12-23 16:51:25Z

0

A HTTP GET doesn't need any specific code or action on the client side: It's just the base URL (http://server/) + path + optional query.

If the URL is correct, then the code above should work. Some pointers what you can try next:

Is the URL really correct? Use Firebug or a similar tool to watch the network traffic which gives you the full URL plus any header fields from the HTTP request.
Maybe the site requires you to log in, first. If so, make sure you set up cookies correctly.
Some sites require a correct "referrer" field (to protect themselves against deep linking). Add the referrer header which your browser used to the request.
The log file of the server is a great source of information to trouble shoot such problems - when you have access to it.

edited Dec 23, 2011 at 16:51

answered Dec 23, 2011 at 16:32

Aaron Digulla

330k111 gold badges626 silver badges840 bronze badges

4 Comments

jacob501 Over a year ago

Sorry. There is a question mark in my code but somehow it disappeared when I put it on here. I think I accidentally hit backspace.

jacob501 Over a year ago

This isn't really an answer to my question.

jacob501 Over a year ago

This doesn't answer my question. Please remove it.

Paulo Scardine Over a year ago

IMHO still more a comment than an answer.

Collectives™ on Stack Overflow

How do you open a URL with Python without using a browser?

2 Answers 2

Comments

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related