python3 urlopen raises 404 error while openning in the browser

Question

I'm using urllib.request with python 3.4.6 to open https://www.ethz.ch/ (The actual url is longer but the problem is the same), which opens fine with Firefox but throws a 404 error with python.

Here is the code

from urllib.request import urlopen
connection = urlopen('https://www.ethz.ch/')

and it gives the following error message

Traceback (most recent call last):
  File "./generate_group_meetings_ical.py", line 9, in <module>
    connection = urlopen('https://www.ethz.ch/')
  File "/usr/lib64/python3.4/urllib/request.py", line 161, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib64/python3.4/urllib/request.py", line 470, in open
    response = meth(req, response)
  File "/usr/lib64/python3.4/urllib/request.py", line 580, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib64/python3.4/urllib/request.py", line 508, in error
    return self._call_chain(*args)
  File "/usr/lib64/python3.4/urllib/request.py", line 442, in _call_chain
    result = func(*args)
  File "/usr/lib64/python3.4/urllib/request.py", line 588, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not found UA

The code used to work fine though. Another piece of information is that I'm not root on the machine and python3 was upgraded from 3.4.5 to 3.4.6. So the comes either from the web server side or from the python side. I'm not a python nor a web expert so I couldn't figure it out myself.

Hope anybody can help me.

Sounds like a user agent problem, try setting the user agent string to something else to see if that's the problem. — Francisco
– Francisco, Commented Jul 5, 2017 at 9:40
Thanks a lot Francisco, it solved the problem. I posted an answer describing it. — Pheidippides
– Pheidippides, Commented Jul 5, 2017 at 12:56
Just had the same problem while building a scraper for exercise sheets at the same university lol — technical_difficulty
– technical_difficulty, Commented Sep 25, 2017 at 18:52

Pheidippides · Accepted Answer · 2017-07-05 12:53:57Z

1

Thanks to Francisco's comment and that post I could make it work with the following code

from urllib.request import Request, urlopen
req = Request('https://www.ethz.ch/', headers={'User-Agent': 'Mozilla/5.0'})
connection = urlopen(req)

I also checked the original version with python 2.7.13 and urllib2 and it worked. Apparently python 3.5 works (answer from Laxmikant) and it was originally working under 3.4.5. So something happened in the upgrade from 3.4.5 to 3.4.6 that caused the error.

answered Jul 5, 2017 at 12:53

Pheidippides

977 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Laxmikant · Accepted Answer · 2017-07-05 09:33:31Z

0

@Pheidippides Check for typos if any in your entire url, it worked for me:

Python 3.5.2 (default, Nov 17 2016, 17:05:23) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
$>from urllib.request import urlopen
$>connection = urlopen('https://www.ethz.ch/')
$>connection.read()

answered Jul 5, 2017 at 9:33

Laxmikant

2,2564 gold badges33 silver badges45 bronze badges

1 Comment

Pheidippides Over a year ago

Thanks @Laxmikant, I ran exactly the commands you entered but at least I now know that it works with python 3.5

Collectives™ on Stack Overflow

python3 urlopen raises 404 error while openning in the browser

2 Answers 2

Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related