Python unicodeDecodeError on parsing a JSON url

Question

I am using python 3.4 and trying to parse what seems like valid JSON output from a url. ex: http://api.stackexchange.com/2.2/questions?order=desc&sort=activity&site=stackoverflow

This is what my code looks like

import json
from urllib.request import urlopen


def jsonify(url):
    response = urlopen(url).read().decode('utf8')
    repo = json.loads(response)
    return repo 


 url = jsonify('http://api.stackexchange.com/2.2/questions?order=desc&sort=activity&site=stackoverflow');

However, I get errors such as UnicodeDecodeError utf-8 codec can't decode byte 0x8b in position 1; invalid start byte

The script works with any other API, like github and so many others, but not with the stackexchange api

@DanD. I get Invalid character in identifier and it points to the . in after headers — user7342807
– user7342807, Commented Apr 6, 2017 at 6:20

xrisk · Accepted Answer · 2017-04-06 06:52:28Z

2

The response is compressed using gzip, you have to decompress it.

$ curl -v http://api.stackexchange.com/2.2/questions\?order\=desc\&sort\=activity\&site\=stackoverflow
*   Trying 198.252.206.16...
* TCP_NODELAY set
* Connected to api.stackexchange.com (198.252.206.16) port 80 (#0)
> GET /2.2/questions?order=desc&sort=activity&site=stackoverflow HTTP/1.1
> Host: api.stackexchange.com
> User-Agent: curl/7.51.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Cache-Control: private
< Content-Type: application/json; charset=utf-8
< Content-Encoding: gzip

See the api.stackexchange docs for more details.

Example of decompression:

import gzip

def jsonify(url):
    response = urlopen(url).read()
    tmp = gzip.decompress(response).decode('utf-8')
    repo = json.loads(tmp)
    return repo

edited Apr 6, 2017 at 6:52

answered Apr 6, 2017 at 6:26

xrisk

3,91826 silver badges48 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

user7342807 Over a year ago

It still doesn't work. I get TypeError: the JSON object must be 'str' not 'bytes'

xrisk Over a year ago

@user7342807 Doesn't work? Which Python version are you on? It works for me on Python 3.6.

user7342807 Over a year ago

python3 -V shows 3.4.5 I am execute my script via python3 fetch_url.py

Collectives™ on Stack Overflow

Python unicodeDecodeError on parsing a JSON url

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related