Python/JSON: How to Resolve UnicodeDecodeError

Question

I have been trying to learn Python recently and following along with the book, Python for Data Analysis and using Python 2.7 with Canopy. In the book, they provided a link to some raw data which I saved and assigned to a path variable. After trying to convert the text file to a list of dictionaries using JSON:

records = [json.loads(line) for line in open(path)]

I received the following error:

---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-17-b1e0b494454a> in <module>()
----> 1 records = [json.loads(line) for line in open(path)]

C:\Users\Marc\AppData\Local\Enthought\Canopy\App\appdata\canopy-1.4.1.1975.win-    x86_64\lib\json\__init__.pyc in loads(s, encoding, cls, object_hook, parse_float, parse_int,     parse_constant, object_pairs_hook, **kw)
    336             parse_int is None and parse_float is None and
    337             parse_constant is None and object_pairs_hook is None and not kw):
--> 338         return _default_decoder.decode(s)
    339     if cls is None:
    340         cls = JSONDecoder

C:\Users\Marc\AppData\Local\Enthought\Canopy\App\appdata\canopy-1.4.1.1975.win-    x86_64\lib\json\decoder.pyc in decode(self, s, _w)
    363 
    364         """
--> 365         obj, end = self.raw_decode(s, idx=_w(s, 0).end())
    366         end = _w(s, end).end()
    367         if end != len(s):

C:\Users\Marc\AppData\Local\Enthought\Canopy\App\appdata\canopy-1.4.1.1975.win-x86_64\lib\json\decoder.pyc in raw_decode(self, s, idx)
    379         """
    380         try:
--> 381             obj, end = self.scan_once(s, idx)
    382         except StopIteration:
    383             raise ValueError("No JSON object could be decoded")

UnicodeDecodeError: 'utf8' codec can't decode byte 0x92 in position 6: invalid start byte

The weird thing is that this worked on a different computer, which I thought was using the same version of Python. Thanks in advance.

Your JSON data is not UTF-8 encoded, it appears. Are you 100% certain it is the same data you are decoding? — Martijn Pieters
– Martijn Pieters, Commented Oct 11, 2014 at 16:03
The "default" encoding that Python will try to use can vary from one computer to the next depending on that system's locale settings, even if the same version of Python is used. This behavior is intentional. — skrrgwasme
– skrrgwasme, Commented Oct 11, 2014 at 16:07
@SLawson: this has nothing to do with default (file) encodings. This is Python 2, where no decoding of file data takes place. The json module does decode, because the JSON standard specifies a (limited) number of encodings that are acceptable. — Martijn Pieters
– Martijn Pieters, Commented Oct 11, 2014 at 16:09
@SLawson: in Python 3, the open() call without explicit encoding parameter will indeed use the system locale to determine a default codec, but that doesn't apply here. — Martijn Pieters
– Martijn Pieters, Commented Oct 11, 2014 at 16:10
The data is definitely the same, maybe I'm wrong about the other computer's Python version though. — user2899059
– user2899059, Commented Oct 11, 2014 at 16:15

Community · Accepted Answer · 2021-10-07 06:06:50Z

2

The data in question contains one U+2019 RIGHT SINGLE QUOTATION MARK character, encoded to UTF-8. But you used copy-and-paste to save the data rather than save the text straight to disk.

In doing so, somewhere along the way the data was decoded, then encoded again, to Windows Codepage 1252:

>>> u'\u2019'.encode('cp1252')
'\x92'

In other words, your data file is not the same. It probably contains the same data but using a different encoding.

The JSON standard states data needs to be encoded to UTF-8, UTF-16 or UTF-32, with UTF-8 being the default, and that is what the Python json module will use if you don't give it an encoding. Because you are feeding it CP-1252 data instead, the decoding fails.

edited Oct 7, 2021 at 6:06

CommunityBot

11 silver badge

answered Oct 11, 2014 at 16:29

Martijn Pieters

1.1m326 gold badges4.2k silver badges3.4k bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python/JSON: How to Resolve UnicodeDecodeError

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related