How to parse somewhat wrong JSON with Python?

Question

I have the following JSON string coming from external input source:

{value: "82363549923gnyh49c9djl239pjm01223", id: 17893}

This is an incorrectly-formatted JSON string ("id" and "value" must be in quotes), but I need to parse it anyway. I have tried simplejson and json-py and seems they could not be set up to parse such strings.

I am running Python 2.5 on Google App engine, so any C-based solutions like python-cjson are not applicable.

Input format could be changed to XML or YAML, in addition to JSON listed above, but I am using JSON within the project and changing format in specific place would not be very good.

Now I've switched to XML and parsing the data successfully, but looking forward to any solution that would allow me to switch back to JSON.

I'm a little confused about how you can switch to XML, yet not be in control of the JSON data. It sounds like you have some external source of data, in either XML or JSON formats, but its JSON output is permanently broken as shown and you can't do anything about it so your only option is to select the XML version instead? Or am I missing something? — Peter Hansen
– Peter Hansen, Commented Dec 19, 2009 at 0:35
you can parse it as YAML without a change, because it is YAML too — mykhal
– mykhal, Commented Dec 19, 2009 at 0:43
Peter, you're right - I have an external source of data which I could control only in one way - by saying I want an input in either JSON, XML or YAML. Nadia, thanks - that's my mistake (and due to I am not very familiar with Stackoverflow's interface at the time). — Serge Tarkovski
– Serge Tarkovski, Commented Dec 19, 2009 at 9:20

mykhal · Accepted Answer · 2009-12-19 00:48:21Z

64

since YAML (>=1.2) is a superset of JSON, you can do:

>>> import yaml
>>> s = '{value: "82363549923gnyh49c9djl239pjm01223", id: 17893}'
>>> yaml.load(s)
{'id': 17893, 'value': '82363549923gnyh49c9djl239pjm01223'}

answered Dec 19, 2009 at 0:48

mykhal

20.1k11 gold badges84 silver badges84 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

mykhal Over a year ago

well, python-yaml (PyYAML) is not yet fully 1.2 compliant, but will handle most cases. to be prepared for problem cases, see en.wikipedia.org/wiki/YAML#cite_ref-6

Serge Tarkovski Over a year ago

mykhal, have you run it on Google App Engine? Seems PyYAML uses C modules and thus cannot be used on GAE.

mykhal Over a year ago

pyyaml is much faster, if using libyaml, but it also is written in pure python, and you can choose between CLoader o Loader (pure py). But don't worry, yaml support is already included in app engine, you can try this in interactive shell shell.appspot.com

Gumbo Over a year ago

YAML is not a strict superset of JSON as YAML requires the mapping keys to be unique while JSON only suggests to use unique keys (MUST vs. SHOULD).

Adam Ernst Over a year ago

One more problem: YAML apparently requires a space after the colon. However for the most part this works like a charm.

|

Gino Mempin · Accepted Answer · 2022-04-25 08:33:05Z

26

You can use demjson.

>>> import demjson
>>> demjson.decode('{foo:3}')
{u'foo': 3}

edited Apr 25, 2022 at 8:33

Gino Mempin

30.5k31 gold badges125 silver badges174 bronze badges

answered Nov 26, 2014 at 17:43

null

9,19417 gold badges72 silver badges99 bronze badges

6 Comments

varela Over a year ago

That helped me to parse JSON without quotes and with formatting that differs from yaml

Johnner Over a year ago

very helpful package for parsing broken json, thanks

msanjay Over a year ago

handled nested objects as well which I found to be an issue with yaml. on windows: py -m pip install demjson ----------- then import demjson s = """get or define the multiline string inline""" j = demjson.decode(s) jsonString = demjson.encode(j)

lenhhoxung Over a year ago

Probably the best python lib for parsing json without quotes, many thanks.

Gino Mempin Over a year ago

The original link is dead: deron.meranda.us/python/demjson/. I edited in the package's page in pypi instead.

|

Gino Mempin · Accepted Answer · 2022-04-25 08:28:11Z

3

The dirtyjson library can handle some almost-correct JSON:

>>> import dirtyjson
>>> 
>>> s = '{value: "82363549923gnyh49c9djl239pjm01223", id: 17893}'
>>> d = dirtyjson.loads(s)
>>> d
AttributedDict([('value', '82363549923gnyh49c9djl239pjm01223'), ('id', 17893)])
>>>
>>> d = dict(d)
>>> d
{'value': '82363549923gnyh49c9djl239pjm01223', 'id': 17893}
>>> d["value"]
'82363549923gnyh49c9djl239pjm01223'
>>> d["id"]
17893

answered Apr 25, 2022 at 8:28

Gino Mempin

30.5k31 gold badges125 silver badges174 bronze badges

Comments

davidosomething · Accepted Answer · 2009-12-19 00:27:33Z

2

You could use a string parser to fix it first, a regex could do it provided that this is as complicated as the JSON will get.

answered Dec 19, 2009 at 0:27

davidosomething

3,4372 gold badges28 silver badges33 bronze badges

1 Comment

Serge Tarkovski Over a year ago

This is possible, but I am considering such type of solution as weird, so for now I am just looking for a json parsing library that could process this broken JSON.

PaulMcG · Accepted Answer · 2020-01-31 14:55:30Z

0

Pyparsing includes a JSON parser example, here is the online source. You could modify the definition of memberDef to allow a non-quoted string for the member name, and then you could use this to parser your not-quite-JSON source text.

[The August, 2008 issue of Python Magazine has a lot more detailed info about this parser. It shows some sample JSON, and code that accesses the parsed results like it was a deserialized object.

edited Jan 31, 2020 at 14:55

answered Dec 19, 2009 at 23:50

PaulMcG

64.1k16 gold badges98 silver badges135 bronze badges

2 Comments

mcoolive Over a year ago

Links are dead.

PaulMcG Over a year ago

Thanks - I fixed the link to the parser to now point to that file in the GitHub repo. I had to drop the Python Magazine link, since there is no longer a public archive of the issues of this magazine.

Collectives™ on Stack Overflow

How to parse somewhat wrong JSON with Python?

5 Answers 5

7 Comments

6 Comments

Comments

1 Comment

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

7 Comments

6 Comments

Comments

1 Comment

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related