Parse JSON from response with python requests

Question

I am trying to parse JSON data from a response:

If use my browswer to go to: /url/gold/stuff/here

I get a response in the browser as:

window.data_gold = [{'gold':' room 123' } , {'better gold':'room 1234'} , {"best gold": "in cellar"}];

How can I extract the JSON data out of the response

window.data_gold = json

My code:

import requests,json

url     = '/url/gold/stuff/here'
r       = requests.get(url,timeout = 30)
newJSON = r.text

The above returns all the text, .json does not work.

It's actually an important distinction @Leonardo.Z made; you have to convert the Javascript into valid JSON, which may or may not be a viable long-term option. — TML
– TML, Commented Apr 30, 2014 at 2:10

Hugh Bothwell · Accepted Answer · 2014-04-30 02:02:52Z

3

Try something like

data = json.loads(r.text[19:-1])

Edit: it doesn't like that, but this works:

import ast

data = ast.literal_eval(r.text[19:-1])

which gives

[{'gold': ' room 123'},
 {'better gold': 'room 1234'},
 {'best gold': 'in cellar'}]

answered Apr 30, 2014 at 2:02

Hugh Bothwell

57k9 gold badges91 silver badges103 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Merlin Over a year ago

good to see you again. The '19' should be all characters up to the square backet?

Hugh Bothwell Over a year ago

Yup, and :-1 to trim the ";" off the end.

dano · Accepted Answer · 2014-04-30 02:14:53Z

3

Here's a hideous looking one-liner that will pull the json out of that particular response and put it in a dict.

d = json.loads(r.text.split("=")[1].replace("'", '"')[:-1])

It's pulling the json itself out of the javascript statement, replacing the single-quotes with double-quotes (because the json module won't accept single-quotes), and then feeding it the json module.

Edit As pointed out by Hugh Bothwell, using ast.literal_eval instead of the json module avoids the single-quote issue, so you're left with

d = ast.literal_eval(r.text.split("=")[1].strip(" ;"))

The .strip(" ;") Will strip the ';' off the end and the whitespace from the beginning.

edited Apr 30, 2014 at 2:14

answered Apr 30, 2014 at 2:05

dano

95.5k20 gold badges234 silver badges231 bronze badges

1 Comment

TML Over a year ago

(because the json module won't accept single-quotes) That's because JSON won't accept single-quotes as delimiters :)

Collectives™ on Stack Overflow

Parse JSON from response with python requests

2 Answers 2

2 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related