Parsing heavily nested JSON in python

Question

I'm connecting to and extracting a JSON via API, but getting stuck parsing beyond a certain point.

I've been able to parse to the sample level included below. I'd like to extract a list of 'symbols' from the resulting JSON list, but only appear able to do this for individual entities. I'd like to do it directly if at all possible (ie, avoid just using a loop).

Namely, this works:

data[0]['acct']['positions'][1]['i']['symbol']

Returning this:

Out[100]: 'JNC'

But this and this don't:

data[0]['acct']['positions'][:]['i']['symbol']

data[0]['acct']['positions'][0:3]['i']['symbol']

Both returning the following error:

Traceback (most recent call last):

  File "<ipython-input-105-72bbee303a08>", line 1, in <module>
    data[0]['securitiesAccount']['positions'][:]['instrument']['symbol']

TypeError: list indices must be integers or slices, not str

Sample JSON below:

[{'a': 0.0,
  'b': 1.0,      
  'i': {'desc': 'CASH_EQUIVALENT', 'id': '9ZZZFD104', 'symbol': 'MMDA1'},
  'c': 72.64},
 {'a': 0.0,
  'b': 33.61716,      
  'i': {'desc': 'EQUITY', 'id': '78464A417', 'symbol': 'JNC'},
  'c': 39.59},
 {'a': 0.0,
  'b': 87.81473,      
  'i': {'desc': 'EQUITY', 'id': '921937793', 'symbol': 'BVV'},
  'c': 19.34}]

Because in with [1] you are accessing an element in the list. Where [:] is the entire list itself and [0:3] is a subset of the list. Which need to be accessed with integer indexes. — Spencer Wieczorek
– Spencer Wieczorek, Commented Jan 29, 2019 at 4:03
Your examples that "don't" do not seem to apply to the sample JSON you provided, since there's no acct, positions etc. in there. Can you update the example data and example code so that they refer to the same and check whether you still have the issue? — Grismar
– Grismar, Commented Jan 29, 2019 at 4:03

iz_ · Accepted Answer · 2019-01-29 05:16:06Z

3

That kind of indexing won't work like it does in NumPy. You can use a list comprehension instead:

result = [record['i']['symbol'] for record in data[0]['acct']['positions'][0:3]]

Obviously, change [0:3] to whatever. If you want all of them, just omit the slice completely.

This is what you tried to do originally:

Say you have a list of dictionaries that looks like this:

foo = [{'bar': 1}, {'bar': 2}, {'bar': 3}]

If you wanted to access [1, 2, 3] with your method, it would be something like foo[:]['bar']. However, this expression is evaluated left to right, and foo[:] would just create a slice of foo, doing essentially nothing. Then you try to get the key 'bar' from foo, a list, which will raise an error.

edited Jan 29, 2019 at 5:16

answered Jan 29, 2019 at 4:11

iz_

16.7k4 gold badges29 silver badges43 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Chris Over a year ago

this is great, thanks...offhand, is there a good reason why the slice the way I originally doesn't work in this circumstance like standard indexing?

Chris Over a year ago

and is there then a concise way to retrieve the data corresponding to the various symbols (eg, data[0][...]['symbol'] = 'BVV'?

iz_ Over a year ago

@Chris I made an edit with a small explanation. Unfortunately, I do not believe there is a way to retrieve data like that. A list comprehension or similar is required.

Chris Over a year ago

got it...so there's no way to retrieve the data associated with a given symbol aside from a list comprehension or loop

Kim · Accepted Answer · 2019-01-29 04:23:02Z

You can use a list comprehension to return a filtered list.

Instead of

data[0]['acct']['positions'][:]['i']['symbol']

You could use:

[ i['i']['symbol'] for i in data[0]['acct'] ]

This loops though data[0]['act'], and constructs a list of ['i']['symbol'] for each of these.

For a sub part of a list you can use:

[ i['i']['symbol'] for i in data[0]['acct'][0:3] ]

Modifying this to work with your sample json:

data = [
 {'a': 0.0,
  'b': 1.0,      
  'i': {'desc': 'CASH_EQUIVALENT', 'id': '9ZZZFD104', 'symbol': 'MMDA1'},
  'c': 72.64},
 {'a': 0.0,
  'b': 33.61716,      
  'i': {'desc': 'EQUITY', 'id': '78464A417', 'symbol': 'JNC'},
  'c': 39.59},
 {'a': 0.0,
  'b': 87.81473,      
  'i': {'desc': 'EQUITY', 'id': '921937793', 'symbol': 'BVV'},
  'c': 19.34}
 ]
print([ i['i']['symbol'] for i in data[0] ])

Prints:

['MMDA1', 'JNC', 'BVV']

Collectives™ on Stack Overflow

Parsing heavily nested JSON in python

2 Answers 2

4 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related