Python pandas json 2D array

Question

relatively new to pandas, I have a json and python files:

{"dataset":{
    "id": 123,
    "data": [["2015-10-16",1,2,3,4,5,6],
             ["2015-10-15",7,8,9,10,11,12],
             ["2015-10-14",13,14,15,16,17]]
}}

&

import pandas
x = pandas.read_json('sample.json')
y = x.dataset.data
print x.dataset

Printing x.dataset and y works fine, but when I go to access a sub-element y, it returns a 'buffer' type. What's going on? How can I access the data inside the array? Attempting y[0][1] it returns out of bounds error, and iterating through returns a strange series of 'nul' characters and yet, it appears to be able to return the first portion of the data after printing x.dataset...

jakevdp · Accepted Answer · 2015-10-22 13:30:03Z

2

The data attribute of a pandas Series points to the memory buffer of all the data contained in that series:

>>> df = pandas.read_json('sample.json')
>>> type(df.dataset)
pandas.core.series.Series
>>> type(df.dataset.data)
memoryview

If you have a column/row named "data", you have to access it by it's string name, e.g.:

>>> type(df.dataset['data'])
list

Because of surprises like this, it's usually considered best practice to access columns through indexing rather than through attribute access. If you do this, you will get your desired result:

>>> df['dataset']['data']
[['2015-10-16', 1, 2, 3, 4, 5, 6],
 ['2015-10-15', 7, 8, 9, 10, 11, 12],
 ['2015-10-14', 13, 14, 15, 16, 17]]

>>> arr = df['dataset']['data']
>>> arr[0][0]
'2015-10-16'

answered Oct 22, 2015 at 13:30

jakevdp

88.5k15 gold badges140 silver badges177 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Onitz Over a year ago

Fantastic! Thanks so much. Ironically, I had just been using indexed keys on a standard json import a little earlier, but hadn't thought to apply it to the pandas read :)

Collectives™ on Stack Overflow

Python pandas json 2D array

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related