Pandas Series to Numpy ndarray

Question

I have a dataframe similar to this (but much larger).

>>> df = pd.DataFrame([ [ 'a', np.array([ 1, 2]) ], [ 'b', np.array([ 3, 4 ]) ] ])
   0       1
0  a  [1, 2]
1  b  [3, 4]

The last column has the shape listed as...

>>> df[1].shape
(2,)

I'd like it to be listed as (2,2). I was able to do this via the following line, but the performance of tolist() is... bad.

>>> np.array(df[1].tolist()).shape
(2, 2)

It could also be a Pandas dataframe as long as it correctly reports the shape. Any other suggestions?

What are you trying to accomplish by having changing the shape to (2,2)? Will df[1] always be an array of 2 numbers? — vielkind
– vielkind, Commented Nov 8, 2018 at 14:38

piRSquared · Accepted Answer · 2018-11-08 14:39:00Z

7

This is not possible!

Pandas keeps each Series as a single dimensional ndarray. If you have multiple dimensions that you are trying to squeeze into it, Pandas will force this to be a single dimensional array with dtype of object.

If you simply want to get the contents and make it into a 2 dimensional array then I'd suggest

np.array(df[1].values.tolist())

Otherwise, I'd suggest you keep them in two different columns.

answered Nov 8, 2018 at 14:39

piRSquared

296k68 gold badges509 silver badges654 bronze badges

Sign up to request clarification or add additional context in comments.

10 Comments

Dan Grahn Over a year ago

It doesn't need to be a Pandas series. It could be a numpy array. I'm wondering if there's better way than converting it to a list first.

piRSquared Over a year ago

The difference between what I've done and what you did was that I accessed the values attribute and used Numpy's tolist rather than going through Pandas' tolist. But it's the best option you have. If you are worried about performance, then you should reconsider putting 2-element array like things in a single column in the first place.

Dan Grahn Over a year ago

It's actually (50, 25). So not quite the same as just a 2-element array. That was a toy problem.

piRSquared Over a year ago

Same thing. Keep in a larger dataframe with appropriate number of columns.

Dan Grahn Over a year ago

I do need to work with it in a (50, 25) format. Do you think reshaping is faster than the conversion?

|

Collectives™ on Stack Overflow

Pandas Series to Numpy ndarray

1 Answer 1

10 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

10 Comments

Your Answer

Sign up or log in

Post as a guest

Related