Extracting element of pandas array of type numpy array

Question

I have a pandas dataframe df whose elements are each a whole numpy array. For example the 6th row of column 'x_grid':

>>> e = df.loc[6,'x_grid']
>>> print(e)

[-11.52616579 -11.48006112 -11.43395646 -11.3878518  -11.34174713
 -11.29564247 -11.24953781 -11.20343315 -11.15732848 -11.11122382
 -11.06511916 -11.01901449 ...

But I cannot use this as a numpy array as it is just given as a string:

>>> print(type(e))

<class 'str'>

How can I store a numpy array to a dataframe so it does not get converted to a string? Or convert this string back to a numpy array in a nice way?

It's worth noting that this DataFrame is loaded from a csv file, which is no doubt where the conversion to string happens. So I guess converting this string back to a numpy array would be the easier route. — Jack Wetherell
– Jack Wetherell, Commented Apr 4, 2019 at 15:50
Plus, there are no commas to seperate the elements in your array. — Erfan
– Erfan, Commented Apr 4, 2019 at 15:52
Look at the source text file. This array is a quoted string, complete with[], Are there also ...? The original dataframe had these array items, and the only way to save such a df to a 2d csv format is turn the complex items into strings. pandas used str(item). Where possible avoid saving such dataframes as csv. — hpaulj
– hpaulj, Commented Apr 4, 2019 at 15:57
This has come up a number times, e.g. stackoverflow.com/questions/51898099/…. literal_eval might have problems with your string because it is missing the commas that normally mark a list. — hpaulj
– hpaulj, Commented Apr 4, 2019 at 16:42

Rafal Janik · Accepted Answer · 2019-04-04 19:50:55Z

0

If you just want to convert all those strings in each row into a list the following will work:

df['x_grid'].str[1:-1].str.split(" ").apply(lambda x: (list(map(float, x))))

# or for a numpy array
df['x_grid'].str[1:-1].str.split(" ").apply(lambda x: (np.array(list(map(float, x)))))

Hope that helps.

edited Apr 4, 2019 at 19:50

answered Apr 4, 2019 at 19:42

Rafal Janik

3091 silver badge6 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Jack Wetherell · Accepted Answer · 2019-04-05 13:01:14Z

0

Thanks to Erfan and hpaulj for the suggestions that combined to answer this question.

The solution is that when setting an element of the dataframe I first convert the numpy array x to a list (so it is comma separated not space separated):

df = df.append({'x_grid': list(x)}, ignore_index=True)

Then after saving to a csv, and loading back in, I extract it back into a numpy array using np.array() and ast.literal_eval() (Note: requires import ast):

x = np.array(ast.literal_eval(df.loc[entry,'x_grid']))

This then returns a correct numpy array x.

answered Apr 5, 2019 at 13:01

Jack Wetherell

5853 gold badges8 silver badges17 bronze badges

Comments

El-Shamouty · Accepted Answer · 2020-04-17 09:55:33Z

0

Want to extend Rafal's answer to avoid numpy throwing exception from empty strings resulting from the x.split:

df['x_grid'].str[1:-1].apply(lambda x: list(filter(None,x.split(' ')))).apply(lambda x: np.array(x).astype(np.float))

answered Apr 17, 2020 at 9:55

El-Shamouty

112 bronze badges

Collectives™ on Stack Overflow

Extracting element of pandas array of type numpy array

3 Answers 3

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related