Store numpy.array in cells of a Pandas.DataFrame

Question

I have a dataframe in which I would like to store 'raw' numpy.array:

df['COL_ARRAY'] = df.apply(lambda r: np.array(do_something_with_r), axis=1)

but it seems that pandas tries to 'unpack' the numpy.array.

Is there a workaround? Other than using a wrapper (see edit below)?

I tried reduce=False with no success.

EDIT

This works, but I have to use the 'dummy' Data class to wrap around the array, which is unsatisfactory and not very elegant.

class Data:
    def __init__(self, v):
        self.v = v

meas = pd.read_excel(DATA_FILE)
meas['DATA'] = meas.apply(
    lambda r: Data(np.array(pd.read_csv(r['filename'])))),
    axis=1
)

Engineero · Accepted Answer · 2018-03-15 21:30:47Z

74

Use a wrapper around the numpy array i.e. pass the numpy array as list

a = np.array([5, 6, 7, 8])
df = pd.DataFrame({"a": [a]})

Output:

             a
0  [5, 6, 7, 8]

Or you can use apply(np.array) by creating the tuples i.e. if you have a dataframe

df = pd.DataFrame({'id': [1, 2, 3, 4],
                   'a': ['on', 'on', 'off', 'off'],
                   'b': ['on', 'off', 'on', 'off']})

df['new'] = df.apply(lambda r: tuple(r), axis=1).apply(np.array)

Output :

     a    b  id            new
0   on   on   1    [on, on, 1]
1   on  off   2   [on, off, 2]
2  off   on   3   [off, on, 3]
3  off  off   4  [off, off, 4]

df['new'][0]

Output :

array(['on', 'on', '1'], dtype='<U2')

edited Mar 15, 2018 at 21:30

Engineero

13k5 gold badges56 silver badges79 bronze badges

answered Aug 7, 2017 at 13:46

Bharath M Shetty

30.6k6 gold badges65 silver badges111 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Cedric H. Over a year ago

That works, but then I'd rather use a dummy class instead of a list.

Cedric H. Over a year ago

Does that work if instead of tuple(r) you do something like np.array([[1,2],[3,4]]), ie. a 2-dim array?

Bharath M Shetty Over a year ago

tuple(r) works with 2D array too. Did you mean replacing tuple(r) with 2D np.array?

Cedric H. Over a year ago

Yes. I mean, I understand your solution, and it works, but what if I want to have a 2D np.array in the new column (and not a 1D array as shown)?

Bharath M Shetty Over a year ago

Can you add the expected output in your question? All the elements in the row should be of numpy array if you want to create a new 2D array. My solution works in that case. If its mixed type you have to use if else first to make it a numpy array.

David Wasserman · Accepted Answer · 2020-02-05 19:10:54Z

33

If you first set a column to have type object, you can insert an array without any wrapping:

df = pd.DataFrame(columns=[1])
df[1] = df[1].astype(object)
df.loc[1, 1] = np.array([5, 6, 7, 8])
df

Output:

    1
1   [5, 6, 7, 8]

answered Feb 5, 2020 at 19:10

David Wasserman

6337 silver badges17 bronze badges

1 Comment

Will Drakus Sep 15 at 15:02

![Lens Studio](images.ctfassets.net/ub38vssza5h3/7FynBv68WKAHYYAe7XQhlk/… =100x)

user1717828 · Accepted Answer · 2017-08-07 19:25:34Z

7

You can wrap the Data Frame data args in square brackets to maintain the np.array in each cell:

one_d_array = np.array([1,2,3])
two_d_array = one_d_array*one_d_array[:,np.newaxis]
two_d_array

array([[1, 2, 3],
       [2, 4, 6],
       [3, 6, 9]])


pd.DataFrame([
    [one_d_array],
    [two_d_array] ])

                                   0
0                          [1, 2, 3]
1  [[1, 2, 3], [2, 4, 6], [3, 6, 9]]

answered Aug 7, 2017 at 19:25

user1717828

7,2718 gold badges41 silver badges61 bronze badges

3 Comments

WestCoastProjects Over a year ago

Those become lists no?

user1717828 Over a year ago

@javadba I set the above output to df and see print(type(df.iloc[0,0])) --> <class 'numpy.ndarray'> This is Pandas 0.23.0. Are you seeing something different with another version?

WestCoastProjects Over a year ago

You are correct actually - one of the more highly upvoted answers made it sound like the ndarray would be converted to a list. I just tested this. without the brackets we get rows in the df and with brackets we get ndarrays as you say

Begoodpy · Accepted Answer · 2023-06-08 07:58:41Z

2

Preset the type of your column to object, this will allow you to store a NumPy array as-is:

df['COL_ARRAY'] = pd.Series(dtype='object')

df['COL_ARRAY'] = df.apply(lambda r: np.array(do_something_with_r), axis=1)

answered Jun 8, 2023 at 7:58

Begoodpy

1,0943 gold badges8 silver badges22 bronze badges

Comments

Kenneth · Accepted Answer · 2021-09-23 06:06:08Z

1

choose eval buildin function is easy to use and easy to read.

# First ensure use object store str
df['col2'] = self.df['col2'].astype(object)
# read
arr_obj = eval(df.at[df[df.col_1=='xyz'].index[0], 'col2']))
# write
df.at[df[df.col_1=='xyz'].index[0], 'col2'] = str(arr_obj)

real store display perfect human readable value:

col_1,  col_2
xyz,    "['aaa', 'bbb', 'ccc', 'ddd']"

answered Sep 23, 2021 at 6:06

Kenneth

313 bronze badges

1 Comment

vvolhejn Over a year ago

Using eval() is usually not a good idea. Transforming freely between strings and code is insecure and confuses the IDE/linter/type-checker/other people reading your code. Consider using the other methods mentioned in this post.

President James K. Polk · Accepted Answer · 2024-09-18 11:32:33Z

1

Suppose you have a DataFrame ds and it has a column named 'class'. If ds['class'] contains strings or numbers, and you want to change them with numpy.ndarrays or lists, the following code would help. In the code, class2vector is a numpy.ndarray or list and ds_class is a filter condition.

ds['class'] = ds['class'].map(lambda x: class2vector if (isinstance(x, str) and (x == ds_class)) else x)

edited Sep 18, 2024 at 11:32

President James K. Polk

42.2k34 gold badges113 silver badges149 bronze badges

answered Jun 13, 2019 at 2:03

yuzhen_3301

193 bronze badges

Comments

allenyllee · Accepted Answer · 2020-01-18 09:02:14Z

0

Just wrap what you want to store in a cell to a list object through first apply, and extract it by index 0of that list through second apply:

import pandas as pd
import numpy as np

df = pd.DataFrame({'id': [1, 2, 3, 4],
                   'a': ['on', 'on', 'off', 'off'],
                   'b': ['on', 'off', 'on', 'off']})


df['new'] = df.apply(lambda x: [np.array(x)], axis=1).apply(lambda x: x[0])

df

output:

    id  a       b       new
0   1   on      on      [1, on, on]
1   2   on      off     [2, on, off]
2   3   off     on      [3, off, on]
3   4   off     off     [4, off, off]

answered Jan 18, 2020 at 9:02

allenyllee

1,1391 gold badge15 silver badges16 bronze badges

Comments

Marcelo · Accepted Answer · 2022-10-08 17:24:52Z

0

Here goes my 2 cents contribution (tested on Python 3.7):

import pandas as pd
import numpy as np

dataArray = np.array([0.0, 1.0, 2.0])
df = pd.DataFrame()
df['User Col A'] = [1]
df['Array'] = [dataArray]

answered Oct 8, 2022 at 17:24

Marcelo

652 silver badges10 bronze badges

Comments

Jérémy Talbot-Pâquet · Accepted Answer · 2023-02-06 05:32:42Z

0

If you only want some of the columns you could do something lie this. Taking the example of @allenyllee,

df = pd.DataFrame({'id': [1, 2, 3, 4],
                   'a': ['on', 'on', 'off', 'off'],
                   'b': ['on', 'off', 'on', 'off']})

df['new'] = df[['a','b']].apply(lambda x: np.array(x), axis=1)

which outputs

   id    a    b         new
0   1   on   on    [on, on]
1   2   on  off   [on, off]
2   3  off   on   [off, on]
3   4  off  off  [off, off]

you can also change the order of [['a', 'b']] if you need a specific order.

answered Feb 6, 2023 at 5:32

Jérémy Talbot-Pâquet

3713 silver badges14 bronze badges

Collectives™ on Stack Overflow

Store numpy.array in cells of a Pandas.DataFrame

9 Answers 9

5 Comments

1 Comment

3 Comments

Comments

1 Comment

Comments

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

9 Answers 9

5 Comments

1 Comment

3 Comments

Comments

1 Comment

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related