Pandas convert columns type from list to np.array

Question

I'm trying to apply a function to a pandas dataframe, such a function required two np.array as input and it fit them using a well defined model.

The point is that I'm not able to apply this function starting from the selected columns since their "rows" contain list read from a JSON file and not np.array.

Now, I've tried different solutions:

#Here is where I discover the problem

train_df['result'] = train_df.apply(my_function(train_df['col1'],train_df['col2']))

#so I've tried to cast the Series before passing them to the function in both these ways:

X_col1_casted = trai_df['col1'].dtype(np.array)
X_col2_casted = trai_df['col2'].dtype(np.array)

doesn't work.

X_col1_casted = trai_df['col1'].astype(np.array)
X_col2_casted = trai_df['col2'].astype(np.array)

doesn't work.

X_col1_casted = trai_df['col1'].dtype(np.array)
X_col2_casted = trai_df['col2'].dtype(np.array)

does'nt work.

What I'm thinking to do now is a long procedure like:

starting from the uncasted column-series, convert them into list(), iterate on them apply the function to the np.array() single elements, and append the results into a temporary list. Once done I will convert this list into a new column. ( clearly, I don't know if it will work )

Does anyone of you know how to help me ?

EDIT: I add one example to be clear:

The function assume to have as input two np.arrays. Now it has two lists since they are retrieved form a json file. The situation is this one:

col1        col2    result
[1,2,3]     [4,5,6]  [5,7,9]
[0,0,0]     [1,2,3]  [1,2,3]

Clearly the function is not the sum one, but a own function. For a moment assume that this sum can work only starting from arrays and not form lists, what should I do ?

Thanks in advance

may you also tell me how ? I need to use it to single cell elements, not to the whole columns in one shot. I need one array per row. — LNRD.CLL
– LNRD.CLL, Commented Sep 21, 2016 at 14:00
what do you mean one array per row? I understood from the question that you want to convert a whole column to a numpy array. — AlvaroP
– AlvaroP, Commented Sep 21, 2016 at 14:03
I've edited the question with an example. The functoin that work per row, assume to have as input np.array and not lists. That's the point. Hoping to be clear now. — LNRD.CLL
– LNRD.CLL, Commented Sep 21, 2016 at 14:10
I actually have the opposite requirement, My pandas dataframe have numpy.ndarray that I want to convert to list so that It cant be stored into DynamoDB table. Does anyone have any inputs on how can I do that — NNM
– NNM, Commented Nov 17, 2022 at 18:37

Nickil Maveli · Accepted Answer · 2016-09-21 14:21:17Z

29

Use apply to convert each element to it's equivalent array:

df['col1'] = df['col1'].apply(lambda x: np.array(x))

type(df['col1'].iloc[0])
numpy.ndarray

Data:

df = pd.DataFrame({'col1': [[1,2,3],[0,0,0]]})
df

answered Sep 21, 2016 at 14:21

Nickil Maveli

29.8k10 gold badges86 silver badges88 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

tenpercent Over a year ago

df['col1'] = df['col1'].apply(np.array) works as well

profPlum Over a year ago

I came here because I wanted to get a one big np.array from a column of type list (i.e. no pandas types at all). For those who want that you can do this: np.stack(df['col1']) (e.g. necessary for keras)

s510 · Accepted Answer · 2023-10-26 15:08:27Z

0

You can apply pd.Series on top of the list. e.g.,

>>> X_train = df.col1.apply(pd.Series).to_numpy()

>>> type(X_train)
numpy.ndarray

answered Oct 26, 2023 at 15:08

s510

2,88219 silver badges25 bronze badges

Collectives™ on Stack Overflow

Pandas convert columns type from list to np.array

2 Answers 2

2 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related