Converting a pandas Series of string ints into a numpy matrix - efficiently

Question

I have a pandas series, each cell containts a string of ints (same number of ints in each string, seperated by a space), e.g.

s = pd.Series(['1 17 -3 ... n'],['100 -27 344 ... n'],...,['0 14 2 ... n'])

I know how to convert a single cell into a numpy vector:

arr = np.asarray(s[0].split(' '), dtype = np.float32)

I also know how to convert the whole series into a matrix, which is my finel goal:

X = np.zeros((len(s), number_of_ints_in_string))
for i, cell in enumerate(s):
    X[i,:] = np.asarray(cell.split(' '), dtype = np.float32)

the problem is that for a very long series (which is my case), this is slow. Is there any faster way to do this?

One way or other you'll have to use string.split on each string. In pandas the elements of that Series are Python strings. Even the np.char.split function uses Python string methods. So there isn't a faster whole-array method. — hpaulj
– hpaulj, Commented Jan 15, 2020 at 6:33

jezrael · Accepted Answer · 2020-01-15 06:16:15Z

Use Series.str.split with expand=True to DataFrame and then conver it to numpy array by DataFrame.to_numpy:

s = pd.Series(['1 17 -3','100 -27 344','0 14 2'])
print (s)
0        1 17 -3
1    100 -27 344
2         0 14 2
dtype: object

arr = s.str.split(expand=True).to_numpy().astype(np.float32)
print (arr)
[[  1.  17.  -3.]
 [100. -27. 344.]
 [  0.  14.   2.]]

Another idea is use list comprehension:

arr = np.asarray([x.split() for x in s], dtype = np.float32)
print (arr)
[[  1.  17.  -3.]
 [100. -27. 344.]
 [  0.  14.   2.]]

Second solution is faster it 300k rows:

s = pd.Series(['1 17 -3','100 -27 344','0 14 2'])
s = pd.concat([s] * 100000, ignore_index=True)


In [52]: %timeit np.asarray([x.split() for x in s], dtype = np.float32)
521 ms ± 10.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [53]: %timeit s.str.split(expand=True).to_numpy().astype(np.float32)
939 ms ± 6.08 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Collectives™ on Stack Overflow

Converting a pandas Series of string ints into a numpy matrix - efficiently

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related