Efficiently splitting a column of numpy array

Question

I have a csv file with 5 columns, in which the second column is time represented in format 10/22/2001 14:00. I want to create another file with this time data split into separate columns. To split the column I used the below code in python

from numpy import loadtxt
import numpy as np
from time import strptime

filename = 'data/file.csv'
data = loadtxt(filename, delimiter=',', dtype=str, skiprows=1)
newdata = np.zeros((data.shape[0],7))
newdata[:,0] = data[:,0]

for i in range(len(data[:,1])):
    tm =  strptime(data[i,1], "%m/%d/%Y %H:%M")
    newdata[i,1] = tm.tm_year
    newdata[i,2] = tm.tm_wday
    newdata[i,3] = tm.tm_hour

newdata[:,4:] =  data[:,2:]

Is there a better way of doing this using numpy methods or other modules of python?

Falko · Accepted Answer · 2014-10-28 06:10:41Z

1

You can shorten the generation of newdata using the following three lines:

Converts the datetime strings into datetime objects:
```
datetimes = [datetime.strptime(d, "%m/%d/%Y %H:%M") for d in data[:, 1]]
```
I assume you use from datetime import datetime.

Collect year, weekday and hour of each datetime object.

yearWeekdayHour = [[dt.year, dt.weekday(), dt.hour] for dt in datetimes]

Horizontally stack all parts together: The first column of the original data, the date and time information as well as the last columns of data.
```
newdata = np.hstack((data[:, 0, None], yearWeekdayHour, data[:, 2:]))
```
Note that the first column is indexed with an additional None to get a 2D array, which is required for horizontal stacking.

I'm not quite sure if the is a better solution. Yours might be more readable, especially for those not perfectly familiar with all those Python list comprehensions. But it might be an alternative worth reading and playing around with. It can be a quite powerful tool.

edited Oct 28, 2014 at 6:10

answered Oct 28, 2014 at 6:05

Falko

18.2k14 gold badges65 silver badges119 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Sanket Over a year ago

I feel your code is more compact than what I posted. I basically wanted to check if the for loop operation can be vectorized somehow.

Collectives™ on Stack Overflow

Efficiently splitting a column of numpy array

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related