Faster way to modify a numpy array

Question

I have a large numpy array with 4 million rows and 4 columns (shape = (4000000,4))

I need to modify/ decrease the number of rows, based on the value in fourth column. For example few of my rows in my data set look like the following:

a = np.array([[1.32, 24.42, 224.21312, 0],[1.32, 24.42, 224.21312, 0],[1.32, 24.42, 224.21312, 1],[1.32, 24.42, 224.21312, 1],[1.32, 24.42, 224.21312, 0]]);

My result should be the following (only rows with last column value = 1)

b = [1.32, 24.42, 224.21312, 1],[1.32, 24.42, 224.21312, 1]

A for loop to go through each row is taking a long time to process.

I have 200 of these arrays, so I am already using multiprocessing for each array.

Looking for suggestions.

Vince W. · Accepted Answer · 2019-04-04 17:22:03Z

3

does this work for you?

a[a[:,3] == 1]

gives:

array([[  1.32   ,  24.42   , 224.21312,   1.     ],
       [  1.32   ,  24.42   , 224.21312,   1.     ]])

answered Apr 4, 2019 at 17:22

Vince W.

3,8493 gold badges39 silver badges65 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Tigran Fahradyan Over a year ago

a[a[:, -1] == 1] is slightly better for n length arrays.

Rarblack · Accepted Answer · 2019-04-04 17:37:03Z

0

You can convert it to dataframe and operate your operations there and then convert back to array:

df = pd.DataFrame(a)
df = df[df[3] == 1]
a = df.as_matrix()

Output:

array([[  1.32   ,  24.42   , 224.21312,   1.     ],
       [  1.32   ,  24.42   , 224.21312,   1.     ]])

answered Apr 4, 2019 at 17:37

Rarblack

4,6744 gold badges24 silver badges36 bronze badges

1 Comment

roganjosh Over a year ago

You haven't really converted anything because it's still a numpy array underneath, this just wraps it in the overhead of pandas. You can operate on the array directly as shown in the other answer.

Collectives™ on Stack Overflow

Faster way to modify a numpy array

2 Answers 2

1 Comment

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related