Selecting rows in numpy array

Question

I have a numpy array (mat) of shape (n,4). The array has four columns and large number (n) of rows. The first three columns represent x, y, z columns in my calculation. I wish to select those rows of the numpy array where the x column has values below a given number (min_x) or values above a given number (max_x), and where the y column has values below a given number (min_y) or values above a given number (max_y) and where the z column has values below a given number (min_z) or values above a given number (max_z).

This is how I am trying to implement this desired functionality presently:

import numpy as np

mark = np.where( ( (mat[:,0]<=min_x) | \
            (mat[:,0]>max_x) ) & \
                 ( (mat[:,1]<=min_y) | \
            (mat[:,1]>max_y) ) & \
                 ( (mat[:,2]<=min_z) | \
            (mat[:,2]>max_z) ) )

mat_new = mat[:,mark[0]]

Is the technique that I am using correct, and the best way to achieve the desired functionality? I will greatly appreciate any help. Thanks.

Did you try it on a sample? Did you get unsatisfactory results? — Mad Physicist
– Mad Physicist, Commented Sep 15, 2018 at 3:22
@MadPhysicist: I tried it on my sample. It gives expected (perfect) results. =) — Ling Guo
– Ling Guo, Commented Sep 15, 2018 at 4:08
Then I'd say it's a good way to do it :) The accepted answer is a bit more clever though. — Mad Physicist
– Mad Physicist, Commented Sep 15, 2018 at 4:12

Brad Solomon · Accepted Answer · 2018-09-15 14:08:41Z

3

What you have now looks fine. But since you are asking about other ways to achieve the desired functionality: you can create a 1-dimensional boolean mask that is either True or False for each row index. Here is an example.

>>> import numpy as np
>>> np.random.seed(444)

>>> shape = 15, 4
>>> mat = np.random.randint(low=0, high=10, size=shape)
>>> mat
array([[3, 0, 7, 8],
       [3, 4, 7, 6],
       [8, 9, 2, 2],
       [2, 0, 3, 8],
       [0, 6, 6, 0],
       [3, 0, 6, 7],
       [9, 3, 8, 7],
       [3, 2, 6, 9],
       [2, 9, 8, 9],
       [3, 2, 2, 8],
       [1, 5, 6, 7],
       [6, 0, 0, 0],
       [0, 4, 8, 1],
       [9, 8, 5, 8],
       [9, 4, 6, 6]])

# The thresholds for x, y, z, respectively
>>> lower = np.array([5, 5, 4])
>>> upper = np.array([6, 6, 7])
>>> idx = len(lower)
# Parentheses are required here.  NumPy boolean ops use | and &
# which have different operator precedence than `or` and `and`
>>> mask = np.all((mat[:, :idx] < lower) | (mat[:, :idx] > upper), axis=1)

>>> mask
array([False, False,  True,  True, False, False,  True, False,  True,
        True, False, False,  True, False, False])

Now indexing mat by mask will constrain it to row indices where mask is True:

>>> mat[mask]
array([[8, 9, 2, 2],
       [2, 0, 3, 8],
       [9, 3, 8, 7],
       [2, 9, 8, 9],
       [3, 2, 2, 8],
       [0, 4, 8, 1]])

What is a bit different about this approach is that it is scalable: instead of specifying each coordinate condition individually, you can specify them in two arrays, one for the upper threshold and one for the lower, and then take advantage of NumPy's vectorization & broadcasting to build the mask.

np.all() says, test that all values are True, row-wise. It captures the "and" conditions from your question, while the | operator captures the "or".

edited Sep 15, 2018 at 14:08

answered Sep 15, 2018 at 2:53

Brad Solomon

41.2k39 gold badges167 silver badges260 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Ling Guo Over a year ago

Thanks Brad, the answer looks awesome (I was a little confused by the output before your edit, but the answer makes so much more sense after the edit.) I should be able to use <= instead of < in your answer (mat[:, :idx] < lower)) , is that right?

Brad Solomon Over a year ago

Yes, that would work too if you want less-than-or-equal

Mad Physicist Over a year ago

Very nice use of broadcasting. I hope you get selected.

DYZ · Accepted Answer · 2018-09-15 03:04:30Z

3

Looks good to me. You can make it a bit more compact by comparing the columns to the midrange values:

mark = (np.abs(mat[:,0] - (max_x + min_x) / 2) > (max_x - min_x) / 2) &
       (np.abs(mat[:,1] - (max_y + min_y) / 2) > (max_y - min_y) / 2) &
       (np.abs(mat[:,2] - (max_z + min_z) / 2) > (max_z - min_z) / 2)

Unfortunately, you cannot control the precise boundary conditions (< vs <=) anymore. Also, this is probably the slowest solution, even slower than the original one.

edited Sep 15, 2018 at 3:04

answered Sep 15, 2018 at 2:47

DYZ

57.3k10 gold badges73 silver badges101 bronze badges

Comments

kevinkayaks · Accepted Answer · 2018-09-15 04:04:06Z

2

I'd just drop the np.where and use the boolean mask instead

x,y,z,_ = mat.T
mask = ( ( (x <= min_x) | (x > max_x) ) &
         ( (y <= min_y) | (y > max_y) ) &
         ( (z <= min_z) | (z > max_z) ) ) 
mat_new = mat[mask]

edited Sep 15, 2018 at 4:04

answered Sep 15, 2018 at 2:48

kevinkayaks

2,7461 gold badge16 silver badges30 bronze badges

Collectives™ on Stack Overflow

Selecting rows in numpy array

3 Answers 3

3 Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

3 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related