Issue when filtering rows of numpy array by multiple conditions

Question

I want to filter a numpy array by multiple conditions. I found this thread and tested the slicing method on my dataset but I get unexpected results. Well, at least for me they're unexpected, as I might simply have a problem understanding the functionality of bitwise operators or something else :/

Just so you get an idea of the data:

test.shape
>>(222988, 2)

stats.describe(all_output[:, 0])
>>DescribeResult(nobs=222988, minmax=(2.594e-05, 74.821), mean=11.106, variance=108.246, [...])

stats.describe(all_output[:, 1])
>>DescribeResult(nobs=222988, minmax=(0.001, 8.999), mean=3.484, variance=7.606, [...])

Now, doing some basic filtering:

test1 = test[(test[:, 0] >= 30) & (test[:, 1] <= 2)] 

test1.shape
>>(337, 2)

Those are actually the rows I don't want to have in my dataset, so if I do what I believe is the opposite...

test2 = test[(test[:, 0] <= 30) & (test[:, 1] >= 2)] 

test2.shape
>>(112349, 2)

I would expect the result to be (222651, 2). I guess that I'm doing some embarrassingly simple thing wrong? Can anyone here push me in the right direction?

Thanks already! -M

The opposite of A and B is notA or notB and the opposite of >= is <. I.e.: test2 = test[(test[:, 0] < 30) | (test[:, 1] > 2)] should do the Job. — SpghttCd
– SpghttCd, Commented Jun 13, 2018 at 12:54

FHTMitchell · Accepted Answer · 2018-06-13 13:03:36Z

0

De morgans law: not (p and q) == (not p) *or* (not q). Anyway, the not operator in numpy is ~ so

 ~((test[:, 0] >= 30) & (test[:, 1] <= 2)) == ((test[:, 0] < 30) | (test[:, 1] > 2))

Either will do what you want, e.g.

test1 = test[~((test[:, 0] >= 30) & (test[:, 1] <= 2))]

edited Jun 13, 2018 at 13:03

answered Jun 13, 2018 at 12:55

FHTMitchell

12.2k2 gold badges40 silver badges50 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

maawoo Over a year ago

Thank you! Now I feel really stupid, because I even used ~ in my code before. But thanks for giving me some context about De morgans law!

Collectives™ on Stack Overflow

Issue when filtering rows of numpy array by multiple conditions

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related