Find if numpy array is in list of numpy arrays -- .all() not working

Question

I am trying to find if a numpy array is in a list of numpy arrays and if so printing 'Yes'. My array and list of arrays looks as such:

many = [np.array([23, 34, 12]), np.array([23, 34, 23]), np.array([45, 23, 48])]
test = np.array([23, 34, 12])

However every attempy I have used .all() with as suggested doesn't seem to work. Here are my attempts:

if np.array([23, 34, 12]) in many:
    print('yes')

if np.all(np.array([23, 34, 12])) in many:
    print('yes')

if np.array([23, 34, 12]).all() in many:
    print('yes')

Every time I get the following error:

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

I don't understand the way in which it is asking me to use .all() here.

Any help would be appreciated

Grismar's answer works great too. In your application, I would suggest to checkout both posts' tests to see which one would be the fastest for your application. — Ehsan
– Ehsan, Commented May 13, 2020 at 8:25

Grismar · Accepted Answer · 2020-05-14 22:13:09Z

3

This is an annoying side effect of how numpy deals with the comparison operator ==.

Consider this:

>>> print(1 == np.array([1,2,3]))
[ True False False]

Clearly the array np.array([1,2,3]) is not 1, but what the expression computes is if all elements on the array are 1 and the result is an array with booleans.

This does what you need, keeping that in mind:

if any((test==a).all() for a in many):
    print('yes')

To explain why yours doesn't work:

if np.array([23, 34, 12]) in many:

This basically amounts to:

if any(x == np.array([23, 34, 12]) for x in many):

And you now know that the result would be like: any([False, False, False], [False, False, False], [False, False, False]) - and any cannot deal with the lists.

After reading the other answer, I wondered about execution times:

from timeit import timeit
import numpy as np

many = [np.array([23, 34, 12]), np.array([23, 34, 23]), np.array([45, 23, 48])]
test = np.array([23, 34, 23])


def one():
    return any((test == a).all() for a in many)


def two():
    for a in many:
        if np.array_equal(a, test):
            return True
    return False


def three():
    return (many == test).all(axis=1).any(axis=0)


def four():
    return test.tolist() in np.stack(many).tolist()


n = 1000000
print('one():', timeit(one, number=n), one())
print('two():', timeit(two, number=n), two())
print('three():', timeit(three, number=n), three())
print('four():', timeit(four, number=n), four())

Results:

one(): 3.0405494 True
two(): 5.1088635 True
three(): 5.7222043 True
four(): 4.961463499999999 True

So, for performance the solution provided in this answer is fastest. Of course, you may prefer an alternative for style.

Under perhaps somewhat more realistic workloads, three() may perform faster. For 10,000 arrays of 1,000 random elements searching for a random 1,000 array, one() still easily outperforms it, but as the number of searched arrays increases and the size of the searched arrays doesn't, three() inches it out.

For example, in my situation, when searching for a random 100 element array in a 1,000,000 random element arrays, three() is about twice as fast as one(). You'll need to decide which solution you prefer based on the type of search you're performing.

As with any "what is the fastest solution" question, the answer is always "it depends" and realistic testing is the way to find your answer.

edited May 14, 2020 at 22:13

answered May 13, 2020 at 1:53

Grismar

32.4k6 gold badges42 silver badges68 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Grismar Over a year ago

Also note that the actual solution is efficient, since any resolves to True as soon as a value from the generator that is passed to it resolves to True, so it doesn't test all arrays, it stops as soon as it finds one for which all elements match.

Ehsan Over a year ago

Thank you for the test. Please find another test for larger input size, that shows a different behavior (method 3 fastest). Testing on smaller size, does not capture all aspects of code speed. I have to admit my expectations were wrong though. Thank you for added knowledge.

Grismar Over a year ago

I assumed the data set would typically not be massive, from the example, but I've added some notes to that effect. You're right that for specific characteristics of the data set, three() is faster, though it's not just a matter of size, but relative size and composition - testing is king.

Ehsan · Accepted Answer · 2020-05-13 08:36:08Z

1

I am suggesting multiple ways besides the other posted answer (EDIT: please checkout the runtime test below):

Probably faster solution than the one suggested: Numpy has a built-in method to test arrays' equality:
```
for a in many:
  if np.array_equal(a, test):
    print('yes')
    break
```
And if your arrays in many are of the same shape, you can still find your answer without ANY loop (EDIT: seems to be the fastest answer by far):
```
if (many==test).all(axis=1).any(axis=0):
  print('yes')
```

You can stack them and search for lists:

if test.tolist() in np.stack(many).tolist():
  print('yes')

And of course in this case, another solution would be raveling many and using views to find the array.

UPDATE: Thanks to @Grismar for added runtime test. While his test is correct for the input given, a better practice is to test it for large inputs (of course the choice would depend on application size). Here are outputs:

many = [np.random.randint(1, size=100) for i in range(1000)]
test = np.random.randint(10, size=100)
many[500] = test

def one():
    return any((test == a).all() for a in many)


def two():
    for a in many:
        if np.array_equal(a, test):
            return True
    return False


def three():
    return (many == test).all(axis=1).any(axis=0)


def four():
    return test.tolist() in np.stack(many).tolist()


n = 10000

one(): 11.013655535 True
two(): 22.107509324999995 True
three(): 4.249062799999997 True
four(): 21.908904289999995 True

And if we remove line many[500] = test, we would have:

one(): 20.931670298 False
two(): 42.155511837999995 False
three(): 4.231607447000009 False
four(): 22.614475677 False

In either cases, for large input sizes, method three seems to be noticeably the fastest. @Grismar's solution is second fastest close to method 4.

edited May 13, 2020 at 8:36

answered May 13, 2020 at 6:28

Ehsan

12.5k2 gold badges24 silver badges36 bronze badges

2 Comments

Grismar Over a year ago

Instead of taking it at face value, I figured I'd run the numbers - although I prefer some of your answers for style and I think the added value for understanding is great, these solutions are in fact all slower.

Ehsan Over a year ago

@Grismar Thank you for testing runtime. I was lazy about it. It is great. I like your solution too. I just added these solutions for extra knowledge. While your test might be correct, it is usually not a good practice to run it on small inputs. I added the same exact test with larger input and 2nd method in this solution proves to be one above the one suggested in your post. Posting test in a bit.

Collectives™ on Stack Overflow

Find if numpy array is in list of numpy arrays -- .all() not working

2 Answers 2

3 Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related