Numpy: efficient way of filtering a very large array with a list of values

Question

Let's say I am manipulating a very large array of ints in numpy ( ). I want to filter it with a sublist of its values sublist. As the array is really large it looks like I need to be smart to do it in teh quickest way.

For instance:

my_array = N.random.randint(size=1e10)
sublist = [4,7,9]
#core where I extract the values of my_array equal to 4, 7 or 9

I've thought about:

cut = N.zeros((len(my_array)),dtype=bool)
for val in sublist:
    cut = cut | (my_array == val)
my_array = my_array[cut]

but it would have to parse the array len(sublist) amount of time.

Still manually:

cut = N.array([value in sublist for value in my_array])
my_array = my_array[cut]

but is there a more numpytonic way of doing so?

Is my_array sorted?

Divakar
– Divakar

2015-07-23 14:20:05 +00:00
Commented Jul 23, 2015 at 14:20 — Divakar
– Divakar, Commented Jul 23, 2015 at 14:20
@Divakar: not in the most general case.

Learning is a mess
– Learning is a mess

2015-07-23 16:04:31 +00:00
Commented Jul 23, 2015 at 16:04 — Learning is a mess
– Learning is a mess, Commented Jul 23, 2015 at 16:04

user2379410 · Accepted Answer · 2015-07-23 14:00:20Z

4

numpy.in1d does exactly this. Your code would look like:

cut = N.in1d(my_array, sublist)
my_array = my_array[cut]

answered Jul 23, 2015 at 14:00

user2379410

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Numpy: efficient way of filtering a very large array with a list of values

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related