Filtering Numpy Array representing state

Question

There are various questions regarding filtering of the numpy arrays including:

But I have a slightly different issue:

>>> x = np.empty(shape=(5,), dtype=[('ts', 'i8'), ('data', 'i8')])
>>> x['ts'] = [0, 1, 2, 5, 6]
>>> x['data'] = [1, 2, 3, 4, 5]
>>> x
array([(0, 1), (1, 2), (2, 3), (5, 4), (6, 5)],
      dtype=[('ts', '<i8'), ('data', '<i8')])
>>> x[(x['ts'] > 2) & (x['ts'] < 4.9)]
array([], dtype=[('ts', '<i8'), ('data', '<i8')])
>>>

This is exactly what I would expect. However, I need the filtered array to include 5 as well. Is there a way to filter it other then with a for or a while loop iterating over rows of the array and including the row with index following the last row matching the condition?

What do you mean when you say you need to include "5 as well"? — dumbPy
– dumbPy, Commented May 21, 2020 at 15:58
@dumbPy I need to filter based on ts where the end condition is < 5 but I need to have the row where ts == 5 also included. — Karlson
– Karlson, Commented May 21, 2020 at 16:01
what's wrong with good old <= 5? x[(x['ts'] > 2) & (x['ts'] <= 5)] — dumbPy
– dumbPy, Commented May 21, 2020 at 16:04
@dumbPy Nothing if you know that 5 is the next ts. When I have to filter I don't know that the 5 is the next ts. All I know is the minimum and maximum — Karlson
– Karlson, Commented May 21, 2020 at 16:07

Sinbud · Accepted Answer · 2020-05-21 17:00:39Z

1

Couldn't find a built-in numpy solution for this kind of 'positive lookbehind' matching problem. Maybe something like this will do:

idx_l = np.where(x['ts']<=2)[0]
idx_r = np.where(x['ts']>=4.9)[0]
x[idx_l[-1]+1:idx_r[0]+1]

To prevent IndexError in case idx_l or idx_r is empty:

idx = np.concatenate([idx_l[:], idx_r[1:]], axis=0)
np.delete(x, idx)

This approach solves the problem when the filtering condition doesn't return any index from which you can take an offset (to include the boundary value). However it will run slower since np.where is called twice.

answered May 21, 2020 at 17:00

Sinbud

405 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Filtering Numpy Array representing state

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related