3

I have an NxM numpy array called data. I also have a length N array called start_indices. I want a new length M array where the ith element is sum(data[i][start_indices[i]:]).

Here's one way to do it:

import numpy as np
data = np.linspace(0, 11, 12).reshape((3, 4))
data
array([[0, 1, 2, 3],
       [4, 5, 6, 7],
       [8, 9, 10, 11]])
start_indices = np.array([0, 1, 2])
sums = []
for start_index, row in zip(start_indices, data):
    sums.append(np.sum(row[start_index:]))
sums = np.array(sums)

Is there a more numpythonic way?

Improve this question
2
  • Your loop could be recast as a list comprehension: np.array([np.sum(row[s:]) for s, row in zip(start,data)]). But no real difference in speed. Commented May 19, 2015 at 4:27
  • ^ indeed.. but the zip operation is not really needed though, might as well index the arrays without making new ones Commented May 19, 2015 at 10:33

3 Answers 3

Reset to default
7

You can create a mask array

>>> mask = start_indices[:,None] <= np.arange(data.shape[1])
>>> (data * mask).sum(axis=1)
array([  6.,  18.,  21.])

For, the last step you may also use np.einsum:

>>> np.einsum('ij,ij->i', data, mask)
array([  6.,  18.,  21.])

though using a mask array here may be inefficient and iterate too many indices.

Alternatively, np.fromiter:

>>> it = (r[i:].sum() for r, i in zip(data, start_indices))
>>> np.fromiter(it, data.dtype)
array([  6.,  18.,  21.])
Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

In small time tests, all variations of the zip iteration take about the same time; the iteration methods are just syntatic sugar. But the masked einsum is better than 2x faster. I wonder how it scales.
This masking is cute, but the matrix multiplication will get very slow as the size of data increases. Even the einsum looses to list comprehension as size(data) is increased. I tested and it seems the np.fromiterand simple comprehension without zipping (my answer!) are the fastest of the methods proposed.
Accepted for fromiter because it's easy to understand and as fast as the cumsum method from hpaulj (assuming I used timeit right).
2

Besides the zip iteration (several forms) and the masked sum, cumsum might be worth testing

data[:,::-1].cumsum(axis=1)[range(data.shape[0]), data.shape[1]-1-start_indices]

The cumsum on the correct axis is easy; most of the expression is used to pull out the desired sums.

In this small case it is faster than the zip iteration, but slower than the masked sums. But rankings may change with size.

I don't consider any of these alternatives more 'pythonic'. They also use approved Python methods. Ones that avoid the zip iteration might get numpy brownie points, but only if they improve speed where it matters.

np.reduceat promises even better speed (first cut, not generalized):

np.add.reduceat(data.ravel(),[0,4,5,8,10])[::2]

This is a test expression, and doesn't take into account the time it takes to generate the indices list

indices = np.array([0,4,4,8,8]); indices[::2] += start_indices
Improve this answer

1 Comment

The reduceat idea is interesting. The cumsum idea is almost perfect.
0 
sums = np.array( [data[i, start_indices[i]:].sum() for i in range(data.shape[0])] )
Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.