Making for loop with index arrays faster

Question

I have the following problem: I have index arrays with repeating indices and would like to add values to an array like this:

grid_array[xidx[:],yidx[:],zidx[:]] += data[:]

However, as I have repeated indices this does not work as it should because numpy will create a temporary array which results in the data for the repeated indices being assigned several times instead of being added to each other (see http://docs.scipy.org/doc/numpy/user/basics.indexing.html).

A for loop like

for i in range(0,n):
    grid_array[xidx[i],yidx[i],zidx[i]] += data[i]

will be way to slow. Is there a way I can still use the vectorization of numpy? Or is there another way to make this assignment faster?

Thanks for your help

By repeating, you mean some of the index pairs can be duplicates? So you might get grid_array(1,3,4) + data[3]+data[4}+data[10]? — hpaulj
– hpaulj, Commented Mar 20, 2015 at 16:59
Exactly, some index pairs will be duplicate, and the case you described will happen. — numberCruncher
– numberCruncher, Commented Mar 20, 2015 at 20:35

Bi Rico · Accepted Answer · 2015-03-20 19:26:48Z

3

How about using bincount?

import numpy as np

flat_index = np.ravel_multi_index([xidx, yidx, zidx], grid_array.shape)
datasum = np.bincount(flat_index, data, minlength=grid_array.size)
grid_array += datasum.reshape(grid_array.shape)

edited Mar 20, 2015 at 19:26

answered Mar 20, 2015 at 19:20

Bi Rico

25.9k3 gold badges57 silver badges75 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

hpaulj · Accepted Answer · 2015-03-20 16:50:37Z

2

This is a buffering issue. The .at provides unbuffered action http://docs.scipy.org/doc/numpy/reference/generated/numpy.ufunc.at.html#numpy.ufunc.at

np.add.at(grid_array, (xidx,yidx,zidx),data)

answered Mar 20, 2015 at 16:50

hpaulj

233k14 gold badges260 silver badges392 bronze badges

2 Comments

numberCruncher Over a year ago

Thanks for making me aware of this.

PhoemueX Over a year ago

Thanks for the great answer. This helped me a lot! And I think this answer is much better than the currently accepted one... In fact, if I ever get >= 150 points, I will award you 50 as a bounty! :)

Kasravnd · Accepted Answer · 2015-03-20 14:29:33Z

0

For add an array to elements of a nested array you just can do grid_array[::]+=data :

>>> grid_array=np.array([[1,2,3],[4,5,6],[7,8,9]])
>>> data=np.array([3,3,3])
>>> grid_array[::]+=data
>>> grid_array
array([[ 4,  5,  6],
       [ 7,  8,  9],
       [10, 11, 12]])

answered Mar 20, 2015 at 14:29

Kasravnd

108k19 gold badges167 silver badges195 bronze badges

1 Comment

numberCruncher Over a year ago

I think this is not what I was looking for, I wanted to add different data to a 3-D array.

numberCruncher · Accepted Answer · 2015-03-20 14:43:19Z

-1

I think I found a possible solution:

def assign(xidx,yidx,zidx,data):
    grid_array[xidx,yidx,zidx] += data
    return

map(assign,xidx,yidx,zidx,sn.part0['mass'])

answered Mar 20, 2015 at 14:43

numberCruncher

6551 gold badge8 silver badges27 bronze badges

4 Comments

hpaulj Over a year ago

map just hides the loop.

numberCruncher Over a year ago

I guess it must be handled differently than the for loop, as it runs a lot faster.

hpaulj Over a year ago

In my small tests I don't get much difference in time with various looping constructs.

Bi Rico Over a year ago

Using map for in place operations (often called side-effects) is a bad idea. If for no other reason, you should avoid it because in python3 map returns a generator and you'll need to write for item in map(assign,xidx,yidx,zidx,sn.part0['mass']): pass which is really ugly.

Collectives™ on Stack Overflow

Making for loop with index arrays faster

4 Answers 4

Comments

2 Comments

1 Comment

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Comments

2 Comments

1 Comment

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related