Optimize and remove for loop in numpy array

Question

I want to remove the for loop from the below code. In my use case, the value of n will be typically much larger where the value in the ranges of 200-700 is not uncommon, and it is inconvenient to list them all down, and adding one more loop would only make this more inefficient.

import numpy as np
n = 3
imgs = np.random.random((16,9,9,n))
transform = np.random.uniform(low=0.0, high=1.0, size=(n,n))
for img in imgs:
    for channel in range(img.shape[2]):
        temp1 = img[:,:,0]
        temp2 = img[:,:,1]
        temp3 = img[:,:,2]

        temp = temp1 * transform[channel][0] + temp2 * transform[channel][1] + temp3 * transform[channel][2]

        img[:,:,channel] = temp/3

Any pointers will be gratefully appreciated.

I assume that img and image in your code (please copy and paste from your actual code) are the same thing. --- --- That said, you cannot vectorize your inner loop because, e.g., temp1 has an initial value in the first loop (when channel==0) and a different value at the end of the same loop, because you change the data segment of the array img, and temp1 is just a view into that data segment. --- --- Is it possible that the code you posted is not doing what it's intended to do? — gboffi
– gboffi, Commented Nov 7, 2021 at 9:57
@gboffi just realized it. Edited the post, thanks for bringing it to my notice. — srdg
– srdg, Commented Nov 7, 2021 at 9:58
OK, thank you for the edit. --- --- In the inner loop, you are modifying the array on which you are operating, so you cannot vectorize the loop. --- --- On the other hand, what you are doing (changing one operand) is really uncommon and I suspect that you just want to perform an inner product between the two arrays, img and transform. — gboffi
– gboffi, Commented Nov 7, 2021 at 10:04
@gboffi I think so, yes. The vague idea was there - but I could not nail it. Please see the accepted answer and my comment on it. — srdg
– srdg, Commented Nov 7, 2021 at 10:06

veedata · Accepted Answer · 2021-11-07 09:03:24Z

2

I think you could maybe avoid the internal loop completely.

From my understanding you are taking a dot product of the transform matrix with the transpose of [temp1 temp2 temp3] matrix and then dividing it by 3.

The following is the representation of the same in an image:

So, all of this could actually be done outside the for loop itself. The code for that would look something like this. P.S. Also edited the variable names at some places where they felt inconsistent.

import numpy as np

n = 3
imgs = np.random.random((16,9,9,n))
transform = np.random.uniform(low=0.0, high=1.0, size=(n,n))

for img in imgs:
    temp_arr = img[:,:, 0:3]
    img[:,:, 0:3] = np.dot(temp_arr, np.transpose(transform))/3

Compared the result against yours and it gives the same output

edited Nov 7, 2021 at 9:03

answered Nov 7, 2021 at 8:10

veedata

1,2381 gold badge12 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

srdg Over a year ago

Thank you! I was sure it was something like this, but could never pin it down.

srdg Over a year ago

One question. The internal loop that you are storing the temp_arr in, is not dependent on n. If I assume that 0:3 can be replaced with 0:n, then it doesn't make sense to apply slicing, right? We can simply do a direct dot product and assign it to the img. Am I correct in my assumption?

veedata Over a year ago

@srdg I am not sure what you meant by direct dot product, but if you meant you want to do img[:,:, 0:n] = np.dot(img[:,:, 0:n], np.transpose(transform))/3, then you sure can do that. Infact, using 0:n will make your code more consistent as it will work equally well for n-channel images.

Andreas Hadjigeorgiou · Accepted Answer · 2021-11-07 08:36:37Z

0

Slicing might not be possible to be applied in the way you think it, but with numba you can get something that works without having to add another for loop.

This code performs the same operation as the code you provided.

import numpy as np
from numba import njit

@njit
def mult_arrays_values( array, values, n):
    temp = np.zeros((array.shape[0], array.shape[1], n))
    for i in range(n):
        temp[:,:,i] = array[:,:,i] * values[i]
    return temp

n = 3
imgs = np.random.random((16,9,9,n))
transform = np.random.uniform(low=0.0, high=1.0, size=(n,n))
for img in imgs:
    for channel in range(img.shape[2]):
        temp = mult_arrays_values(img[:,:,:], transform[channel][:], n)
        temp = np.add.reduce(temp , axis=2)
        img[:,:,channel] = temp/n

answered Nov 7, 2021 at 8:36

Andreas Hadjigeorgiou

3564 silver badges12 bronze badges

1 Comment

Andreas Hadjigeorgiou Over a year ago

@srdg is this working?

gboffi · Accepted Answer · 2021-11-07 10:55:01Z

You can also use numpy.einsum

In [77]: tr = np.arange(9).reshape((3,3))
    ...: imgs = np.arange(24).reshape((2,2,2,3)).transpose((1,0,2,3))
    ...: print('original imgs\n', imgs)
    ...: for img in imgs:
    ...:     img[:,:,:] = np.dot(img, tr.T)
    ...: print('imgs after loop\n', imgs)
    ...: imgs = np.arange(24).reshape((2,2,2,3)).transpose((1,0,2,3))
    ...: print('result of einsum\n', np.einsum('ijkl,ml', imgs, tr))
original imgs
 [[[[ 0  1  2]
   [ 3  4  5]]

  [[12 13 14]
   [15 16 17]]]


 [[[ 6  7  8]
   [ 9 10 11]]

  [[18 19 20]
   [21 22 23]]]]
imgs after loop
 [[[[  5  14  23]
   [ 14  50  86]]

  [[ 41 158 275]
   [ 50 194 338]]]


 [[[ 23  86 149]
   [ 32 122 212]]

  [[ 59 230 401]
   [ 68 266 464]]]]
result of einsum
 [[[[  5  14  23]
   [ 14  50  86]]

  [[ 41 158 275]
   [ 50 194 338]]]


 [[[ 23  86 149]
   [ 32 122 212]]

  [[ 59 230 401]
   [ 68 266 464]]]]

Collectives™ on Stack Overflow

Optimize and remove for loop in numpy array

3 Answers 3

3 Comments

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

3 Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related