1

I have a very large 400x300x60x27 array (lets call it 'A'). I took the maximum values which is now a 400x300x60 array called 'B'. Basically I need to find the index in 'A' of each value in 'B'. I have converted them both to lists and set up a for loop to find the indices, but it takes an absurdly long time to get through it because there are over 7 million values. This is what I have:

B=np.zeros((400,300,60))
C=np.zeros((400*300*60))
B=np.amax(A,axis=3)
A=np.ravel(A)
A=A.tolist()
B=np.ravel(B)
B=B.tolist()
for i in range(0,400*300*60):
    C[i]=A.index(B[i])

Is there a more efficient way to do this? Its taking hours and hours and the program is still stuck on the last line.

Improve this question
3
  • 1
    Do you want to find a value in the entire 4D array, or do you want to find it out of the 27 elements it is coming from? Do you need one hit per element or all hits? Do you have a use-case which you can share, or is it secret? Commented Jul 23, 2017 at 16:56
  • Did you test this on small arrays; ones where you can actually see what's going on, and experiment with alternative approaches? Have you looked at argmax? Commented Jul 23, 2017 at 17:04
  • (I can not edit the previous one any more, but it feels like a field of 3x3x3 tensors over a 400x300x60 volume, but that is just a blind guess - however if this is the case you might be looking for some gradient/direction on a per-voxel basis, and then you certainly would not want to search in the entire array, and perhaps numpy.argmax is what you are looking for) Commented Jul 23, 2017 at 17:06

2 Answers 2

Reset to default
1

You don't need amax, you need argmax. In case of argmax, the array will only contain the indices rather than values, the computational efficiency of finding the values using indices are much better than vice versa.

So, I would recommend you to store only the indices. Before flattening the array.

instead of np.amax, run A.argmax, this will contain the indices. But before you're flattening it to 1D, you will need to use a mapping function that causes the indices to 1D as well. This is probably a trivial problem, as you'd need to just use some basic operations to achieve this. But that would also consume some time as it needs to be executed quite some times. But it won't be a searching probem and would save you quite some time.

Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

argmax did work, thank you very much! I've been searching everywhere I don't know why I didn't find it before.
As I'm not very good with numpy, I couldn't update your code. Can you please publish the updated code that you've got with my suggestion. That way, people can understand it better (I can't update because I don't have the dataset and I can't test it's efficiency otherwise)
1

You are getting those argmax indices and because of the flattening, you are basically converting to linear index equivalents of those.

Thus, a solution would be to add in the proper offsets into the argmax indices in steps leveraging broadcasting at each one of them, like so -

m,n,r,s = A.shape
idx = A.argmax(axis=3)
idx += s*np.arange(r)
idx += r*s*np.arange(n)[:,None]
idx += n*r*s*np.arange(m)[:,None,None] # idx is your C output

Alternatively, a compact way to put it would be like so -

m,n,r,s = A.shape
I,J,K = np.ogrid[:m,:n,:r]
idx = n*r*s*I + r*s*J + s*K + A.argmax(axis=3)
Improve this answer

3 Comments

Can np.ravel_multi_index be used?
@hpaulj For that you would need to create the mesh of all indices along the first three axes, stack them into a multi-dim ndarray and then use ravel_multi_index. Don't think that would be any more efficient than the posted ones.
Yes, we still need the ogrid; It just does the n*r*s multiplications for us: np.ravel_multi_index((I, J, K, idx), A.shape)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.