Numpy sub-array assignment with advanced, mixed indexing

Question

Original question

I am getting a very odd error message when I try to assign some of the elements of an array. I am using a combination of a slice and a set of indices. See the following simple example.

 import scipy as sp

 a = sp.zeros((3, 4, 5))
 b = sp.ones((4, 5))

 I = sp.array([0, 1, 3])

 b[:, I] = a[0, :, I]

This code raises the following ValueError:

ValueError: shape mismatch: value array of shape (3,4) could not be broadcast to indexing result of shape (3,4)

--

Follow up

Be careful when using a combination of a slice and seq. of integers. As pointed out on github:

x = rand(3, 5, 7)

print(x[0, :, [0,1]].shape)
# (2, 5)

print(x[0][:, [0,1]].shape)
# (5, 2)

This is how numpy is designed to work, but it is nevertheless a bit confusing that x[0][:, I] is not the same as x[0, :, I]. Since this is the behavior I want I choose to use x[0][:, I] in my code.

Looks like you have a typo: would [0,1 4] actually be [0,1,4] (with a comma between 1 and 4)? — heltonbiker
– heltonbiker, Commented Mar 24, 2016 at 18:54

hpaulj · Accepted Answer · 2016-03-24 19:34:58Z

3

Looks like there are some errors in copying your code to question.

But I suspect there's a known problem with indexing:

In [73]: a=np.zeros((2,3,4)); b=np.ones((3,4)); I=np.array([0,1])

Make I 2 elements. Indexing b gives the expected (3,2) shape. 3 rows from the slice, 2 columns from I indexing

In [74]: b[:,I].shape
Out[74]: (3, 2)

But with 3d a we get the transpose.

In [75]: a[0,:,I].shape
Out[75]: (2, 3)

and assignment would produce an error

In [76]: b[:,I]=a[0,:,I]
...
ValueError: array is not broadcastable to correct shape

It's putting the 2 element dimension defined by I first, and the 3 element from : second. It's a case of mixed advanced indexing that has been discussed earlier - and there's a bug issue as well. (I'll have to look those up).

You are probably using a newer numpy (or scipy) and getting a different error message.

It's documented that indexing with two arrays or lists, and slice in the middle, puts the slice at the end, e.g.

In [86]: a[[[0],[0],[1],[1]],:,[0,1]].shape
Out[86]: (4, 2, 3)

The same thing is happening with a[0,:,[0,1]]. But there's a good argument that it shouldn't be this way.

As to a fix, you could transpose a value, or change the indexing

In [88]: b[:,I]=a[0:1,:,I]

In [90]: b[:,I]=a[0,:,I].T

In [91]: b
Out[91]: 
array([[ 0.,  0.,  1.,  1.],
       [ 0.,  0.,  1.,  1.],
       [ 0.,  0.,  1.,  1.]])

In [92]: b[:,I]=a[0][:,I]

https://github.com/numpy/numpy/issues/7030

https://github.com/numpy/numpy/pull/6256

edited Mar 24, 2016 at 19:34

answered Mar 24, 2016 at 19:08

hpaulj

233k14 gold badges260 silver badges392 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Will Martin Over a year ago

Thanks that is very helpful to know . . . slices in the middle of index lists get pushed to the last dimension.

thecardkid · Accepted Answer · 2016-03-24 19:00:10Z

1

First of all it looks like you're missing a comma on the line 6:

I = sp.array([0,1,4])

Secondly, I would expect the value 4 in the array I to raise an IndexError, since both a and b have a max dimension of 4. I suspect you might want:

I = sp.array([0,1,3])

Making these changes run the program for me, and I got b as:

[[ 0.  0.  1.  0.]
 [ 0.  0.  1.  0.]
 [ 0.  0.  1.  0.]]

Which I suspect is what you want.

answered Mar 24, 2016 at 19:00

thecardkid

9997 silver badges7 bronze badges

1 Comment

Will Martin Over a year ago

Thanks I just noticed that this test case does actually run so I changed the example. For some reason the smaller array sizes didn't raise the error I am experiencing.

heltonbiker · Accepted Answer · 2016-03-24 19:09:12Z

1

Here I get this error with indices [0,1,4]:

IndexError: index 4 is out of bounds for axis 2 with size 4

Its suggesting the value 4 is being used as an index, while the SIZE 4 implies the max index would be 3.

EDIT: now that you changed it to [0, 1, 3], it's running fine here.

EDIT: with your current code, I get the same error, but when I print the arrays themselves, they have a transverse shape:

print b[:, I]
print a[0, :, I]

[[ 1.  1.  1.]
 [ 1.  1.  1.]
 [ 1.  1.  1.]
 [ 1.  1.  1.]]

[[ 0.  0.  0.  0.]
 [ 0.  0.  0.  0.]
 [ 0.  0.  0.  0.]]

edited Mar 24, 2016 at 19:09

answered Mar 24, 2016 at 18:58

heltonbiker

27.7k30 gold badges151 silver badges270 bronze badges

Collectives™ on Stack Overflow

Numpy sub-array assignment with advanced, mixed indexing

Original question

Follow up

3 Answers 3

1 Comment

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

Original question

Follow up

3 Answers 3

1 Comment

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related