Building and shifting numpy array

Question

I have such a numpy array:

And I want to shift this array with N so I can get a new array, such that first 4 rows will be the first 4 rows from the beginning array, and the below rows will be the rows from the previous step.

I dont know how to explain it correctly, but look at the example where I shift with 1 (not the whole array, but I would like to do this for the whole array):

And when I shift with 2 (not the whole array, but I would like to do this for the whole array):

And if I shift with 3, I will have 12 zeros in first column, and 3 previous rows ..

And here's the code:

import numpy as np
import numpy 

def shift_one(arr):
  arrT = arr.transpose()
  arrT_shape = arrT.shape[0]

  col1 = arrT[:, [0]]
  prev1 = numpy.zeros([arrT_shape, 1])   
  x1 = numpy.vstack([col1, prev1])

  col2 = arrT[:, [1]]
  prev2 = arrT[:, [0]]
  x2 = numpy.vstack([col2, prev2])

  col3 = arrT[:, [2]]
  prev3 = arrT[:, [1]]
  x3 = numpy.vstack([col3, prev3])

  col4 = arrT[:, [3]]
  prev4 = arrT[:, [2]]
  x4 = numpy.vstack([col4, prev4])

  col5 = arrT[:, [4]]
  prev5 = arrT[:, [3]]
  x5 = numpy.vstack([col5, prev5])

  # ... and so on, until we have index [23] and [22] and x24

  res = numpy.hstack([x1, x2, x3, x4, x5])
  return res 

def shift_two(arr):
    arrT = arr.transpose()
    arrT_shape = arrT.shape[0]
    new_size = 2 * arrT_shape

    col1 = arrT[:, [0]]
    prev1 = numpy.zeros([new_size, 1])
    x1 = numpy.vstack([col1, prev1])

    col22 = arrT[:, [1]]
    col21 = arrT[:, [0]]
    prev2 = numpy.zeros([arrT_shape, 1])
    x2 = numpy.vstack([col22, col21, prev2])

    col32 = arrT[:, [2]]
    col31 = arrT[:, [1]]
    col30 = arrT[:, [0]]
    x3 = numpy.vstack([col32, col31, col30])

    col42 = arrT[:, [3]]
    col41 = arrT[:, [2]]
    col40 = arrT[:, [1]]
    x4 = numpy.vstack([col42, col41, col40])

    col52 = arrT[:, [4]]
    col51 = arrT[:, [3]]
    col50 = arrT[:, [2]]
    x5 = numpy.vstack([col52, col51, col50])

    # ... and so on, until we have index [23], [22], [21] and x24

    res = numpy.hstack([x1, x2, x3, x4, x5])
    return res

arr1 = np.array([[0,   2,   0, 324], 
        [1, 2, 0,324], 
        [2, 2, 0, 324], 
        [3, 2, 0, 324], 
        [4, 2, 0, 324], 
        [5, 2, 0, 324], 
        [6, 2, 0, 324], 
        [7, 2, 0, 324], 
        [8, 2, 0, 324], 
        [9, 2, 0, 324], 
        [ 10, 2, 0, 324], 
        [ 11, 2, 0, 324], 
        [ 12, 2, 0, 324], 
        [ 13, 2, 0, 324], 
        [ 14, 2, 0, 324], 
        [ 15, 2, 0, 324], 
        [ 16, 2, 0, 324], 
        [ 17, 2, 0, 324], 
        [ 18, 2, 0, 324], 
        [ 19, 2, 0, 324], 
        [ 20, 2, 0, 324], 
        [ 21, 2, 0, 324], 
        [ 22, 2, 0, 324], 
        [ 23, 2, 0, 324]])

print(arr1)

print('\n\n')
one = shift_one(arr1)
print(one)

print('\n\n')
two = shift_two(arr1)
print(two)

Basically, I have a problem how to write a function, which will shift it with a given N ... I know how to write it step by step as I did, but I have a problem transforming it into something more usable. Thank you.

So again, an example for array with 5 columns (the original array has 24 columns):

Your original array has 24 rows. Your results seem to indicate you meant for it to be 24 columns. Which is it? — kabanus
– kabanus, Commented Jun 2, 2020 at 12:16
@kabanus: I did array transposition. Then, I wrote only for 5, but step by step. I need some help with X columns and delays > 2 — Brian Brown
– Brian Brown, Commented Jun 2, 2020 at 12:17
Then please show the transposed input, if that's what you want.... — Mad Physicist
– Mad Physicist, Commented Jun 2, 2020 at 12:26

Mad Physicist · Accepted Answer · 2020-06-02 15:51:19Z

Looped

If you want to do this for the general case, you'll can start with a simple loop. If the input array has shape (M, N), the output will be (M * (delay + 1), N).

Stacking is one way to combine arrays. You can minimize the required number of stacks by always doubling the result:

def stagger(a, delay):
    m = a.shape[0]
    z = 1
    d = delay + 1
    while z < d:
        k = min(d - z, z)    # This will only kick in for the last iteration
        new = np.concatenate([np.zeros((m * k, z)), a[:m * k, :-z]], axis=1)
        a = np.concatenate([a, new], axis=0)
        z *= 2
    return a

Vectorized

That being said, there is a fully vectorized way to do this using the monster that is np.lib.stride_tricks.asstrided. Imagine an array that is the first row of your original data padded on the left with delay + 1 zeros:

>>> d = delay + 1    # delay = 7
>>> row = np.concatenate([np.zeros(d), a[0]])
>>> row
array([ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.,  2.,  3.,  4.,  5.,  6.,  7., 
        8.,  9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23.])

You can view this array (without copying the data) as a staggered 2D buffer by adding a fake second dimension whose stride is one element shorter than it ought to be:

>>> m, n = a.shape
>>> staggered_row = np.lib.stride_tricks.as_strided(row[d:], shape=(d, n), strides=(-row.strides[0], row.strides[0]))
>>> staggered_row
array([[ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10., 11.,
        12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23.],
       [ 0.,  0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10.,
        11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22.],
       [ 0.,  0.,  0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9.,
        10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21.],
       [ 0.,  0.,  0.,  0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,
         9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20.],
       [ 0.,  0.,  0.,  0.,  0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,
         8.,  9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19.],
       [ 0.,  0.,  0.,  0.,  0.,  0.,  1.,  2.,  3.,  4.,  5.,  6.,
         7.,  8.,  9., 10., 11., 12., 13., 14., 15., 16., 17., 18.],
       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.,  2.,  3.,  4.,  5.,
         6.,  7.,  8.,  9., 10., 11., 12., 13., 14., 15., 16., 17.],
       [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.,  2.,  3.,  4.,
         5.,  6.,  7.,  8.,  9., 10., 11., 12., 13., 14., 15., 16.]])

Here, the negative stride adds the elements of the zero padding into the array. Keep in mind that this is generally not safe and should be done with extreme care. The output will not be writable by default, because all the rows refer to largely the same memory locations.

Now imagine doing the same thing to your entire array: padding and adding a new dimension that lets you step back into the padding. If you order the dimensions carefully, a reshape of the view will create a new copy that looks exactly how you want in one step. Specifically, starting with shape (M, N), we pad to (D + M, N), create a view that is (D, M, N), and reshape to (D * M, N):

def staggerx(a, delay):
    d = delay + 1
    m, n = a.shape
    padded = np.concatenate([np.zeros((m, d)), a], axis=1)
    view = np.lib.stride_tricks.as_strided(padded[:, d:], shape=(d, m, n), strides=(-padded.strides[1], padded.strides[0], padded.strides[1]))
    result = view.reshape(-1, n)
    return result

A reshape of a crazily strided array will always make a copy because the data is not contiguous by the way.

kabanus · Accepted Answer · 2020-06-02 12:26:16Z

1

If the data is not too humongous concatenating a couple of times will work:

>>> x
array([[  0,   1,   2,   3,   4],
       [  2,   2,   2,   2,   2],
       [  0,   0,   0,   0,   0],
       [324, 324, 324, 324, 324]])
>>> np.concatenate([x] + [np.concatenate([np.zeros(x.shape[0]*i).reshape(-1, i), x[: , i:]] ,axis=1) for i in range(1,x.shape[1])], axis=0)
array([[  0.,   1.,   2.,   3.,   4.],
       [  2.,   2.,   2.,   2.,   2.],
       [  0.,   0.,   0.,   0.,   0.],
       [324., 324., 324., 324., 324.],
       [  0.,   1.,   2.,   3.,   4.],
       [  0.,   2.,   2.,   2.,   2.],
       [  0.,   0.,   0.,   0.,   0.],
       [  0., 324., 324., 324., 324.],
       [  0.,   0.,   2.,   3.,   4.],
       [  0.,   0.,   2.,   2.,   2.],
       [  0.,   0.,   0.,   0.,   0.],
       [  0.,   0., 324., 324., 324.],
       [  0.,   0.,   0.,   3.,   4.],
       [  0.,   0.,   0.,   2.,   2.],
       [  0.,   0.,   0.,   0.,   0.],
       [  0.,   0.,   0., 324., 324.],
       [  0.,   0.,   0.,   0.,   4.],
       [  0.,   0.,   0.,   0.,   2.],
       [  0.,   0.,   0.,   0.,   0.],
       [  0.,   0.,   0.,   0., 324.]])

Each time I generate a 4xi matrix of zeros, and take the i+1: columns from the original matrix to generate the next part. Finally, I concatenate all these arrays.

edited Jun 2, 2020 at 12:26

answered Jun 2, 2020 at 12:22

kabanus

26.3k7 gold badges48 silver badges79 bronze badges

4 Comments

Brian Brown Over a year ago

Thank you so much. But I also need a way how to do this for big arrays and quite big delays.

kabanus Over a year ago

@BrianBrown Giga big? Give it a try, if this takes more than a few seconds comment here.

Mad Physicist Over a year ago

@kabanus. I believe that I've found a vectorized solution

Mad Physicist Over a year ago

@kabanus. Thanks. I've crashed python more times than I care to admit with that function.

Collectives™ on Stack Overflow

Building and shifting numpy array

2 Answers 2

Comments

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related