Construct single numpy array from smaller arrays of different sizes

Question

I have an array of values, x. Given 'start' and 'stop' indices, I need to construct an array y using sub-arrays of x.

import numpy as np
x = np.arange(20)
start = np.array([2, 8, 15])
stop = np.array([5, 10, 20])
nsubarray = len(start)

Where I would like y to be:

y = array([ 2,  3,  4,  8,  9, 15, 16, 17, 18, 19])

(In practice the arrays I am using are much larger).

One way to construct y is using a list comprehension, but the list needs to be flattened afterwards:

import itertools as it
y = [x[start[i]:stop[i]] for i in range(nsubarray)]
y = np.fromiter(it.chain.from_iterable(y), dtype=int)

I found that it is actually faster to use a for-loop:

y = np.empty(sum(stop - start), dtype = int)
a = 0
for i in range(nsubarray):
    b = a + stop[i] - start[i]
    y[a:b] = x[start[i]:stop[i]]
    a = b

I was wondering if anyone knows of a way that I can optimize this? Thank you very much!

EDIT

The following tests all of the times:

import numpy as np
import numpy.random as rd
import itertools as it


def get_chunks(arr, start, stop):
    rng = stop - start
    rng = rng[rng!=0]      #Need to add this in case of zero sized ranges
    np.cumsum(rng, out=rng)
    inds = np.ones(rng[-1], dtype=np.int)
    inds[rng[:-1]] = start[1:]-stop[:-1]+1
    inds[0] = start[0]
    np.cumsum(inds, out=inds)
    return np.take(arr, inds)


def for_loop(arr, start, stop):
    y = np.empty(sum(stop - start), dtype = int)
    a = 0
    for i in range(nsubarray):
        b = a + stop[i] - start[i]
        y[a:b] = arr[start[i]:stop[i]]
        a = b
    return y

xmax = 1E6
nsubarray = 100000
x = np.arange(xmax)
start = rd.randint(0, xmax - 10, nsubarray)
stop = start + 10

Which results in:

In [379]: %timeit np.hstack([x[i:j] for i,j in it.izip(start, stop)])
1 loops, best of 3: 410 ms per loop

In [380]: %timeit for_loop(x, start, stop)
1 loops, best of 3: 281 ms per loop

In [381]: %timeit np.concatenate([x[i:j] for i,j in it.izip(start, stop)])
10 loops, best of 3: 97.8 ms per loop

In [382]: %timeit get_chunks(x, start, stop)
100 loops, best of 3: 16.6 ms per loop

Daniel · Accepted Answer · 2014-03-11 19:18:54Z

3

This is a bit complicated, but quite fast. Basically what we do is create the index list based off vector addition and the use np.take instead of any python loops:

def get_chunks(arr, start, stop):
     rng = stop - start
     rng = rng[rng!=0]      #Need to add this in case of zero sized ranges
     np.cumsum(rng, out=rng)
     inds = np.ones(rng[-1], dtype=np.int)
     inds[rng[:-1]] = start[1:]-stop[:-1]+1
     inds[0] = start[0]
     np.cumsum(inds, out=inds)
     return np.take(arr, inds)

Check that it is returning the correct result:

xmax = 1E6
nsubarray = 100000
x = np.arange(xmax)
start = np.random.randint(0, xmax - 10, nsubarray)
stop = start + np.random.randint(1, 10, nsubarray)

old = np.concatenate([x[b:e] for b, e in izip(start, stop)])
new = get_chunks(x, start, stop)
np.allclose(old,new)
True

Some timings:

%timeit np.hstack([x[i:j] for i,j in zip(start, stop)])
1 loops, best of 3: 354 ms per loop

%timeit np.concatenate([x[b:e] for b, e in izip(start, stop)])
10 loops, best of 3: 119 ms per loop

%timeit get_chunks(x, start, stop)
100 loops, best of 3: 7.59 ms per loop

edited Mar 11, 2014 at 19:18

answered Mar 11, 2014 at 17:51

Daniel

19.6k7 gold badges64 silver badges74 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

turnerm Over a year ago

Wow thanks so much, this is definitely the fastest one. I've edited the main post!

Saullo G. P. Castro · Accepted Answer · 2014-03-11 13:59:40Z

2

Perhaps using zip, np.arange and np.hstack:

np.hstack([np.arange(i, j) for i,j in zip(start, stop)])

edited Mar 11, 2014 at 13:59

answered Mar 11, 2014 at 13:54

Saullo G. P. Castro

59.4k28 gold badges191 silver badges244 bronze badges

6 Comments

turnerm Over a year ago

I tried this, but it's a bit slower than the loop -- see my edit above.

askewchan Over a year ago

Faster than the loop for me if I use np.hstack([x[i:j] for i,j in izip(start, stop)])

askewchan Over a year ago

And this is marginally faster: np.hstack(map(lambda i,j: x[i:j], start, stop))

turnerm Over a year ago

@askewchan -- this definitely improves the performance, but for me it is still slower than the loop, I've edited my main post.

askewchan Over a year ago

@turnerm, clearly the differences are small then. Perhaps you should use the one you like the most. On my computer, I get 30.2 µs per loop for the for-loop, and 24.3 µs per loop for the lambda-map. I'm sure they scale differently, and I'm just testing with the small example you've posted.

|

Bi Rico · Accepted Answer · 2014-03-11 15:37:04Z

1

This is almost 3 times faster than the loop for me, almost all the time difference comes from replacing fromiter with concatenate:

import numpy as np
from itertools import izip

y = [x[b:e] for b, e in izip(start, stop)]
y = np.concatenate(y)

answered Mar 11, 2014 at 15:37

Bi Rico

25.9k3 gold badges57 silver badges75 bronze badges

3 Comments

turnerm Over a year ago

Thanks so much, that absolutely works! I was blindly following this post to flatten my list: stackoverflow.com/questions/22326882/…

hpaulj Over a year ago

That link is to yourself.

turnerm Over a year ago

Oh whoops, I meant this: stackoverflow.com/questions/15366053/…

bserra · Accepted Answer · 2014-03-11 13:25:18Z

0

Would it be okay using slices instead of np.arrays?

import numpy as np
x = np.arange(10)
start = slice(2, 8)
stop = slice(5, 10)

print np.concatenate((x[start], x[stop]))

answered Mar 11, 2014 at 13:25

bserra

5405 silver badges14 bronze badges

1 Comment

turnerm Over a year ago

I have not tried that, although I am not sure if this method is practical for 100,000's of slices (which is what I am really dealing with unfortunately). I will give it a shot though, and post timings. By the way, I don't think my explanation of 'start' and 'stop' were totally clear, see my edit, i.e., in your example it should be: s1 = slice(2,5), s2 = slice(8,10), y = np.concatenate((x[s1], x[s2]))

Collectives™ on Stack Overflow

Construct single numpy array from smaller arrays of different sizes

4 Answers 4

1 Comment

6 Comments

3 Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

1 Comment

6 Comments

3 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related