Python Dynamic Array allocation, Matlab style

Question

I'm trying to move a few Matlab libraries that I've built to the python environment. So far, the biggest issue I faced is the dynamic allocation of arrays based on index specification. For example, using Matlab, typing the following:

x = [1 2];
x(5) = 3;

would result in:

x = [ 1     2     0     0     3]

In other words, I didn't know before hand the size of (x), nor its content. The array must be defined on the fly, based on the indices that I'm providing.

In python, trying the following:

from numpy import *
x = array([1,2])
x[4] = 3

Would result in the following error: IndexError: index out of bounds. On workaround is incrementing the array in a loop and then assigned the desired value as :

from numpy import *
x = array([1,2])

idx = 4
for i in range(size(x),idx+1):
    x = append(x,0)

x[idx] = 3
print x

It works, but it's not very convenient and it might become very cumbersome for n-dimensional arrays.I though about subclassing ndarray to achieve my goal, but I'm not sure if it would work. Does anybody knows of a better approach?

Thanks for the quick reply. I didn't know about the setitem method (I'm fairly new to Python). I simply overwritten the ndarray class as follows:

import numpy as np

class marray(np.ndarray):

    def __setitem__(self, key, value):

        # Array properties
        nDim = np.ndim(self)
        dims = list(np.shape(self))

        # Requested Index
        if type(key)==int: key=key,
        nDim_rq = len(key)
        dims_rq = list(key)

        for i in range(nDim_rq): dims_rq[i]+=1        

        # Provided indices match current array number of dimensions
        if nDim_rq==nDim:

            # Define new dimensions
            newdims = []
            for iDim in range(nDim):
                v = max([dims[iDim],dims_rq[iDim]])
                newdims.append(v)

            # Resize if necessary
            if newdims != dims:
              self.resize(newdims,refcheck=False)

        return super(marray, self).__setitem__(key, value)

And it works like a charm! However, I need to modify the above code such that the setitem allow changing the number of dimensions following this request:

a = marray([0,0])
a[3,1,0] = 0

Unfortunately, when I try to use numpy functions such as

self = np.expand_dims(self,2)

the returned type is numpy.ndarray instead of main.marray. Any idea on how I could enforce that numpy functions output marray if a marray is provided as an input? I think it should be doable using array_wrap, but I could never find exactly how. Any help would be appreciated.

Your append-in-loop algorithm takes quadratic time, so it'll become very slow for large arrays. — Fred Foo
– Fred Foo, Commented Jul 13, 2012 at 14:44
Take it as an opportunity to get rid of all your ad-hoc array-reallocations ;) — sebastian
– sebastian, Commented Nov 14, 2013 at 14:49
a) If you don't intend to have sparse values, i.e. you want to set them sequentially you could something like np.fromiter. b) if you still intend on subclass ndarray, this article should clear up why the type is not preserved and how you can fix it docs.scipy.org/doc/numpy-1.10.1/user/basics.subclassing.html — Erik
– Erik, Commented Jun 19, 2016 at 19:59

Community · Accepted Answer · 2017-05-23 12:33:25Z

Took the liberty of updating my old answer from Dynamic list that automatically expands. Think this should do most of what you need/want

class matlab_list(list):
    def __init__(self):
        def zero():
            while 1:
                yield 0
        self._num_gen = zero()

    def __setitem__(self,index,value):
        if isinstance(index, int):
            self.expandfor(index)
            return super(dynamic_list,self).__setitem__(index,value)

        elif isinstance(index, slice):
            if index.stop<index.start:
                return super(dynamic_list,self).__setitem__(index,value)
            else:
                self.expandfor(index.stop if abs(index.stop)>abs(index.start) else index.start)
            return super(dynamic_list,self).__setitem__(index,value)

    def expandfor(self,index):
            rng = []
            if abs(index)>len(self)-1:
                if index<0:
                    rng = xrange(abs(index)-len(self))
                    for i in rng:
                        self.insert(0,self_num_gen.next())
                else:
                    rng = xrange(abs(index)-len(self)+1)
                    for i in rng:
                        self.append(self._num_gen.next())

# Usage
spec_list = matlab_list()
spec_list[5] = 14

Bi Rico · Accepted Answer · 2012-07-13 18:45:50Z

This isn't quite what you want, but...

x = np.array([1, 2])

try:
    x[index] = value
except IndexError:
    oldsize = len(x)   # will be trickier for multidimensional arrays; you'll need to use x.shape or something and take advantage of numpy's advanced slicing ability
    x = np.resize(x, index+1) # Python uses C-style 0-based indices
    x[oldsize:index] = 0 # You could also do x[oldsize:] = 0, but that would mean you'd be assigning to the final position twice.
    x[index] = value

>>> x = np.array([1, 2])
>>> x = np.resize(x, 5)
>>> x[2:5] = 0
>>> x[4] = 3
>>> x
array([1, 2, 0, 0, 3])

Due to how numpy stores the data linearly under the hood (though whether it stores as row-major or column-major can be specified when creating arrays), multidimensional arrays are pretty tricky here.

>>> x = np.array([[1, 2, 3], [4, 5, 6]])
>>> np.resize(x, (6, 4))
array([[1, 2, 3, 4],
       [5, 6, 1, 2],
       [3, 4, 5, 6],
       [1, 2, 3, 4],
       [5, 6, 1, 2],
       [3, 4, 5, 6]])

You'd need to do this or something similar:

>>> y = np.zeros((6, 4))
>>> y[:x.shape[0], :x.shape[1]] = x
>>> y
array([[ 1.,  2.,  3.,  0.],
       [ 4.,  5.,  6.,  0.],
       [ 0.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  0.],
       [ 0.,  0.,  0.,  0.]])

clampfferj · Accepted Answer · 2012-07-13 15:54:56Z

0

A python dict will work well as a sparse array. The main issue is the syntax for initializing the sparse array will not be as pretty:

listarray = [100,200,300]
dictarray = {0:100, 1:200, 2:300}

but after that the syntax for inserting or retrieving elements is the same

dictarray[5] = 2345

answered Jul 13, 2012 at 15:54

clampfferj

311 bronze badge

1 Comment

osoucy Over a year ago

Unfortunately, this will not work in n-dimensions and will probably not be as efficient as an array.

Collectives™ on Stack Overflow

Python Dynamic Array allocation, Matlab style

3 Answers 3

Comments

Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related