5

Currently, I have a 3D Python list in jagged array format.
A = [[[0, 0, 0], [0, 0, 0], [0, 0, 0]], [[0], [0], [0]]]

Is there any way I could convert this list to a NumPy array, in order to use certain NumPy array operators such as adding a number to each element.
A + 4 would give [[[4, 4, 4], [4, 4, 4], [4, 4, 4]], [[4], [4], [4]]].

Assigning B = numpy.array(A) then attempting to B + 4 throws a type error.
TypeError: can only concatenate list (not "float") to list

Is a conversion from a jagged Python list to a NumPy array possible while retaining the structure (I will need to convert it back later) or is looping through the array and adding the required the better solution in this case?

Improve this question
2
  • 1
    Conversion to NumPy array makes sense when the intended operations are compute heavy and are to performed with some regular pattern. So, to just add a scalar 4 might not be worth the trouble. Commented Oct 27, 2016 at 17:06
  • Is the jagged nature the result of some process? I would be useful if your lists elements of [0] could have been expanded to include a nodata value, say [0, -1, -1] resulting in a more uniform list structure. This could then be easily converted to a numpy masked array. The masked array's nodata value would be set to -1 then all subsequent calculations would exclude those cell locations. You might find this useful if it is easier to 'fix' the lists than play around with patching that data structure to meet array requirements. Commented Oct 27, 2016 at 18:18

5 Answers 5

Reset to default
4

The answers by @SonderingNarcissit and @MadPhysicist are already quite nice.

Here is a quick way of adding a number to each element in your list and keeping the structure. You can replace the function return_number by anything you like, if you want to not only add a number but do something else with it:

def return_number(my_number):
    return my_number + 4    

def add_number(my_list):

    if isinstance(my_list, (int, float)):
        return return_number(my_list)
    else:
        return [add_number(xi) for xi in my_list]

A = [[[0, 0, 0], [0, 0, 0], [0, 0, 0]], [[0], [0], [0]]]

Then 

print(add_number(A))

gives you the desired output:

[[[4, 4, 4], [4, 4, 4], [4, 4, 4]], [[4], [4], [4]]]

So what it does is that it look recursively through your list of lists and everytime it finds a number it adds the value 4; this should work for arbitrarily deep nested lists. That currently only works for numbers and lists; if you also have e.g. also dictionaries in your lists then you would have to add another if-clause.

Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

For OP's purposes this works very well. My solution is a pretty comprehensive upgrade to yours that I have been working on iteratively via a series of related answers.
@MadPhysicist: Did not see your edits; your solution is indeed far more general than mine (upvoted it now) while mine might be easier to understand if one does not have much Python/programming background.
2

Since numpy can only work with regular-shaped arrays, it checks that all the elements of a nested iterable are the same length for a given dimension. If they are not, it still creates an array, but of type np.object instead of np.int as you would expect:

>>> B = np.array(A)
>>> B
array([[[0, 0, 0], [0, 0, 0], [0, 0, 0]],
       [[0], [0], [0]]], dtype=object)

In this case, the "objects" are lists. Addition is defined for lists, but only in terms of other lists that extend the original, hence your error. [0, 0] + 4 is an error, while [0, 0] + [4] is [0, 0, 4]. Neither is what you want.

It may be interesting that numpy will make the object portion of your array nest as low as possible. Array you created is actually a 2D numpy array containing lists, not a 1D array containing nested lists:

>>> B[0, 0]
[0, 0, 0]
>>> B[0, 0, 0]
Traceback (most recent call last):

  File "<ipython-input-438-464a9bfa40bf>", line 1, in <module>
    B[0, 0, 0]

IndexError: too many indices for array

As you pointed out, you have two options when it comes to ragged arrays. The first is to pad the array so it is non-ragged, convert it to numpy, and only use the elements you care about. This does not seem very convenient in your case.

The other method is to apply functions to your nested array directly. Luckily for you, I wrote a snippet/recipe in response to this question, which does exactly what you need, down to being able to support arbitrary levels of nesting and your choice of operators. I have upgraded it here to accept non-iterable nested elements anywhere along the list, including the original input and do a primitive form of broadcasting:

from itertools import repeat

def elementwiseApply(op, *iters):
    def isIterable(x):
        """
        This function is also defined in numpy as `numpy.iterable`.
        """
        try:
            iter(x)
        except TypeError:
            return False
        return True

    def apply(op, *items):
        """
        Applies the operator to the given arguments. If any of the
        arguments are iterable, the non-iterables are broadcast by
        `itertools.repeat` and the function is applied recursively
        on each element of the zipped result.
        """
        elements = []
        count = 0
        for iter in items:
            if isIterable(iter):
                elements.append(iter)
                count += 1
            else:
                elements.append(itertools.repeat(iter))
        if count == 0:
            return op(*items)
        return [apply(op, *items) for items in zip(*elements)]

    return apply(op, *iters)

This is a pretty general solution that will work with just about any kind of input. Here are a couple of sample runs showing how it is relevant to your question:

>>> from operator import add
>>> elementwiseApply(add, 4, 4)
8
>>> elementwiseApply(add, [4, 0], 4)
[8, 4]
>>> elementwiseApply(add, [(4,), [0, (1, 3, [1, 1, 1])]], 4)
[[8], [4, [5, 7, [5, 5, 5]]]]
>>> elementwiseApply(add, [[0, 0, 0], [0, 0], 0], [[4, 4, 4], [4, 4], 4])
[[4, 4, 4], [4, 4], 4]
>>> elementwiseApply(add, [(4,), [0, (1, 3, [1, 1, 1])]], [1, 1, 1])
[[5], [1, [2, 4, [2, 2, 2]]]]

The result is always a new list or scalar, depending on the types of the inputs. The number of inputs must be the number accepted by the operator. operator.add always takes two inputs, for example.

Improve this answer

2 Comments

I really admire the generality of this solution, it provides a varied amount of functionality in the general field I am asking in; however, it is a bit too broad for my concerns, and I believe someone looking for an answer to a question similar to mine would be interested more in a direct solution than broader one. For those reasons I chose @Cleb 's answer.
@Wintro. I think you did the right thing. This is more just a snippet that I have been having fun improving iteratively for a number of similar questions.
1

Looping and adding is likely better, since you want to preserve the structure of the original. Plus, the error you mentioned indicates that you would need to flatten the numpy array and then add to each element. Although numpy operations tend to be faster than list operations, converting, flattening, and reverting is cumbersome and will probably offset any gains.

Improve this answer

Comments

1

It we turn your list into an array, we get a 2d array of objects 

In [1941]: A = [[[0, 0, 0], [0, 0, 0], [0, 0, 0]], [[0], [0], [0]]]
In [1942]: A = np.array(A)
In [1943]: A.shape
Out[1943]: (2, 3)
In [1944]: A
Out[1944]: 
array([[[0, 0, 0], [0, 0, 0], [0, 0, 0]],
       [[0], [0], [0]]], dtype=object)

When I try A+1 it iterates over the elements of A and tries to do +1 for each. In the case of numeric array it can do that in fast compiled code. With an object array it has to invoke the + operation for each element.

In [1945]: A+1
...
TypeError: can only concatenate list (not "int") to list

Lets try that again with a flat iteration over A:

In [1946]: for a in A.flat:
      ...:     print(a+1)
....
TypeError: can only concatenate list (not "int") to list

The elements of A are lists; + for a list is a concatenate:

In [1947]: for a in A.flat:
      ...:     print(a+[1])
      ...:     
[0, 0, 0, 1]
[0, 0, 0, 1]
[0, 0, 0, 1]
[0, 1]
[0, 1]
[0, 1]

If the elements of A were themselves arrays, I think the +1 would work.

In [1956]: for i, a in np.ndenumerate(A):
      ...:     A[i]=np.array(a)
      ...:     
In [1957]: A
Out[1957]: 
array([[array([0, 0, 0]), array([0, 0, 0]), array([0, 0, 0])],
       [array([0]), array([0]), array([0])]], dtype=object)
In [1958]: A+1
Out[1958]: 
array([[array([1, 1, 1]), array([1, 1, 1]), array([1, 1, 1])],
       [array([1]), array([1]), array([1])]], dtype=object)

And to get back to the pure list form, we have apply tolist to both the elements of the object array and to the array itself:

In [1960]: A1=A+1
In [1961]: for i, a in np.ndenumerate(A1):
      ...:     A1[i]=a.tolist()

In [1962]: A1
Out[1962]: 
array([[[1, 1, 1], [1, 1, 1], [1, 1, 1]],
       [[1], [1], [1]]], dtype=object)
In [1963]: A1.tolist()
Out[1963]: [[[1, 1, 1], [1, 1, 1], [1, 1, 1]], [[1], [1], [1]]]

This a rather round about way of adding a value to all elements of nested lists. I could have done that with one iteration:

In [1964]: for i,a in np.ndenumerate(A):
      ...:     A[i]=[x+1 for x in a]
      ...:     
In [1965]: A
Out[1965]: 
array([[[1, 1, 1], [1, 1, 1], [1, 1, 1]],
       [[1], [1], [1]]], dtype=object)

So doing math on object arrays is hit and miss. Some operations do propagate to the elements, but even those depend on how the elements behave.

Improve this answer

2 Comments

I don't think that this really applies to OPs actual question.
I am converting from a jagged Python list to a NumPy array ... while retaining the structure (I will need to convert it back later) . Staying with pure lists might be faster due to array overhead, but if you do want to go the array route, this is a/the way to do it.
1

It is unfortunate that the input structure is a jagged list. If one could adjust the method used to generate the list by assigning no data values, then there is so much more one can do. I made this comment in the initial post, but I will demonstrate how the design of the originals could be altered to facilitate obtaining more data while enabling the return of a list.

I have done this as a function so I could comment the inputs and outputs for further reference.

def num_46():
    """(num_46)... Masked array from ill-formed list
    :  http://stackoverflow.com/questions/40289943/
    :  converting-a-3d-list-to-a-3d-numpy-array
    :  A =[[[0, 0, 0], [0, 0, 0], [0, 0, 0]], 
    :      [[0, 0, 0], [0, 0, 0], [0, 0, 0]], [[0], [0], [0]]]
    """
    frmt = """
    :Input list...
    {}\n
    :Masked array data
    {}\n
    :A sample calculations:
    :  a.count(axis=0) ... a.count(axis=1) ... a.count(axis=2)
    {}\n
    {}\n
    {}\n
    : and finally:  a * 2
    {}\n
    :Return it to a list...
    {}
    """
    a_list = [[[0, 1, 2], [3, 4, 5], [6, 7, 8]],
              [[9, 10, 11], [12, 13, 14], [15, 16, 17]],
              [[18, -1, -1], [21, -1, -1], [24, -1, -1]]]
    mask_val = -1
    a = np.ma.masked_equal(a_list, mask_val)
    a.set_fill_value(mask_val)
    final = a.tolist(mask_val)
    args = [a_list, a,
            a.count(axis=0), a.count(axis=1), a.count(axis=2),
            a*2, final]
    print(dedent(frmt).format(*args))
    return a_list, a, final


#----------------------
if __name__ == "__main__":
    """Main section...   """
    A, a, c = num_46()

Some results that show that the use of masked arrays may be preferable to jagged/malformed list structure. 

:Input list...
[[[0, 1, 2], [3, 4, 5], [6, 7, 8]],
 [[9, 10, 11], [12, 13, 14], [15, 16, 17]],
 [[18, -1, -1], [21, -1, -1], [24, -1, -1]]]

:Masked array data
[[[0 1 2]
  [3 4 5]
  [6 7 8]]

 [[9 10 11]
  [12 13 14]
  [15 16 17]]

 [[18 - -]
  [21 - -]
  [24 - -]]]

:A sample calculations:
:  a.count(axis=0) ... a.count(axis=1) ... a.count(axis=2)
[[3 2 2]
 [3 2 2]
 [3 2 2]]

[[3 3 3]
 [3 3 3]
 [3 0 0]]

[[3 3 3]
 [3 3 3]
 [1 1 1]]

: and finally:  a * 2
[[[0 2 4]
  [6 8 10]
  [12 14 16]]

 [[18 20 22]
  [24 26 28]
  [30 32 34]]

 [[36 - -]
  [42 - -]
  [48 - -]]]

:Return it to a list...
[[[0, 1, 2], [3, 4, 5], [6, 7, 8]], [[9, 10, 11], [12, 13, 14], [15, 16, 17]], [[18, -1, -1], [21, -1, -1], [24, -1, -1]]]

Hope this helps someone.

Improve this answer

2 Comments

Masking my data with some None values could be a good idea, but later on I am running multi-recursively through the list thousands of times, and the list is far bigger than shown here. So I think that a None-check for every single value and the added values in the list would seriously hinder performance.
@Wintro I was more curious as to how the list singletons arrived in the first place. I they were added during the construction, then the masked array doesn't check for them. It would be the case for processing the list however. This would be more useful if you intended to do the analysis with the array rather than the list. In any case, conversion to an array has its overhead which would have to be weighed against processing solely in list form.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.