Converting a data structure of dtype=object to numpy array of dtype=float64

Lets take as the starting point a list of lists or equivalently an object array of lists:

A = [[], [], [], [1,2,1], [], [2, 2, 0, 0], []]
A = array([[], [], [], [1,2,1], [], [2, 2, 0, 0], []], dtype=object)

If the sublists were all the same length, np.array([...]) would give you a 2d array, one row for each sublist, and columns matching their common length. But since they are unequal in length, it can only make it a 1d array, where each element is a pointer to one of these sublists - i.e. dtype=object.

I can imagine 2 ways of constructing a 2d array:

• pad each sublist to a common length
• insert each sublist into an empty array of the appropriate size.

Basically it requires common Python iteration; it's not a common enough task to have a wiz-bang numpy function.

For example:

In [346]: n=len(A)
In [348]: m=max([len(x) for x in A])
In [349]: AA=np.zeros((n,m),int)
In [350]: for i,x in enumerate(A):
   .....:     AA[i,:len(x)] = x
In [351]: AA
Out[351]: 
array([[0, 0, 0, 0],
       [0, 0, 0, 0],
       [0, 0, 0, 0],
       [1, 2, 1, 0],
       [0, 0, 0, 0],
       [2, 2, 0, 0],
       [0, 0, 0, 0]])

To get a sparse matrix:

In [352]: from scipy import sparse
In [353]: MA=sparse.coo_matrix(AA)
In [354]: MA
Out[354]: 
<7x4 sparse matrix of type '<class 'numpy.int32'>'
    with 5 stored elements in COOrdinate format>

Nothing magical, just straight forward sparse matrix construction. I suppose you could bypass the dense matrix

There is a list-of-lists sparse format that looks a bit like your data.

In [356]: Ml=MA.tolil()

In [357]: Ml.rows
Out[357]: array([[], [], [], [0, 1, 2], [], [0, 1], []], dtype=object)

In [358]: Ml.data
Out[358]: array([[], [], [], [1, 2, 1], [], [2, 2], []], dtype=object)

Conceivably you could construct an empty sparse.lil_matrix((n,m)) matrix, and set it's .data attribute directly. But you'd also have to calculate the rows attribute.

You could also look at the data, row. col attributes of the coo format matrix, and decide it would be easy to construct the equivalent from your A list of lists.

One way or other you have to decide how the non-zero rows get padded to the full length.

This:

array = numpy.array ( [ x['feature1'] for x in ver ] )

Or you need to be more clear in your example...

You can access the value of a dictionary item by using its key:

d ={'a':1}
d['a'] --> 1

To access items in a list, you can iterate over it or use its index

a = [1,  2]

for thing in a:
    # do something with thing

a[0]  --> 1

map conveniently applies a function to all the items of an iterable and returns a list of the results. operator.getitem returns a function that will retrieve an item from an object.

import operator
import numpy as np
feature1 = operator.getitem('feature1')
a = np.asarray(map(feature1, vec))

vec = [{'is_Primary': 1, 'feature1': [2, 2, 2, 0, 0.03333333333333333, 0], 'object_id': ObjectId('557beda51d41c8e4d1aeac25'), 'vectorized': 1},
       {'is_Primary': 0, 'feature1': [2, 2, 1, 0, 0.5, 0], 'object_id': ObjectId('557beda51d41c8e4d1aeac25'), 'vectorized': 1}]

>>> a = np.asanyarray(map(feature1, vec))
>>> a.shape
(2, 6)
>>> print a
[[ 2.          2.          2.          0.          0.03333333  0.        ]
 [ 2.          2.          1.          0.          0.5         0.        ]]
>>> 
>>> for thing in a[1,:]:
    print type(thing)

<type 'numpy.float64'>
<type 'numpy.float64'>
<type 'numpy.float64'>
<type 'numpy.float64'>
<type 'numpy.float64'>
<type 'numpy.float64'>
>>>