python: numpy: concatenation of named arrays

Question

Consider the following simple example:

x = numpy.array([(1,2),(3,4)],dtype=[('a','<f4'),('b','<f4')])
y = numpy.array([(1,2),(3,4)],dtype=[('c','<f4'),('d','<f4')])
numpy.hstack((x,y))

One will get the following error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python33\lib\site-packages\numpy\core\shape_base.py", line 226, in vstack
    return _nx.concatenate(list(map(atleast_2d,tup)),0)
TypeError: invalid type promotion

If array had not titles it works

x = numpy.array([(1,2),(3,4)],dtype='<f4')
y = numpy.array([(1,2),(3,4)],dtype='<f4')
numpy.hstack((x,y))

If I remove the names from x and y it works too.

Question: how to concatenate, vstack or hstack of titled numpy array ? Note: numpy.lib.recfunctions.stack_arrays doesn't work well either

senderle · Accepted Answer · 2013-09-02 22:13:33Z

4

The problem is that the types are different. The "title" is part of the type, and y uses different names from x, so the types are incompatible. If you use compatible types, everything works fine:

>>> x = numpy.array([(1, 2), (3, 4)], dtype=[('a', '<f4'), ('b', '<f4')])
>>> y = numpy.array([(5, 6), (7, 8)], dtype=[('a', '<f4'), ('b', '<f4')])
>>> numpy.vstack((x, y))
array([[(1.0, 2.0), (3.0, 4.0)],
       [(5.0, 6.0), (7.0, 8.0)]], 
      dtype=[('a', '<f4'), ('b', '<f4')])
>>> numpy.hstack((x, y))
array([(1.0, 2.0), (3.0, 4.0), (5.0, 6.0), (7.0, 8.0)], 
      dtype=[('a', '<f4'), ('b', '<f4')])
>>> numpy.dstack((x, y))
array([[[(1.0, 2.0), (5.0, 6.0)],
        [(3.0, 4.0), (7.0, 8.0)]]], 
      dtype=[('a', '<f4'), ('b', '<f4')])

Sometimes dstack, etc. are smart enough to coerce types in a sensible way, but numpy has no way to know how to combine record arrays with different user-defined field names.

If you want to concatenate the datatypes, then you have to create a new datatype. Don't make the mistake of thinking that the sequence of names (x['a'], x['b']...) constitutes a true dimension of the array; x and y above are 1-d arrays of blocks of memory, each of which contains two 32-bit floats that can be accessed using the names 'a' and 'b'. But as you can see, if you access an individual item in the array, you don't get another array as you would if it were truly a second dimension. You can see the difference here:

>>> x = numpy.array([(1, 2), (3, 4)], dtype=[('a', '<f4'), ('b', '<f4')])
>>> x[0]
(1.0, 2.0)
>>> type(x[0])
<type 'numpy.void'>

>>> z = numpy.array([(1, 2), (3, 4)])
>>> z[0]
array([1, 2])
>>> type(z[0])
<type 'numpy.ndarray'>

This is what allows record arrays to contain heterogenous data; record arrays can contain both strings and ints, but the trade-off is that you don't get the full power of an ndarray at the level of individual records.

The upshot is that to join individual blocks of memory, you actually have to modify the dtype of the array. There are a few ways to do this but the simplest I could find involves the little-known numpy.lib.recfunctions library (which I see you've already found!):

>>> numpy.lib.recfunctions.rec_append_fields(x, 
                                             y.dtype.names, 
                                             [y[n] for n in y.dtype.names])
rec.array([(1.0, 2.0, 1.0, 2.0), (3.0, 4.0, 3.0, 4.0)], 
      dtype=[('a', '<f4'), ('b', '<f4'), ('c', '<f4'), ('d', '<f4')])

edited Sep 2, 2013 at 22:13

answered Sep 2, 2013 at 13:26

senderle

152k36 gold badges218 silver badges244 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Hanan Shteingart Over a year ago

But this is not what I am looking for... I want the new array to have titles inherited from the join... e.g. after hstack I want to have titles: 'a','b','c','d'. Why does python care about the names and not just for the type?! Drives me nuts. I think I need to use Pandas rather than directly numpy.

senderle Over a year ago

@HananShteingart, you're using the wrong approach then -- you have to create a brand new datatype. It seems like you're wrongly assuming that x and y are 2-d arrays. They're not. See my edits above.

Collectives™ on Stack Overflow

python: numpy: concatenation of named arrays

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related