Convert list of tuples to structured numpy array

Question

I have a list of Num_tuples tuples that all have the same length Dim_tuple

xlist = [tuple_1, tuple_2, ..., tuple_Num_tuples]

For definiteness, let's say Num_tuples=3 and Dim_tuple=2

xlist = [(1, 1.1), (2, 1.2), (3, 1.3)]

I want to convert xlist into a structured numpy array xarr using a user-provided list of column names user_names and a user-provided list of variable types user_types

user_names = [name_1, name_2, ..., name_Dim_tuple]
user_types = [type_1, type_2, ..., type_Dim_tuple]

So in the creation of the numpy array,

dtype = [(name_1,type_1), (name_2,type_2), ..., (name_Dim_tuple, type_Dim_tuple)]

In the case of my toy example desired end product would look something like:

xarr['name1']=np.array([1,2,3])
xarr['name2']=np.array([1.1,1.2,1.3])

How can I slice xlist to create xarr without any loops?

without any loops is this possible without loops? What about list comprehension? Also, have you tried anything? — Aleksander Lidtke
– Aleksander Lidtke, Commented Jan 27, 2015 at 18:03
Yes, though the only thing I've gotten to work are hard-coding solutions that first involve xlist --> np.array(xlist). — aph
– aph, Commented Jan 27, 2015 at 18:11
For example, xtemp = np.array(xlist), and x1=np.array(xtemp[:,1]), this creates a numpy array of one-element tuples, which is not what I want. I can't seem to get the slicing right, that's the entire problem. Should be simple, I realize. — aph
– aph, Commented Jan 27, 2015 at 18:16

chrisaycock · Accepted Answer · 2016-12-14 21:19:42Z

54

A list of tuples is the correct way of providing data to a structured array:

In [273]: xlist = [(1, 1.1), (2, 1.2), (3, 1.3)]

In [274]: dt=np.dtype('int,float')

In [275]: np.array(xlist,dtype=dt)
Out[275]: 
array([(1, 1.1), (2, 1.2), (3, 1.3)], 
      dtype=[('f0', '<i4'), ('f1', '<f8')])

In [276]: xarr = np.array(xlist,dtype=dt)

In [277]: xarr['f0']
Out[277]: array([1, 2, 3])

In [278]: xarr['f1']
Out[278]: array([ 1.1,  1.2,  1.3])

or if the names are important:

In [280]: xarr.dtype.names=['name1','name2']

In [281]: xarr
Out[281]: 
array([(1, 1.1), (2, 1.2), (3, 1.3)], 
      dtype=[('name1', '<i4'), ('name2', '<f8')])

http://docs.scipy.org/doc/numpy/user/basics.rec.html#filling-structured-arrays

edited Dec 14, 2016 at 21:19

chrisaycock

38.1k15 gold badges94 silver badges128 bronze badges

answered Jan 27, 2015 at 18:10

hpaulj

233k14 gold badges260 silver badges392 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

K.Nicholas Over a year ago

This got me to exactly where I started from.

Michael B. Currie · Accepted Answer · 2023-08-13 07:16:50Z

-2

hpaulj's answer is interesting but horrifying :)

The modern Pythonic way to have named columns is to use pandas, a highly popular package built on top of numpy:

import pandas as pd

xlist = [(1, 1.1), (2, 1.2), (3, 1.3)]

# Cast name1 to int because pandas' default is float
df = pd.DataFrame(xlist, columns=['name1', 'name2']).astype({'name1':int})
print(df)

This gives you a DataFrame, df, which is the structure you want:

   name1  name2
0      1    1.1
1      2    1.2
2      3    1.3

You can do all kinds of wonderful things with this, like slicing and various operations.

For example, to the create the xarr dictionary requested in the original question:

>>> xarr = {k:np.array(v) for k,v in df.to_dict(orient='list').items()}
>>> xarr
{'name1': array([1, 2, 3]), 'name2': array([1.1, 1.2, 1.3])}

edited Aug 13, 2023 at 7:16

answered Oct 17, 2022 at 16:39

Michael B. Currie

14.8k11 gold badges48 silver badges61 bronze badges

3 Comments

fwyzard Over a year ago

This answer does not answer the original question, and suggests using a much larger and bloated package that can be avoided using the accepted answer.

Michael B. Currie Over a year ago

OK fwyzard, good point; I have added a one-liner to convert to the exact format requested in the original question. Also, pandas' "bloat" could be considered a feature, not a bug: unless you have a very good reason, it is probably better to rely on a well-supported package rather than to try to "roll your own" solution to a basic data transformation task.

fwyzard Over a year ago

Hi Michael, I do agree with reusing well-supported software. However import pandas takes more than 0.5 seconds on an dual AMD EPYC 7763 server... for a simple transformation, that's likely more than the time taken by the operation itself :-( I'd be much happier with pandas if it could be used in a more modular way.

Collectives™ on Stack Overflow

Convert list of tuples to structured numpy array

2 Answers 2

1 Comment

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related