NumPy: load heterogenous columns of data from list of strings

Question

I'm working with array data stored in an ASCII file (similar to this thread). My file is at least 2M lines (158 MB), and is divided into multiple sections with different schemas. In my module to read the format, I want to read the whole file via lines = open('myfile.txt', 'r').readlines(), so I can index the positions of each section, then read each section that I need into NumPy data structures.

For example, one excerpt of a section is:

>>> print lines[5:10]
['      1 0.1000 0.300E-03 0.000E+00 0.300E-03 0.000E+00 0.000E+00 0.300E-03 0.100E-03\n', 
'      2 0.1000 0.120E-02 0.000E+00 0.120E-02 0.000E+00 0.000E+00 0.120E-02 0.100E-03\n', 
'      3 0.1000 0.100E-02 0.000E+00 0.100E-02 0.000E+00 0.000E+00 0.100E-02 0.100E-03\n', 
'      4 0.1000 0.110E-02 0.000E+00 0.110E-02 0.000E+00 0.000E+00 0.110E-02 0.100E-03\n', 
'      5 0.1000 0.700E-03 0.000E+00 0.700E-03 0.000E+00 0.000E+00 0.700E-03 0.100E-03\n']

Which has the schema [int, float, float, float, float, float, float, float, float], and a later part will have have a simpler [int, float] schema:

>>> print lines[20:25]
['       1         0.00000E+00\n',
'       2         0.43927E-07\n',
'       3         0.44006E-07\n',
'       4         0.44020E-07\n',
'       5         0.44039E-07\n']

How can I quickly load in different sections of the lines with NumPy? I see there is np.loadtxt, but it requires a file handle, and reads all the way to the end. I also see np.from* functions, but I'm not sure how to use them with my already read lines. Do I need to read the file twice?

With regards to the heterogeneous data types, I figure I can use a compound dtype, like np.dtype([('col1', '<i2'), ('col2', 'f4'), ('col3', 'f4'), ('col4', 'f4'), ('col5', 'f4'), ('col6', 'f4'), ('col7', 'f4'), ('col8', 'f4'), ('col9', 'f4')]), correct?

highBandWidth · Accepted Answer · 2011-04-14 03:25:08Z

3

StringIO can make file-type objects from strings. So you could do

from StringIO import StringIO
m = np.loadtxt(StringIO('\n'.join(lines[5:10])))

Or even easier, just do

m = np.fromiter(lines[5:10],np.dtype([('col1', '<i2'), ('col2', 'f4'), ('col3', 'f4')]))

edited Apr 14, 2011 at 3:25

answered Apr 14, 2011 at 1:26

highBandWidth

17.4k23 gold badges87 silver badges132 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Mike T Over a year ago

I was getting a related MemoryError with the string join of a few million lines. So instead, I'm feeding my readline file input to a StringIO buffer to pass on to np.loadtxt (I need to pre-process the input via readline to find relevant sections). All great tips, thanks!

Mike T Over a year ago

Note: the np.fromiter example does not work as expected, for unknown reasons.

Collectives™ on Stack Overflow

NumPy: load heterogenous columns of data from list of strings

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related