Opening text file in python as an array or list of list

Question

I am trying to open a text file in python as an array or a list of list. The file looks like below.
Also, here is a link to the text file.
ftp://rammftp.cira.colostate.edu/demaria/ebtrk/ebtrk_atlc.txt

AL0188 ALBERTO   080518 1988 32.0  77.5  20 1015 -99 -99  -99 -99   0  0  0  0   0  0  0  0   0  0  0  0 *   218.  
AL0188 ALBERTO   080600 1988 32.8  76.2  20 1014 -99 -99  -99 -99   0  0  0  0   0  0  0  0   0  0  0  0 *   213.  
AL0188 ALBERTO   080712 1988 41.5  69.0  35 1002 -99 -99 1012  60 100100 50 50   0  0  0  0   0  0  0  0 *   118.  
AL0188 ALBERTO   080718 1988 43.0  67.5  35 1002 -99 -99 1008  50 100100 50 50   0  0  0  0   0  0  0  0 *   144.  
AL0188 ALBERTO   080800 1988 45.0  65.5  35 1004 -99 -99 1008  50 -99-99-99-99   0  0  0  0   0  0  0  0 *    22.  
AL0188 ALBERTO   080806 1988 47.0  63.0  35 1006 -99 -99 1008  50 -99-99-99-99   0  0  0  0   0  0  0  0 *    64.

I have tried using NumPy genfromtxt but it returned with an error, because it couldn't tell for example that 100100 is two elements in two columns. It treated it as one entry in a column, and so returned error saying the number of columns in each row didn't match.

Is there some way to fix this? Thank you

Here is a link to the text file. rammftp.cira.colostate.edu/demaria/ebtrk/ebtrk_atlc.txt — Ron
– Ron, Commented Jan 28, 2014 at 12:44
read the file line-by line, using with open("file") as fd; for line in fd:... and manually fix the bad 100100. — Fredrik Pihl
– Fredrik Pihl, Commented Jan 28, 2014 at 12:46
The file, which is the one in the link above, has 10000+ lines.. Do I have to fix each problematic line manually?.. — Ron
– Ron, Commented Jan 28, 2014 at 12:49
sortof, if there is a "pattern" to the broken data then you could automate that, i.e. if there supposed to be a fixed number of columns in each row, then it's easy to find all broken rows and trying to adopt. — Fredrik Pihl
– Fredrik Pihl, Commented Jan 28, 2014 at 12:52

hochl · Accepted Answer · 2014-01-28 14:10:01Z

3

You can supply the delimiter sizes as argument. Example:

import numpy as np
import sys

with open('ebtrk_atlc.txt', 'rU') as f:
    data = np.genfromtxt(f,
                         dtype=None,
                         delimiter=[7, 10, 7, 4, 5, 6, 4, 5, 4, 4, 5, 4, 4, 3, 3, 3])
    print data

will give as output (omitting the first few lines)

('AL0188 ', 'ALBERTO   ', 80712, 1988, 41.5, 69.0, 35, 1002, -99, -99, 1012, 60, 100, 100, 50, 50)
('AL0188 ', 'ALBERTO   ', 80718, 1988, 43.0, 67.5, 35, 1002, -99, -99, 1008, 50, 100, 100, 50, 50)
('AL0188 ', 'ALBERTO   ', 80800, 1988, 45.0, 65.5, 35, 1004, -99, -99, 1008, 50, -99, -99, -99, -99)

As you see the 100100 field got separated. Of course you have to supply the correct field types and dimensions, this example just demonstrates that it is possible. For example, changing the code to

import numpy as np
import re
import sys

with open('ebtrk_atlc.txt', 'rU') as f:
    dt = "a7,a10,a7,i4,f5,f6,i4,i5,i4,i4,i5,i4,i4,i3,i3,i3"
    data = np.genfromtxt(f,
                         dtype=dt,
                         delimiter=map(int, re.split(",?[a-z]", dt[1:])),
                         autostrip=True)

will change the result to

('AL0188', 'ALBERTO', '080712', 1988, 41.5, 69.0, 35, 1002, -99, -99, 1012, 60, 100, 100, 50, 50)
('AL0188', 'ALBERTO', '080718', 1988, 43.0, 67.5, 35, 1002, -99, -99, 1008, 50, 100, 100, 50, 50)
('AL0188', 'ALBERTO', '080800', 1988, 45.0, 65.5, 35, 1004, -99, -99, 1008, 50, -99, -99, -99, -99)

Stripping away the whitespace around the strings and explicitly setting some types to float. Further documentation can be found here, check the example at the bottom.

edited Jan 28, 2014 at 14:10

answered Jan 28, 2014 at 12:57

hochl

13.1k10 gold badges58 silver badges92 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Ron Over a year ago

Thank you so much! It almost worked. Just one minor problem. When I copied your code and ran it, it said "TypeError: data type not understood".. Btw I am especially interested in keeping the opening zeros, e.g. '080712' of first line.

hochl Over a year ago

Sorry I just did not repeat the whole program, I've fixed my answer so you can copy&paste it. Does it work now? For me it reads your complete file without an error.

Ron Over a year ago

I hate to say this but still it says "TypeError: data type not understood". I have copied yours, and just replaced the spaces in front of the lines dt and data with tab...I wonder where this can go wrong..

hochl Over a year ago

Hmmm. I have just copied it from this posting and for me it works. Not sure what the problem is, bad indentation would be reported as an IndentationError in most cases. You should set your editor to show whitespace and other nonprintable tokens maybe you can find something ;-) In ViM that would be :set list.

Eelco Hoogendoorn Over a year ago

sounds like a numpy version mismatch

|

Fredrik Pihl · Accepted Answer · 2014-01-28 14:46:54Z

Old-fashion parsing is possible since the structure is nicely ordered, a bit length but it seems to do the trick.

Before:

$ awk '{print NF}' ebtrk_atlc.txt  | sort | uniq -c
     79 17
     16 18
     92 19
    494 20
    308 21
    405 22
   1769 23
    897 24
   1329 25
   5444 26
     27 27

after:

$ awk '{print NF}' log  | sort | uniq -c
   8778 27
   2082 28

code:

#!/usr/bin/env python

def chunks(l, n):
    return [l[i:i+n] for i in range(0, len(l), n)]

with open("ebtrk_atlc.txt") as fd:
    for line in fd:
        cols=line.strip().split()
        # 26 columns seems to be the target
        # after column 13, split on -
        if len(cols) < 26:
            tmp = []
            for i in cols[-13:]:
                if '-' in i:
                    for n in i.split('-'):
                        if n:
                            tmp.append('-' + n)

                elif len(i) == 6 or len(i) == 9 or len(i) == 12:
                    for n in chunks(i, 3):
                        tmp.append(n)
                elif len(i) == 8:
                    # 50100100 split in 2-3-3-fashion
                    tmp.append(i[0:1])
                    tmp.append(i[2:4])
                    tmp.append(i[5:7])
                elif len(i) == 5:
                    # 50100 split in 2-3-fashion
                    tmp.append(i[0:1])
                    tmp.append(i[2:4])
                elif len(i) == 7:
                    # 0285195 split in 3-3-fashion
                    tmp.append(i[0])
                    tmp.append(i[1:3])
                    tmp.append(i[4:6])
                elif len(i) == 11:
                    # 30120160200 split in 2-3-3-3-fashion
                    tmp.append(i[0:1])
                    tmp.append(i[2:4])
                    tmp.append(i[5:7])
                    tmp.append(i[8:10])
                elif len(i) == 10:
                    # 0180180210 split in 3-3-3-fashion
                    tmp.append(i[0])
                    tmp.append(i[1:3])
                    tmp.append(i[4:6])
                    tmp.append(i[7:9])
                else:
                    tmp.append(i)

                # one final loop to fix strings beginning with a 0
                tmp2 = []
                for i in tmp:
                    if i.startswith('0') and len(i) > 2:
                        tmp2.append(i[0])
                        tmp2.append(i[1:])
                    else:
                        tmp2.append(i)

            # rebuild list
            data = cols[0:-13] + tmp2

            print len(data), data
        else:
            print len(cols), cols

Collectives™ on Stack Overflow

Opening text file in python as an array or list of list

2 Answers 2

6 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

6 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related