Help parsing text file in python

Question

Really been struggling with this one for some time now, i have many text files with a specific format from which i need to extract all the data and file into different fields of a database. The struggle is tweaking the parameters for parsing, ensuring i get all the info correctly.

the format is shown below:

WHITESPACE HERE of unknown length.
K       PA   DETAILS 
2 4565434   i need this sentace as one DB record
2 4456788   and this one 
5 4879870   as well as this one, content will vary! 

X Max - there sometimes is a line beginning with 'Max' here which i don't need
There is a Line here that i do not need!
WHITESPACE HERE of unknown length.

The tough parts were 1) Getting rid of whitespace, and 2)defining the fields from each other, see my best attempt, below:

dict = {}
    XX = (open("XX.txt", "r")).readlines()

    for line in XX:
            if line.isspace():
            pass
        elif line.startswith('There is'):
            pass
        elif line.startswith('Max', 2):
            pass
        elif line.startswith('K'):
            pass
        else:
            for word in line.split():
                if word.startswith('4'):                    
                    tmp_PA = word
                elif word == "1" or word == "2" or word == "3" or word == "4" or word == "5":
                    tmp_K = word
                else:
                    tmp_DETAILS = word
            cu.execute('''INSERT INTO bugInfo2 (pa, k, details) VALUES(?,?,?)''',(tmp_PA,tmp_K,tmp_DETAILS))

At the minute, i can pull the K & PA fields no problem using this, however my DETAILS is only pulling one word, i need the entire sentance, or at least 25 chars of it.

Thanks very much for reading and I hope you can help! :)

K

word == "1" or word == "2" or word == "3" or word == "4" or word == "5": could be rewritten as word in ['1', '2', '3', '4', '5']. — rplnt
– rplnt, Commented Sep 27, 2011 at 9:11
basically, you have a line which you want to extract the first two numbers from, you can either use regular expression matching, or just put the first item from the splitting in tmp_PA, the second in tmp_K and concatenate the rest in tmp_DETAILS. (btw, I think you will get into trouble with lines like 4 467493 ...) — steabert
– steabert, Commented Sep 27, 2011 at 9:12

Jan Hudec · Accepted Answer · 2011-09-27 09:29:53Z

2

You are splitting the whole line into words. You need to split into first word, second word and the rest. Like line.split(None, 2).

It would probably use regular expressions. And use the oposite logic, that is if it starts with number 1 through 5, use it, otherwise pass. Like:

pattern = re.compile(r'([12345])\s+\(d+)\s+\(.*\S)')
f = open('XX.txt', 'r') # No calling readlines; lazy iteration is better
for line in f:
    m = pattern.match(line)
    if m:
        cu.execute('''INSERT INTO bugInfo2 (pa, k, details) VALUES(?,?,?)''',
            (m.group(2), m.group(1), m.group(3)))

Oh, and of course, you should be using prepared statement. Parsing SQL is orders of magnitude slower than executing it.

answered Sep 27, 2011 at 9:29

Jan Hudec

76.8k13 gold badges138 silver badges190 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

user948581 · Accepted Answer · 2011-09-27 09:13:59Z

If I understand correctly your file format, you can try this script

filename = 'bug.txt'
f = file(filename,'r')

foundHeaders = False

records = []

for rawline in f:
    line = rawline.strip()

    if not foundHeaders:
        tokens = line.split()
        if tokens == ['K','PA','DETAILS']:
            foundHeaders = True
        continue

    else:
        tokens = line.split(None,2)
        if len(tokens) != 3:
            break

        try:
            K = int(tokens[0])
            PA = int(tokens[1])
        except ValueError:
            break

        records.append((K,PA,tokens[2]))


f.close()

for r in records:
    print r # replace this by your DB insertion code

This will start reading the records when it encounters the header line, and stop as soon as the format of the line is no longer (K,PA,description).

Hope this helps.

thebwt · Accepted Answer · 2011-09-27 10:08:20Z

0

Here is my attempt using re

import re

stuff = open("source", "r").readlines()

whitey = re.compile(r"^[\s]+$")
header = re.compile(r"K       PA   DETAILS")
juicy_info = re.compile(r"^(?P<first>[\d])\s(?P<second>[\d]+)\s(?P<third>.+)$")

for line in stuff:
    if whitey.match(line):
        pass
    elif header.match(line):
        pass
    elif juicy_info.match(line):
        result = juicy_info.search(line)
        print result.group('third')
        print result.group('second')
        print result.group('first')

Using re I can pull the data out and manipulate it on a whim. If you only need the juicy info lines, you can actually take out all the other checks, making this a REALLY concise script.

import re

stuff = open("source", "r").readlines()

#create a regular expression using subpatterns. 
#'first, 'second' and 'third' are our own tags ,
# we could call them Adam, Betty, etc.
juicy_info = re.compile(r"^(?P<first>[\d])\s(?P<second>[\d]+)\s(?P<third>.+)$")

for line in stuff:
    result = juicy_info.search(line)
    if result:#do stuff with data here just use the tag we declared earlier.            
        print result.group('third')
        print result.group('second')
        print result.group('first')

answered Sep 27, 2011 at 10:08

thebwt

1262 bronze badges

1 Comment

thebwt Over a year ago

doh, and people ninja'd me while I was scripting.

Jan Hudec · Accepted Answer · 2011-09-27 15:57:16Z

0

import re


reg = re.compile('K[ \t]+PA[ \t]+DETAILS[ \t]*\r?\n'\
                 + 3*'([1-5])[ \t]+(\d+)[ \t]*([^\r\n]+?)[ \t]*\r?\n')

with open('XX.txt') as f:

    mat = reg.search(f.read())

    for tripl in ((2,1,3),(5,4,6),(8,7,9)):

        cu.execute('''INSERT INTO bugInfo2 (pa, k, details) VALUES(?,?,?)''',
                   mat.group(*tripl)

I prefer to use [ \t] instead of \s because \s matches the following characters:
blank , '\f', '\n', '\r', '\t', '\v'
and I don't see any reason to use a symbol representing more that what is to be matched, with risks to match erratic newlines at places where they shouldn't be

Edit

It may be sufficient to do:

import re

reg = re.compile(r'^([1-5])[ \t]+(\d+)[ \t]*([^\r\n]+?)[ \t]*$',re.MULTILINE)

with open('XX.txt') as f:
    for mat in reg.finditer(f.read()):
        cu.execute('''INSERT INTO bugInfo2 (pa, k, details) VALUES(?,?,?)''',
                   mat.group(2,1,3)

edited Sep 27, 2011 at 15:57

Jan Hudec

76.8k13 gold badges138 silver badges190 bronze badges

answered Sep 27, 2011 at 11:01

eyquem

27.7k7 gold badges43 silver badges46 bronze badges

1 Comment

Jan Hudec Over a year ago

I don't think the OP meant there is fixed number of entries. So your original solution is wrong. The updated one should work.

Collectives™ on Stack Overflow

Help parsing text file in python

4 Answers 4

Comments

Comments

1 Comment

Edit

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Comments

Comments

1 Comment

Edit

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related