Pythonic String Testing

Question

For my Information Retrieval class I have to make an index of terms from a group of files. Valid terms contain an alphabetical character, so to test I just made a simple function and use an if/then control statement. Thus far I have:

ALPHA = ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 
'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z']

def content_test(term):
    for a in ALPHA:
        if a in term:
            return True
    return False

class FileRead():

    def __init__(self, filename):
        f = open(filename, 'r')
        content = f.read()
        self.terms = content.split()

    def clean(self):
        for term in self.terms:
            if content_test(term) is False:
                try:
                    terms.remove(term)
                except:
                    pass

Now this all works fine (I think...) however I've been trying to learn more higher level python and I can't help but think that there is a more pythonic way of checking term validity (maybe using map(), or a lambda function?).

Am I correct or am I just overthinking it?

Small cleanup that you might find handy. import string; ALPHA = string.lowercase. — istruble
– istruble, Commented Feb 16, 2012 at 18:03

kindall · Accepted Answer · 2012-02-16 18:09:00Z

2

You can start by simplifying content_test():

def content_test(term):
    return any(c.isalpha() for c in term)

In fact, that's simple enough that you don't really need a separate function for it anymore.

What I'd do in this case is write a generator that yields only valid terms from the file. Then just convert that to a list using the list() constructor. This way you can read just a line at a time, which will save you a good bit of memory if the files are large.

def read_valid_terms(filename):
    with open(filename) as f:
        for line in f:
            for term in line.split():
                if any(c.isalpha() for c in term):
                    yield term

terms = list(read_valid_terms("terms.txt"))

Or if you are just going to iterate over the terms anyway, and only once, then just do that directly rather than making a list:

for term in read_valid_terms("terms.txt"):
    print term,
print

edited Feb 16, 2012 at 18:09

answered Feb 16, 2012 at 17:53

kindall

185k36 gold badges291 silver badges321 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

phihag Over a year ago

Instead of using the various quirks of the print statement, I'd rather suggest the print function in the last two lines.

kindall Over a year ago

Eh, just a quickie demo, what's inside the for loop is immaterial to the example.

bgporter · Accepted Answer · 2012-02-16 17:49:57Z

1

In Python, string objects already contain a method that does that for you:

>>> "abc".isalpha()
True
>>> "abc22".isalpha()
False

answered Feb 16, 2012 at 17:49

bgporter

36.9k8 gold badges65 silver badges67 bronze badges

4 Comments

cha0site Over a year ago

We need islower here too =D

pdeuchler Over a year ago

The problem is valid terms can contain non alpha chars (just not exclusively)

bgporter Over a year ago

Ah, that wasn't clear. So you mean that it just needs to contain a single alphabetic character someplace in the string?

pdeuchler Over a year ago

Yes, exactly (comment length)

Gandaro · Accepted Answer · 2012-02-16 18:05:07Z

1

While you could use a regular expression, a pythonic way would be to use any:

import string
def content_test(term):
    return any((c in string.ascii_lowercase) for c in term)

If you also want to allow upper-case and locale-dependent characters, you can use str.isalpha.

A couple of additional notes:

FileRead should inherit from object, to make sure it's a new-style class.
Instead of writing if content_test(term) is False:, you can simply write if not content_test(term):.
clean can be written a lot, ahem, cleaner, by using filter:

def clean(self):
    self.terms = filter(content_test, self.terms)

You're not closing the file f, and may therefore leak the handle. Use the with statement to automatically close it, like this:

with open(filename, 'r') as f:
    content = f.read()
    self.terms = content.split()

edited Feb 16, 2012 at 18:05

Gandaro

3,4531 gold badge19 silver badges19 bronze badges

answered Feb 16, 2012 at 17:50

phihag

289k75 gold badges475 silver badges489 bronze badges

1 Comment

pdeuchler Over a year ago

I'm selecting kindall's answer as the correct one, but your's still helped a lot, appreciate it

campos.ddc · Accepted Answer · 2012-02-16 19:59:41Z

0

Using regular expressions:

import re

# Match any number of non-whitespace characters, with an alpha char in it.
terms = re.findall('\S*[a-zA-Z]\S*', content)

answered Feb 16, 2012 at 19:59

campos.ddc

7715 silver badges12 bronze badges

Collectives™ on Stack Overflow

Pythonic String Testing

4 Answers 4

2 Comments

4 Comments

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

2 Comments

4 Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related