Python NLTK extract sentence containing a keyword

Question

My objective is to extract sentences from a text file that contain any word that is in my list of keywords. My script cleans up the text file and uses NLTK to tokenize the sentences and remove stopwords. That part of the script works ok and produces output that looks correct ['affirming updated 2020 range guidance long-term earnings dividend growth outlooks provided earlier month', 'finally look forward increasing engagement existing prospective investors months come', 'turn'] The script that I wrote to extract sentences containing a keyword does not work the way I want. It extracts the keywords but not the sentences in which they occur. The output looks like this; [' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', 'impact', 'zone']

    fileinC=nltk.sent_tokenize(fileinB)
    fileinD=[]
    for sent in fileinC:
        fileinD.append(' '.join(w for w in word_tokenize(sent) if w not in allinstops))
    fileinE=[sent.replace('\n', " ") for sent in fileinD]

    #extract sentences containing keywords
    fileinF=[]
        for sent in fileinE:
    fileinF.append(' '.join(w for w in word_tokenize(sent) if w  in keywords))

thorntonc · Accepted Answer · 2020-08-16 16:25:01Z

1

It is likely that the conditional append in your last line causes the issue, it is more intuitive to break it down into smaller steps like so:

fileinF = []
for sent in fileinE:
    # tokenize and lowercase tokens of the sentence
    tokenized_sent = [word.lower() for word in word_tokenize(sent)]
    # if any item in the tokenized sentence is a keyword, append the original sentence
    if any(keyw in tokenized_sent for keyw in keywords):
        fileinF.append(sent)

edited Aug 16, 2020 at 16:25

answered Aug 12, 2020 at 23:38

thorntonc

2,1261 gold badge10 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python NLTK extract sentence containing a keyword

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related