Memory Error while running python script on 4GB file

Question

I am trying to count number of words that has length between 1 and 5, file size is around 4GB end I am getting memory error.

import os 
files = os.listdir('C:/Users/rram/Desktop/') 
for file_name in files:     
    file_path = "C:/Users/rram/Desktop/"+file_name     
    f = open (file_path, 'r')    
    text = f.readlines()
    update_text = '' 
    wordcount = {}
    for line in text:         
        arr = line.split("|")
        word = arr[13]
        if 1<=len(word)<6:
            if word not in wordcount:
                wordcount[word] = 1
        else:
            wordcount[word] += 1
            update_text+= '|'.join(arr)
print (wordcount)     #print update_text
print 'closing', file_path, '\t', 'total files' , '\n\n'
f.close()

At the end i get a MemoryError on this line text = f.readlines()

Can you pelase help to optimize it.

delete this line text = f.readlines() you can iterate over the file handle — Chris_Rands
– Chris_Rands, Commented Jun 1, 2018 at 8:33
You should iterate over the lines like for line in f:. Don't overload your memory reading all of the file at once. — BcK
– BcK, Commented Jun 1, 2018 at 8:33
Sorry, indentation moved out when copy pasting. @MohamedALANI — Hari
– Hari, Commented Jun 1, 2018 at 8:44
can i use f.readline() , for faster output since it loaded into memory and performs the operation. — Hari
– Hari, Commented Jun 1, 2018 at 9:44

marc_s · Accepted Answer · 2022-02-15 05:21:18Z

3

As suggested in the comments you should read the file line by line and not the entire file.

For example :

count = 0
with open('words.txt','r') as f:
    for line in f:
        for word in line.split():
          if(1 <= len(word) <=5):
              count=count+1
print(count)

EDIT :

If you only want to count the words in 14-th column and split by | instead then :

count = 0
with open('words.txt','r') as f:
    for line in f:
        iterator = 0
        for word in line.split("|"):
            if(1 <= len(word) <=5 and iterator == 13):
                count=count+1
            iterator = iterator +1
print(count)

note that you should avoid to write this

arr = line.split("|")
word = arr[13]

since the line may contains less than 14 words, which can result in a segmentation error.

edited Feb 15, 2022 at 5:21

marc_s

760k186 gold badges1.4k silver badges1.5k bronze badges

answered Jun 1, 2018 at 8:39

Cryckx

7196 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Hari Over a year ago

In my file there are many records and in each record fields are separated by |, and i am particularly looking at the column number 14.

Daniel Over a year ago

@FlorentJousse: if you are concerned, that arr has not enough elements, use count += len(arr) >= 13 and 1 <= len(arr[13]) <= 5

Hari Over a year ago

Dear @FlorentJousse , i am sure my records have length more than 14, there are around 70 columnns and if no data is there just pipes are available.like |||||||||. Thank you very much. Is their any other advantage of Iterator int the above code?

Collectives™ on Stack Overflow

Memory Error while running python script on 4GB file

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related