Python: Getting Memory Error for large CSV files

Question

I have the following code snippet and I get the memory error on the last line rows = list(reader)

for file in fileList:
fileName, fileExtension = os.path.splitext(file)
if fileExtension == ".csv":
    with open(path + '\\' + file, "rU") as f:
        reader = csv.reader(f, delimiter=',', dialect="excel")
        rows = list(reader)

Is there any other approach I can use?

Yes. Don't create a giant list out of your csv file. Just iterate over the reader and it will lazily produce each line. — juanpa.arrivillaga
– juanpa.arrivillaga, Commented Feb 13, 2019 at 18:21
@juanpa.arrivillaga I tried running a for loop on the readerto process every row and even that didn't help. — Adarsh Ravi
– Adarsh Ravi, Commented Feb 13, 2019 at 18:28
@AdarshRavi i would suggest dumping it in a sqlite db and then just querying whatever you need so you don't have to load the whole thing in at once — SuperStew
– SuperStew, Commented Feb 13, 2019 at 18:37

blhsing · Accepted Answer · 2019-02-13 19:41:27Z

2

Since you've now stated in the comments that you simply want to fix the formatting of the rows, you definitely don't need all the rows at once. You should iterate through the csv reader one row at a time, fix the formatting of the row, write the row to another csv file, and then move on to the next row:

with open(path + '\\' + file, "rU") as f, open(path + '\\' + file + '.fixed', "w") as o:
    reader = csv.reader(f, delimiter=',', dialect="excel")
    writer = csv.writer(o, dialect='excel')
    for row in reader:
        # fix the formatting of the row here
        writer.writerow(row)

answered Feb 13, 2019 at 19:41

blhsing

109k9 gold badges88 silver badges132 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python: Getting Memory Error for large CSV files

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related