Performance of File Operations in Python

Question

Is it better to read an entire file before performing operations or is it better to perform operations while reading from the file?

If I was reading the entire file first, I'd store the information line-by-line in a list and if I was reading the file and operating on the data at the same time, I would be reading line-by-line and executing my operation after a line is read.

For the sake of the discussion, let's say the file isn't obscenely large. It would be nice to hear thoughts on small files and large files and if actions would differ. Also, I presume the operations also play a role; I'm reading URL's and downloading files.

As a stylistic choice, I like to read a file all at once, like with open(filename) as file: data=file.read().split("\n"). Just because I don't like putting large pieces of code in with blocks, nor do I want to remember to close files long after I've opened them. But that's just, like, my opinion, man. — Kevin
– Kevin, Commented Jul 11, 2014 at 19:06
For the case of fetching URLs from an URL list, it simply doesn't matter. The time spent on HTTP requests will outweigh any file operations by several magnitudes. So, use the more memory efficient and readable approach: for line in f — Lukas Graf
– Lukas Graf, Commented Jul 11, 2014 at 19:08
putting elements into a list using readlines() then iterating over seems just slightly slower than just iterating over the file object in a file that has 100 lines. — Padraic Cunningham
– Padraic Cunningham, Commented Jul 11, 2014 at 19:08

Christian Thieme · Accepted Answer · 2014-07-11 20:15:11Z

1

Why don't you find out yourself for example using the timeit module

import timeit

WORDS = "/usr/share/dict/words"

def a():
    num_lines = 0
    num_chars = 0
    with open(WORDS) as f:
        lines = f.readlines()
        num_lines = len(lines)
        for line in lines:
            num_chars += len(line)
    return num_lines, num_chars


def b():
    num_lines = 0
    num_chars = 0
    with open(WORDS) as f:
        for line in f:
            num_chars += len(line)
            num_lines += 1
    return num_lines, num_chars

if __name__ == '__main__':
    print timeit.timeit("a()", setup="from __main__ import a", number=100)
    print timeit.timeit("b()", setup="from __main__ import b", number=100)

answered Jul 11, 2014 at 20:15

Christian Thieme

1,1246 silver badges6 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Mikey Over a year ago

I can certainly construct such a script and perhaps solve my single specific case but my intention was to have a discussion with people who are more knowledgeable than me about various scenarios.

Collectives™ on Stack Overflow

Performance of File Operations in Python

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related