Reading a large file in python

To get a rough idea of where the memory is going, you can use the gc.get_objects function. Wrap your above code in a make_graph() function (this is best practice anyway), and then wrap the call to this function with a KeyboardInterrupt exception handler which prints out the gc data to a file.

def main():
    try:
        make_graph()
    except KeyboardInterrupt:
        write_gc()

def write_gc():
    from os.path import exists
    fname = 'gc.log.%i'
    i = 0
    while exists(fname % i):
        i += 1
    fname = fname % i
    with open(fname, 'w') as f:
        from pprint import pformat
        from gc import get_objects
        f.write(pformat(get_objects())


if __name__ == '__main__':
    main()

Now whenever you ctrl+c your program, you'll get a new gc.log. Given a few samples you should be able to see the memory issue.

There are a few things you can do:

1. Run your code on a subset of data. Measure time required. Extrapolate to the full size of your data. That will give you an estimate how long it will run.

   counter = 0 with open("final_edge_list.txt","r") as f: for line in f: counter += 1 if counter == 200000: break try: ...

   On 1M lines it runs ~8 sec on my machine, so for 2.2Gb file with about 100M lines it suppose to run ~15 min. Though, once you get over you available memory, it won't hold anymore.

2. Your graph seems symmetric

   graph[src][destination] = weight
   graph[destination][src] = weight
   

   In your graph processing code use symmetry of graph, reduce memory usage by half.

3. Run profilers on you code using subset of the data, see what happens there. Simplest would be to run

   python -m cProfile --sort cumulative youprogram.py
   

   There is a good article on speed and memory profilers: http://www.huyng.com/posts/python-performance-analysis/

Python's numeric types use quite a lot of memory compared to other programming languages. For my setting it appears to be 24 bytes for each number:

>>> import sys
>>> sys.getsizeof(int())
24
>>> sys.getsizeof(float())
24

Given you have hundreds of millions of lines in that 2.2 GB input file the reported memory consumption should not come unexpected.

To add another thing, some versions of the Python interpreter (including CPython 2.6) are known for keeping so called free lists for allocation performance, especially for objects of type int and float. Once allocated, this memory will not be returned to the operating system until your process terminates. Also have a look at this question I posted when I first discovered this issue:

• Python: garbage collection fails?

Suggestions to work around this include:

• use a subprocess to do the memory hungry computation, e.g., based on the multiprocessing module
• use a library that implements the functionality in C, e.g., numpy, pandas
• use another interpreter, e.g., PyPy

• You don't need graph to be defaultdict(dict), user dict instead; graph[src, destination] = weight and graph[destination, src] = weight will do. Or only one of them.
• To reduce memory usage, try store resulting dataset in scipy.sparse matrix, it consumes less memory and might be compressed.
• What do you plan to do with your nodes list afterwards?