memory profiling: list comprehension vs. numpy array

Question

I am trying to perform memory profiling of list vs numpy arrays.

%%file memory.py

import numpy as np

@profile
def allocate():
    vector_list = [float(i) for i in range(10000)]
    np.arange(0,10000,dtype='d')

allocate()

Running memory profiler in the shell:

!python -m memory_profiler memory.py

gives the following output:

Line #    Mem usage    Increment   Line Contents
================================================
     4   39.945 MiB    0.000 MiB   @profile
     5                             def allocate():
     6   39.949 MiB    **0.004 MiB**       vector_list = [float(i) for i in range(10000)]
     7   40.039 MiB    **0.090 MiB**       np.arange(0,10000,dtype='d')

Increment in memory of line 6 vs line 7 shows that numpy array was way more expensive than a list. What am I doing wrong?

Since you're largely interested in those two, you could just check the sizes of both objects using sys.getsizeof (which should work reasonably well for a list and a np.arange object), instead of relying extensively on a memory profiling tool. — Moses Koledoye
– Moses Koledoye, Commented Jul 19, 2017 at 22:40
@MosesKoledoye yeah, but you have to have a grip on CPython internals to use sys.getsizeof correctly. For example, you would need sum(map(sys.getsizeof, vector_list)) + sys.getsizeof(vector_list) to get an accurate picture of the memory usage of vector_list. And sys.getsizeof(np.arange(0,10000)) — juanpa.arrivillaga
– juanpa.arrivillaga, Commented Jul 19, 2017 at 22:42
@MosesKoledoye in other words, sys.getsizeof does not work reasonably well, naively, with a list. If you did it with vector_list, it would be off by about 240000 bytes — juanpa.arrivillaga
– juanpa.arrivillaga, Commented Jul 19, 2017 at 22:44
@MosesKoledoye yep. Check out my answer here, although, the original question was about the memory usage of a bunch of dicts. But I go into how numpy can be extremely memory efficient, but it also demonstrates the subtleties of getting the actual memory usage of a Python container. E.g. string interning, small-int caching, etc. — juanpa.arrivillaga
– juanpa.arrivillaga, Commented Jul 19, 2017 at 22:54

AGN Gazer · Accepted Answer · 2017-07-19 22:51:07Z

3

I do not know what is memory profiler reporting - I get very different numbers from you:

Line #    Mem usage    Increment   Line Contents
================================================
     3   41.477 MiB    0.000 MiB   @profile
     4                             def allocate():
     5   41.988 MiB    0.512 MiB       vector_list = [float(i) for i in range(10000)]
     6   41.996 MiB    0.008 MiB       np.arange(0,10000,dtype='d')

I would recommend the following two links for your reading: Python memory usage of numpy arrays and Size of list in memory

I have also modified your code as follows:

import numpy as np
import sys

@profile
def allocate():
    vector_list = [float(i) for i in range(10000)]
    npvect = np.arange(0,10000,dtype='d')
    listsz = sum(map(sys.getsizeof, vector_list)) + sys.getsizeof(vector_list)
    print("numpy array size: {}\nlist size: {}".format(npvect.nbytes, listsz)) 
    print("getsizeof(numpy array): {}\n".format(sys.getsizeof(npvect))) 

allocate()

and it outputs:

numpy array size: 80000
list size: 327632
getsizeof(numpy array): 80096

edited Jul 19, 2017 at 22:51

answered Jul 19, 2017 at 22:34

AGN Gazer

8,4272 gold badges31 silver badges49 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

juanpa.arrivillaga Over a year ago

This does not correctly account for the memory required for that list. You need to use sum(map(sys.getsizeof, vector_list)) + sys.getsizeof(vector_list)

AGN Gazer Over a year ago

I have edited my answer to account for your comment. Thanks! @juanpa.arrivillaga

Collectives™ on Stack Overflow

memory profiling: list comprehension vs. numpy array

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related