I learned about numpy arrays before discovering that python has the ability to work with arrays without numpy via "import array". The book "High Performance Python" describes python's array library as "Memory-efficient arrays for primitive types."
Does numpy provide similarly performant memory-efficient arrays for primitive types? Is there any use case for python's array library that isn't already covered by numpy.array? Any advice appreciated, thank you.
There is an interesting example of their difference. Pay attention to this code:
import numpy as np
from array import array
def np_method_resize(isize, n):
q = np.empty(isize)
for i in range(n):
if i >= q.shape[0]:
q.resize(q.shape[0] * 2)
q[i] = i
return q
def arr(n):
q = array('d')
for i in range(n):
q.append(i)
return q
isize = 1000
n = 10000000
%timeit -r 10 a = np_method_resize(isize, n)
%timeit -r 10 a = arr(n)
The results show that array.array is slightly faster for this specific use case. I recommend you take a look at the link below:
numpy when you want to 'grow' arrays. Either make the array full size, or use list append during the iterative process. Here we see that array append is more list like. Again if all you wan to do is store some numbers array is ok, but if you need to do some calculatios numpy is better (at least for large sizes).In [191]: import numpy as np; from array import array
In [192]: import sys
An array.array is memory efficient compared to a list, but essentially the same as numpy array.
In [193]: alist = list(range(10000))
In [194]: anArray = array('i', alist)
In [195]: sys.getsizeof(anArray)
Out[195]: 40080
In [196]: ndArray = np.array(alist)
In [197]: sys.getsizeof(ndArray)
Out[197]: 40112
In [198]: len(alist), len(anArray), len(ndArray)
Out[198]: (10000, 10000, 10000)
list stores references to objects that are stored elsewhere in memory. Those objects may 'primitives' like int, but may also be other lists, etc.
sys.getsizeof gives essentially the same size for array and ndarray, give or take a bit for size overhead. sys.getsizeof is not useful when applied to a list.
To do math a alist values we have to use a list comprehension.
In [200]: timeit [i**2 for i in alist]
5.74 ms ± 18.1 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Same for an array.array:
In [201]: timeit [i**2 for i in anArray]
6 ms ± 14.5 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
We can use a list comprehension on ndArray, but using it's own math method is much faster:
In [202]: timeit [i**2 for i in ndArray]
2.33 ms ± 21.9 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
In [203]: timeit ndArray**2
14.6 µs ± 36.8 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)
I'm a little surprised that the list comprehension on ndarray is faster; usually iteration like this on a ndarray is slower.
'math' on a list is a join or replication. Same for array.array.
In [205]: len(alist+alist)
Out[205]: 20000
In [206]: len(anArray+anArray)
Out[206]: 20000
In [207]: len(anArray*2)
Out[207]: 20000
ndarray is faster; usually iteration like this on a ndarray is slower." It's not faster, you misread the units. The listcomp is in milliseconds, the exponentiation of the ndarray is in microseconds, so the listcomp is taking over 100x longer. Unless you mean that it's faster than the listcomp on other types?
i*2 behaves as expected. I suspect i**2 requires some extra processing to ensure the result is not a long-integer.
[python] "import array"only yields 36 results. It is not used, or asked about, very often, especially compared to[numpy]. There are other packages likepandasthat are built onnumpy.[python] "import numpy"also only yields 2,033 results, while searching[numpy]yields 114,117.arraymodule provides. If you want fast numerical calculations, linear algebra, and other scientific computing capabilities, using n-dimensional arrays then you want numpy.