is there any efficient way to store numpy array

Question

i prepared an image dataset with Numpy array and i need to store them as efficient as possible

ill give you an example

for 6 image its like 2.2 MB

and after using Numpy compressed format its over 7.5 MB

this is what i come up with

 np.savez_compressed("./img.npz",images) # array of images

any way to improve that

Using.npz is also fine to save compressed numpy arrays, but I'd recommend you to take a look once at parquet file format. It compresses very well and data read/write speed is also high and memory footprint is less compared to other several file formats. — ThePyGuy
– ThePyGuy, Commented Aug 16, 2021 at 5:48

Elguja Lomsadze · Accepted Answer · 2021-08-16 06:02:19Z

1

Try to use numpy's savetxt method

from numpy import savetxt

Then you can save it as a CSV file (file might be large in size but efficient to load next) or you can do it as a simple ".txt" file

Example:

x = y = z = np.arange(0.0,5.0,1.0)
np.savetxt('test.csv', x, delimiter=',') # CSV
np.savetxt('test.txt', x, delimiter=',') # TXT

You can choose your own delimiter also.

answered Aug 16, 2021 at 6:02

Elguja Lomsadze

1282 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

if save it as txt or csv can i load it directly to numpy array?

CSV is great for its readability and portability, but it is not more efficient than NPY or PNG.