How can I save a big `numpy` as '*.npz' array with limited filesystem capacity?

Question

I have a numpy array which saved as an uncompressed '*npz' file is about 26 GiB as it is numpy.float32 and numpy.savez() ends with:

OSError: Failed to write to /tmp/tmpl9v3xsmf-numpy.npy: 6998400000 requested and 3456146404 written

I suppose saving it compressed may save the day, but with numpy.savez_compressed() I have also:

OSError: Failed to write to /tmp/tmp591cum2r-numpy.npy: 6998400000 requested and 3456157668 written

as numpy.savez_compressed() saves the array uncompressed first.

The obvious "use additional storage" I do not consider an answer. ;)

[EDIT]

The tag low-memory refers to disk memory, not RAM.

Can't you use a lighter format like float16, int8, uint8, etc.? — Kasravnd
– Kasravnd, Commented Feb 28, 2018 at 12:24
If you have such a big array and need that precision, that is how much it takes to store it. The only way you could really reduce it (besides generic compression) is if there are known patterns in the data, e.g. is it a sparse array, or are there repeated or derived values? If all the values have about the same exponent maybe storing only the mantissa in int16/uint16 could be enough? Also, do you know what is your file system? It may limit the size of the files that you can store. — javidcf
– javidcf, Commented Feb 28, 2018 at 12:47
@CharlesDuffy well, I suppose then you have to save it to a BytesIO object first, then compress that. Which, due to memroy demand, is probably no solution either... — swenzel
– swenzel, Commented Feb 28, 2018 at 13:36

Charles Duffy · Accepted Answer · 2018-02-28 18:07:34Z

1

With the addition of ZipFile.open(..., mode='w') in Python 3.6, you can do better:

import numpy as np
import zipfile
import io

def saveCompressed(fh, **namedict):
     with zipfile.ZipFile(fh, mode="w", compression=zipfile.ZIP_DEFLATED,
                          allowZip64=True) as zf:
         for k, v in namedict.items():
             with zf.open(k + '.npy', 'w', force_zip64=True) as buf:
                 np.lib.npyio.format.write_array(buf,
                                                 np.asanyarray(v),
                                                 allow_pickle=False)

edited Feb 28, 2018 at 18:07

answered Feb 28, 2018 at 16:18

Charles Duffy

299k43 gold badges441 silver badges496 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

abukaj Over a year ago

Looks almost exactly as implementation I am testing right now, with the exception of with zf.open(k + '.npy', mode='w', force_zip64=True) as buf:

Charles Duffy Over a year ago

Using zf.open() is the key difference, since it allows the file created inside the zip to be written incrementally (thus, with a sane ZipFile implementation, with bounded memory usage).

abukaj Over a year ago

I mean the , force_zip64=True part.

abukaj Over a year ago

Your code ends with

ValueError: Can't close the ZIP file while there is an open writing handle on it. Close the writing handle before closing the zip.

. I guess it is about the size of the array. Would you mind to include the frorce_zip64=True part in your answer?

abukaj · Accepted Answer · 2018-02-28 13:36:04Z

Note: I would be more than happy to accept a more RAM-efficient solution.

I have browsed the numpy.savez_compressed() code and decided to reimplement part of its functionality:

import numpy as np
import zipfile
import io

def saveCompressed(fh, **namedict):
     with zipfile.ZipFile(fh,
                          mode="w",
                          compression=zipfile.ZIP_DEFLATED,
                          allowZip64=True) as zf:
         for k, v in namedict.items():
             buf = io.BytesIO()
             np.lib.npyio.format.write_array(buf,
                                             np.asanyarray(v),
                                             allow_pickle=False)
             zf.writestr(k + '.npy',
                         buf.getvalue())

It causes my system to swap, but at least I am able to store my data (sham data used in the example):

>>> A = np.ones(12 * 6 * 6 * 1 * 6 * 6 * 10000* 5* 9, dtype=np.float32)
>>> saveCompressed(open('test.npz', 'wb'), A=A)
>>> A = np.load('test.npz')['A']
>>> A.shape
(6998400000,)
>>> (A == 1).all()
True

Collectives™ on Stack Overflow

How can I save a big `numpy` as '*.npz' array with limited filesystem capacity?

2 Answers 2

4 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related