Python : Numpy memory error on creating a 3d array. Whats the better way to fill a 3d array

Question

I am making a 3d array of zeros and then filling it. But, due to the size of the numpy array it runs into memory issues even with 64 gb ram. Am i doing it wrong?

X_train_one_hot shape is (47827, 30, 20000) and encInput is of shape (47827, 30, 200)

X_train_one_hot_shifted = np.zeros((X_train_one_hot.shape[0], 30, 20200))
#X_train_one_hot.shape[0] = 48000
for j in range(0, X_train_one_hot.shape[0]):
    current = np.zeros((30, 20000))
    current[0][0] = 1

    current[1:] = X_train_one_hot[j][0:29]
#     print(current.shape, encInput[i].shape)
    combined = np.concatenate((current,encInput[j]), axis=1)

    X_train_one_hot_shifted[j] = combined

Any ideas to reduce memory consumption? Another interesting thing is since the shape of X_train_one_hot is also almost same, but that does not throw any error.

EDIT : The program gets killed in the for loop with the error message :

TERM_MEMLIMIT: job killed after reaching LSF memory usage limit.

Also, most of the array is sparse since X_train_one_hot a one_hot encoding of 20000 size

Could you also share the exact memory error, and also at which point in the execution you are getting the error? I'm assuming that it is an error and not a crash. — fountainhead
– fountainhead, Commented Feb 28, 2019 at 10:52
@fountainhead - my bad. should have added it. Its a crash because of less memory. I have updated the details now — Dreams
– Dreams, Commented Feb 28, 2019 at 11:06

lightalchemist · Accepted Answer · 2019-02-28 11:07:46Z

3

Imtinan Azhar is correct. You simply do not have enough RAM to hold the array.

You have a few options.

1) You seem to have a very sparse matrix even though the size is large. So you can try to use one of the sparse matrix representation from Scipy.

If you are throwing the array into a library package such as Scikit-Learn or one of those Deep Learning libraries, this will likely not work.

2) Most DL libraries don't need you to load all your data at once. You can prepare your data in batches - create this matrix in batch and save it out to file (preferably using a sparse matrix representation). Then use a data generator to feed your algorithm, or manually load in batches of your data for your algorithm.

3) If these are all not possible, then you can try to memory map the array using Numpy's memmap. Some further examples can be found here.

4) Another option is to use dask and manually get slices of the data when necessary.

Personally, I would go with option 2, or 1 if your algorithms that take in the matrix can handle (or be modified to handle) sparse matrices.

answered Feb 28, 2019 at 11:07

lightalchemist

10.2k4 gold badges49 silver badges55 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

fountainhead Over a year ago

Doesn't numpy use virtual memory, over above the RAM capacity? Or are we saying that even virtual memory is being exceed here?

Dreams Over a year ago

ya, i think doing the second option is better.

lightalchemist Over a year ago

@fountainhead I don't quite get what you are saying. The OS always uses virtual memory addressing but memory is limited by physical RAM and swap file. Depending on the OS, it can put data in the swap file even if there is still physical RAM available. Also, a process might have a cap on max amount of RAM it can use (for e.g., set by ulimit). Then there is the case that the OS determines that even the swap file will run out of space, so it kills the process. Lastly, even if there is space available in the swap file, swapping this much data in and out of disk will probably cause disk trashing.

fountainhead Over a year ago

@lightalchemist: Mine was more of a question, than a stmt. Yes, all of those scenarios indeed exist. What prompted me to ask about virtual mem was that the answers seemed to just compare program requirements with RAM size, without even considering whether and how much virtual memory has been configured (and yes, whether and how much.per-process limit has been configured). Perhaps, the main focus of the answers were on suggesting workarounds.:-)

Imtinan Azhar · Accepted Answer · 2019-02-28 10:17:32Z

1

Lets see your X_train_one_hot_shifted.shape is (48000,30,20200) that is 28983162000 floats.

28983162000*8 gives you the memory consumption for this array in bytes. Which is 231865296000 bytes

Lets simplify this

231865296000 b

226430953.125 kb

221123.977661 mb

215.941384435 gb

You need 215Gb of RAM to fit X_train_one_hot_shifted into your RAM, i think the shape 20200 is a typo, look it up

answered Feb 28, 2019 at 10:17

Imtinan Azhar

1,77314 silver badges27 bronze badges

3 Comments

Dreams Over a year ago

Thanks for the answer. the shape is 20200 because am concatenating another array. But, the shape is similar to X_train_one_hot which is the same size, and it does not throw any error

Imtinan Azhar Over a year ago

It should throw a memory error, but regardless of it, the array is too big to be fit in your RAM

fountainhead Over a year ago

Doesn't numpy use virtual memory, over above the RAM capacity?

Collectives™ on Stack Overflow

Python : Numpy memory error on creating a 3d array. Whats the better way to fill a 3d array

2 Answers 2

4 Comments

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related