MemoryError when creating a very large numpy array [duplicate]

Question

I'm trying to create a very large numpy array of zeros and then copy values from another array into the large array of zeros. I am using Pycharm and I keep getting: MemoryError even when I try and only create the array. Here is how I've tried to create the array of zeros:

import numpy as np

last_array = np.zeros((211148,211148))

I've tried increasing the memory heap in Pycharm from 750m to 1024m as per this question: https://superuser.com/questions/919204/how-can-i-increase-the-memory-heap-in-pycharm, but that doesn't seem to help.

Let me know if you'd like any further clarification. Thanks!

You have created an array that is over 100GB in size, assuming the size of int is 4 bytes — smac89
– smac89, Commented May 13, 2016 at 15:22
Oh lord, I had no idea. That is terrifying. It is a very sparse array though. Is there any way to create an empty array with values in certain positions, such as: last_array[211148][9] but everywhere else would be empty? — Andrew Earl
– Andrew Earl, Commented May 13, 2016 at 15:28
This may be helpful: stackoverflow.com/questions/1857780/… — Keozon
– Keozon, Commented May 13, 2016 at 15:35
@Smac89 Assuming he's using 64 bit Python, np.zeros will create a float64 array by default, in which case he's looking at about 356 GB. — ali_m
– ali_m, Commented May 13, 2016 at 19:17

amicitas · Accepted Answer · 2016-05-13 16:00:33Z

10

Look into using the sparse array capabilities within scipy:
scipy.sparse documentation.

There are a set of examples and tutorials on the scipy.sparse library here:
Scipy lecture notes: Sparse Matrices in SciPy

This may help you solve your memory issues, as well as make everything run faster.

To create an empty sparse array with values in certain positions as you asked in your comment:

Is there any way to create an empty array with values in certain positions, such as: last_array[211147][9] but everywhere else would be empty?

from scipy.sparse import *
values = [42]
row_ind = [211147]
col_ind = [9] 
last_array = csc_matrix((values, (row_ind, col_ind)), shape=(211148,211148))

print(last_array[211147,9])

edited May 13, 2016 at 16:00

answered May 13, 2016 at 15:39

amicitas

13.9k5 gold badges41 silver badges50 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Andrew Earl Over a year ago

Thanks a ton! I'm working on how to get a bunch of values from one array into specific locations in the sparse matrix now.

Andrew Earl Over a year ago

Here is a follow-up question if you are interested: stackoverflow.com/questions/37218550/…

Collectives™ on Stack Overflow

MemoryError when creating a very large numpy array [duplicate]

1 Answer 1

2 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Linked

Related