1

I am currently working with a 2d numpy object array filled with collections.counter objects Each counter is basically a histogram.

  • Keys are always from a limited set of integers eg between 0 and 1500
  • number of items in each counter is variable, most are small but some have every key

This all works fine for my needs with smaller datasets but with a dataset around the 500 million cells mark the memory use is around 120Gb which is a little high.

Interestingly numpy.save writes it out to a 4gb file which makes me think there is something better i can be doing.

Any suggestions on how i can reduce my memory usage.

I considered a 3d array but because of the amount of empty counts it would have to hold it required even more memory.

I make lots of use of counter.update in constructing the array so any method needs a quick/neat way of getting similar functionality.

The access after the data is created isnt a big issue as long for each cell i can get the value for each key - no need for a dictionaries indexing.

Below is a very simplified example that produces a small dataset that is roughly analogous to what ive described above. My code would have a skew further towards less keys per counter and higher counts per key

def counterArray_init(v):
    return collections.Counter([v])

e = np.random.random_integers(0,1500,[10,10])
row_len, col_len = e.shape
counterArray = np.zeros([row_len,col_len], dtype= object)
vinit = np.vectorize(counterArray_init)
counterArray[:,:] = vinit(e)
for row in xrange(1,row_len):
    for col in xrange(0,col_len):
       counterArray[row,col].update(counterArray[row - 1,col])
return counterArray

Thanks

Edit: I have realised that in my smaller counters the keys used fall within a small range. The random example code above is not a good example of this behaviour. As a result i am investigating using an object array filled with different length int arrays and a separate array that stores the minimum key value for each of those int arrays. It seems like an odd solution but initial testing looks like its using only about 20% of the memory used by the counter method.

Improve this question
2
  • The 3D array would be just a basic numpy array with dtype=int, right? No Counter objects? Commented Mar 7, 2012 at 7:08
  • Thats right just a 3d numpy array of ints. Commented Mar 8, 2012 at 2:05

3 Answers 3

Reset to default
1

If there are lots of empty counts,a sparse matrix representation may be a good fit, where the memory use is proportional to the number of non-empty elements in the array. SciPy has decent support for what it sounds like you're looking at: scipy.sparse

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

Will look into the sparse matrix although if im reading rightly they are limited to 2d so i will need to flatten 1 dimension. Also have you got any pointers on how to do the equivalent of counter.update
1

I would definitively use a 3D array, since keys are integers. What is the maximum count for a specific item? If it is below 255, you could also change the datatype of the array to be 8 bits.

Improve this answer

1 Comment

good idea but counts in the 100k + range are possible. With that limitation imposed a normal 3d array isnt going to work with a reasonable memory limit.
0

A quick test shows storing tuples instead of Counter objects would save you only about 20%. (might be worth double checking in your use case). And if storing as plain int array is even less efficient, then there are few other choices.

Sparse arrays are a good space-saver, but don't offer the same boadcasting as a normal array, they are usually just used to create or store data, then converted to normal arrays for computations. If you walk your indicies with regular python loops, then sparse arrays might be a good solution.

numpy.save must be doing some kind of compression. Compression which is not likely to be useful on data in active use in memory. Are you using pyTables or h5py? How are you currently managing 120G of data - Virtual Memory?

Improve this answer

4 Comments

was that 20% saving with a list of tuples in the object array?
Im not expecting to be as compact as numpy.save. I have access to a machine with 500G of ram but i would like the solution to scale to bigger arrays or move to lesser machines. pytables is where i will look if i dont get significant improvments with a different data structure.
@Art All i did was compare the size of a python tuple to a Counter with the same number of elements. I used this: code.activestate.com/recipes/546530 I figured it might be easy to convert tuples to Collections for processing and then back to tuples again for storage. Thereby saving the storage of all the extra overhead in the more complex object. Of course your mileage will vary with the number of elements stored per object.
You wont find a datastructure that is substantially more efficient. But if a dense 3d array representation isn't that much bigger, storing it as a chuncked and compressed hdf5 array is probably the way to go. On a more fundamental level; I doubt this massive array is a goal in itself; I imagine it will be reduced in subsequent stages; can you not fuse those program stages, and only compute parts of the required datastructure in a lazy fashion? Im really surprised about the numpy.save performance by the way. Does that actually load back correctly, or is that a bug with Counter?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.