Avoid the exception java.lang.OutOfMemoryError without growing the heap space?

Question

I met a problem with the Java heap space in which I try to group the consecutive elements of one array in order to create a matrix for computing his transposed. I have a lot of values in the array (26726400) and I try to have buckets of size 29. But when I tested the following code, I get the exception java.lang.OutOfMemoryError: Java heap space

val arr = new Array[Int](256 * 3600 * 29)
    arr: Array[Int] = Array(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,...
scala> arr.grouped(29).toArray
java.lang.OutOfMemoryError: Java heap space

My purpose is to transpose the matrix. If I run sbt -mem 2048, this code works but is it an another way to do this task without growing the heap space ?

som-snytt · Accepted Answer · 2015-10-26 22:02:14Z

1

This may not save much memory, though it is surely more efficient than grouped, which does a couple of copies between buffers internally.

scala> val arr = new Array[Int](256 * 3600 * 29)
arr: Array[Int] = Array(0, 0, 0,...

scala> Array.tabulate(256 * 3600, 29)((i,j) => arr(i * 29 + j))
res0: Array[Array[Int]] = Array(Array(0, 0, 0,...

It's noticeably faster in my scientific trial.

You could also use 1-dim tabulate, allocate Array.ofDim(29) and Array.copy.

answered Oct 26, 2015 at 22:02

som-snytt

39.6k2 gold badges49 silver badges133 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Haspemulator · Accepted Answer · 2015-10-26 21:13:52Z

0

Well, the default memory for a JVM instance on machines with >1Gb of RAM is RAM/4. So, add more memory to your computer, and you won't have to pass that parameter to sbt.

Joking aside, you have at least 3 copies of the data here. First is the original arr instance, then the result of grouped operation, then the result of toArray call. And it could even be more, I'm not sure about the implicit conversion to ArrayOps, which is required by calling the grouped method (it's not defined on the Array class, actually).

Given your data size and type, one copy takes ~101Mb of memory, excluding any overhead associated with storage. To solve the problem, reduce the amount of copies you make. For example, I don't really understand why you need the last toArray call.

As a side note, if it's not a homework, consider using some existing libraries for matrix operations, like jBLAS.

answered Oct 26, 2015 at 21:13

Haspemulator

11.4k9 gold badges55 silver badges78 bronze badges

1 Comment

alifirat Over a year ago

I call toArray because I need to apply the function transpose on the 2D array.

Collectives™ on Stack Overflow

Avoid the exception java.lang.OutOfMemoryError without growing the heap space?

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related