How to deal with a "java.lang.OutOfMemoryError: Java heap space" error when reading a file? [duplicate]

Question

I have the following code for reading a file of about 300MB of size and the execution is interrumped by a "java.lang.OutOfMemoryError: Java heap space" exception:

import java.io.*;
import java.util.*;
 
class MainClass {
   static ArrayList<String> getAttribute(String str) {
      if(str==null) return null;      
      ArrayList<String> a = new ArrayList<>();
      int i = 0;
      int j = -1;
      while(i<str.length() && j<str.length()) {
         j = str.indexOf(',',i);
         if(j==-1) j = str.length();   
         a.add(str.substring(i,j));
         i = str.indexOf(',',j)+1;
      }      
      return a;
   }
   
   static ArrayList<ArrayList<String>> readFile(String in) throws IOException {
      ArrayList<ArrayList<String>> a = new ArrayList<>();
      try(FileInputStream fin = new FileInputStream(in);) {         
         BufferedReader br = new BufferedReader(new InputStreamReader(fin));
         String str = br.readLine();
         while(str!=null) {
            a.add(getAttribute(str));
            str = br.readLine(); //line 26
         }
      }
      return a;
   }   
   
   public static void main(String args[]) throws IOException
   {      
      readFile("stats1.csv"); //line 34
   }
}

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at MainClass.readFile(MainClass.java:26)
        at MainClass.main(MainClass.java:34)

What is the cause of this error? or is it just that the file is too big?

Can you add the complete stacktrace? Which line should cause the problem? — flaxel
– flaxel, Commented Sep 29, 2020 at 16:55
@flaxel OK, I updated with the stracktrace and line numbers. — gian_
– gian_, Commented Sep 29, 2020 at 17:20
Well, you are reading the whole file in memory, that will eventually blow up. Instead load and parse the file line by line. It will probably be faster and the file size suddenly doesn't matter anymore. — M. Deinum
– M. Deinum, Commented Sep 29, 2020 at 17:21
"What is the cause of this error? or is it just that the file is too big?" Well, are you able to reproduce the error using a small file? — Karl Knechtel
– Karl Knechtel, Commented Sep 29, 2020 at 17:21
@M.Deinum Thanks for the response. I was actually asked to do it like that, copying the whole file into one bidimensional ArrayList. — gian_
– gian_, Commented Sep 29, 2020 at 18:13

Abra · Accepted Answer · 2020-09-29 18:52:57Z

Look at the source code for the default constructor, that takes no arguments, of class ArrayList. It creates an array of size ten. Then look at the source code for method add() in class ArrayList. Once ten elements have been added, the array needs to be increased. It is increased by half. In other words, when you add the eleventh element, the array is increased to size 15. When you add the sixteenth element, the array is increased to 22 and so on. Once you get to big numbers, adding a single element will cause a huge increase in the array size. Hence it is recommended to use the ArrayList constructor that accepts an initial capacity argument. So maybe you could count the number of lines in the file to get an idea of what capacity your ArrayList requires. Refer to the SO question Java NIO Files count() Method for Counting the Number of Lines for one way to count the number of lines in a file.

Collectives™ on Stack Overflow

How to deal with a "java.lang.OutOfMemoryError: Java heap space" error when reading a file? [duplicate]

1 Answer 1

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Linked

Related