getting Java OutOfMemoryError: Java heap space error that I can't debug

Question

I am struggling to figure out what's causing this OutofMemory Error. Making more memory available isn't the solution, because my system doesn't have enough memory. Instead I have to figure out a way of re-writing my code.

I've simplified my code to try to isolate the error. Please take a look at the following:

File[] files = new File(args[0]).listFiles();

int filecnt = 0;

LinkedList<String> urls = new LinkedList<String>();

for (File f : files) {
    if (filecnt > 10) {
        System.exit(1);
    }

    System.out.println("Doing File " + filecnt + " of " + files.length + " :" +                f.getName());

    filecnt++;
    FileReader inputStream = null;
    StringBuilder builder = new StringBuilder();

    try {
        inputStream = new FileReader(f);
        int c;
        char d;

        while ((c = inputStream.read()) != -1) {
            d = (char)c;
            builder.append(d);
        }
    }

    finally {
        if (inputStream != null) {
            inputStream.close();
        }
    }   

    inputStream.close();

    String mystring = builder.toString();
    String temp[] = mystring.split("\\|NEWandrewLINE\\|");

    for (String s : temp) {
        String temp2[] = s.split("\\|NEWandrewTAB\\|");
        if (temp2.length == 22) { 
            urls.add(temp2[7].trim());
        }
    }
}

I know this code is probably pretty confusing :) I have loads of text files in the directory that is specified in args[0]. These text files were created by me. I used |NEWandrewLINE| to indicate a new row in the text file, and |NEWandrewTAB| to indicate a new column. In this code snippet, I am trying to access the URL of each stored row (which is in the 8th column of each row). So, I read in the whole text file. String split on |NEWandrewLINE| and then string split again on the substrings on |NEWandrewTAB|. I add the URL to the LinkedList (called "urls") with the line: urls.add(temp2[7].trim())

Now, the output of running this code is:

Doing File 0 of 973 :results1322453406319.txt
Doing File 1 of 973 :results1322464193519.txt
Doing File 2 of 973 :results1322337493419.txt
Doing File 3 of 973 :results1322347332053.txt
Doing File 4 of 973 :results1322330379488.txt
Doing File 5 of 973 :results1322369464720.txt
Doing File 6 of 973 :results1322379574296.txt
Doing File 7 of 973 :results1322346981999.txt
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:2882)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:572)
at java.lang.StringBuilder.append(StringBuilder.java:203)
at Twitter.main(Twitter.java:86)

Where main line 86 relates to the line builder.append(d); in this example.

But the thing I don't understand is that if I comment out the line urls.add(temp2[7].trim()); I don't get any error. So the error seems to be caused by the linkedlist "urls" overfilling. But why then does the reported error relate to the StringBuilder?

Is there any reason why you are reading the files one byte at a time? Especially since you say they are text files anyway, so it may be a bit more efficient to read in larger chunks, e.g. line by line or something like that. — ivantod
– ivantod, Commented Jan 5, 2012 at 13:26
Also, out of curiosity, what heap size are you using, and how big are those files? — Viruzzo
– Viruzzo, Commented Jan 5, 2012 at 13:31
ivantod: there was originally a reason why i was reading in one byte at a time :) but i can't remember now, and there doesn't need to be now. viruzzo: the default heap size (not sure what that is) but if i increase to 2.5GB i still get the error, but after more files have been processed. the files range from between 1 MB to 85 MB, and there are nearly 1000 of them — Andrew
– Andrew, Commented Jan 5, 2012 at 13:58

jarnbjo · Accepted Answer · 2012-01-05 14:25:27Z

4

Try to replace urls.add(temp2[7].trim()); with urls.add(new String(temp2[7].trim()));.

I suppose that your problem is that you are in fact storing the entire file content and not just the extracted URL field in your urls list, although that's not really obvious. It is actually an implementation specific issue with the String class, but usually String#split and String#trim return new String objects, which contain the same internal char array as the original string and only differs in their offset and length fields. Using the new String(String) constructor makes sure that you only keep the relevant part of the original data.

answered Jan 5, 2012 at 14:25

jarnbjo

34.4k7 gold badges75 silver badges95 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Andrew Over a year ago

that worked! thanks. i'm also going to re-write the way i read the files in, because the consensus is that i'm doing it very inefficiently, but many thanks for spotting that

Peter Lawrey · Accepted Answer · 2012-01-05 13:28:04Z

1

The linked list is using more memory each time you add a string. This means you can be left it not enough memory to build your StringBuilder.

The way to avoid this issue to write the results to a file instead of to a List as you don't appear to have enough memory to keep the List in memory.

answered Jan 5, 2012 at 13:28

Peter Lawrey

535k83 gold badges770 silver badges1.2k bronze badges

1 Comment

Andrew Over a year ago

yeah, i guess so. there's more to my code, that i haven't included here. i also store the content of the webpage, not just the URL, and then i parse the content. but maybe i need to store the content and URL in a file, and then open that file and process it

Sam · Accepted Answer · 2012-01-05 13:30:41Z

1

Because this is

out of memory and not out of heap
you have LOTS of small temporary objects

I would suggest you give your JVM a -X maximum heap size limit that fits in your RAM.

To use less memory I would use a buffered reader to pull in the entire line and save on the temporary object creation.

answered Jan 5, 2012 at 13:30

Sam

2,96919 silver badges18 bronze badges

Comments

Kai · Accepted Answer · 2012-01-05 13:31:16Z

1

The simple answer is: you should not load all the URLs from the text files into memory. You are surely doing this because you want to process them in a next step. So instead of adding them to a List in memory do the next step (maybe storing in a database or check if it is reachable) and forget that URL.

answered Jan 5, 2012 at 13:31

Kai

39.8k14 gold badges92 silver badges106 bronze badges

1 Comment

Andrew Over a year ago

you're right: there's more to my code which i've not included here. i read in page content too, and the URL, and i need to process it in the next step. i think what you suggest will work, so i'll rewrite my code this way

Joeri Hendrickx · Accepted Answer · 2012-01-05 13:32:43Z

1

How many URLS do you have? Looks like you're just storing more of them than you can handle.

As far as I can see, the linked list is the only object that is not scoped inside the loop, so cannot be collected.

For an OOM error, it doesn't really matter where it is thrown.

To check this properly, use a profiler (look at JVisualVM for a free one, and you probably already have it). You'll see which objects are in the heap. You can also have the JVM dump its memory into a file when it crashes, then analyse that file with visualvm. You should see that one thing is grabbing all of your memory. I'm suspecting it's all the URLs.

answered Jan 5, 2012 at 13:32

Joeri Hendrickx

17.5k4 gold badges43 silver badges53 bronze badges

Comments

Araejay · Accepted Answer · 2012-01-05 14:11:39Z

There are several experts in here already, so, I'l be brief to the problems:

Inappropriate use of String Builder:

StringBuilder builder = new StringBuilder();

try {
    inputStream = new FileReader(f);
    int c;
    char d;

    while ((c = inputStream.read()) != -1) {
        d = (char)c;
        builder.append(d);
    }
}

Java is beautiful when you process small amounts of data at a time, remember the garbage collector.

Instead, I would recommend that you read the file (Text file) 1 line at a time, process the line, and move on, never create a huge memory ball of StringBuilder just to get a String,

Immagine of your text file is 1 GB in size, you are done mate.

Add the real process while reading the file (as in item #1)
You dont need to close InputStream again, the code in finally block is good enough.

regards

light_303 · Accepted Answer · 2012-01-05 13:23:00Z

0

if the linkedlist eats your memory every command which allocates memory afterwards may fail with an OOM error. So this looks like your problem.

answered Jan 5, 2012 at 13:23

light_303

2,1313 gold badges20 silver badges37 bronze badges

1 Comment

Sean Owen Over a year ago

The OOM does not occur at LinkedList.add()

Sean Owen · Accepted Answer · 2012-01-05 13:28:23Z

0

You're reading the files into memory. At least one file is simply too big to fit into the default JVM heap. You can allow it use a lot more memory with an arg like -Xmx1g on the command line after java.

By the way this is really inefficient to read a file one character at a time!

answered Jan 5, 2012 at 13:28

Sean Owen

67.1k23 gold badges144 silver badges175 bronze badges

1 Comment

Andrew Over a year ago

yes, i think i need to re-write the way i read in the file. by the way, i used -Xmx3g and i still get an error after 30 or so files

Gur Kamal Singh Badal · Accepted Answer · 2012-01-05 13:34:06Z

Instead of trying to split the string (which basically creates an array of substrings based on the split) - thereby using more than double the memory each time you use the slpit, you should try to do regex based matching of the start and end patterns, extract individual sub-strings one by one and then extract the URL from that.

Also, if your file is large, I would suggest that you not even load all of that into memory at once ... stream its contents to a buffer (of manageable size) and use the pattern based search on that (and keep removing / adding more to the buffer as you progress through the file contents).

The implementation will slow down the program a bit but will use a considerably lesser amount of memory.

Ashwinee K Jha · Accepted Answer · 2012-01-05 13:36:33Z

0

One major problem in your code is that you read whole file into a string builder, then convert it into string and then split it into smaller parts. So if file size is large you will get into trouble. As suggested by others process the file line by line as that should save a lot of memory.

Also you should check what is the size of your list after processing each file. If the size is very large you may want to use different approach or increase the memory for your process via -Xmx option.

answered Jan 5, 2012 at 13:36

Ashwinee K Jha

9,3572 gold badges27 silver badges20 bronze badges

2 Comments

Andrew Over a year ago

the weird thing is that if i only allow one URL to be added to the list for each file, i still get the Error after only 7 files... increasing the memory doesn't help, because i still get the error, but after more files have been processed. i'll try writing the code so that i don't need to store the URLs

Ashwinee K Jha Over a year ago

Before re-writing code it will be best to take a memory dump and see what is consuming memory. If list of URL's is not too big then processing the files itself may be consuming memory.

Collectives™ on Stack Overflow

getting Java OutOfMemoryError: Java heap space error that I can't debug

10 Answers 10

1 Comment

1 Comment

Comments

1 Comment

Comments

Comments

1 Comment

1 Comment

Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

10 Answers 10

1 Comment

1 Comment

Comments

1 Comment

Comments

Comments

1 Comment

1 Comment

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related