java outOfMemoryError with stringbuilder

Question

I'm getting a java outOfMemoryError when I call this method - i'm using it in a loop to parse many large files in sequence. my guess is that result.toString() is not getting garbage collected properly during the loop. if so, how should i fix it?

private String matchHelper(String buffer, String regex, String method){
    Pattern abbrev_p = Pattern.compile(regex);//norms U.S.A., B.S., PH.D, PH.D.
    Matcher abbrev_matcher = abbrev_p.matcher(buffer);
    StringBuffer result = new StringBuffer();
    while (abbrev_matcher.find()){
            abbrev_matcher.appendReplacement(result, abbrevHelper(abbrev_matcher));
    }
    abbrev_matcher.appendTail(result);
    String tempResult = result.toString(); //ERROR OCCURS HERE
  return tempResult;

}

How big is a "large file"? It may be that you're just not allocating enough memory to the JVM. — Ash
– Ash, Commented Mar 12, 2010 at 7:09
how about the pure String of OString Length: 2769348 ? Most of the String is the nexString of the photos captured — Jeff Bootsholz
– Jeff Bootsholz, Commented Jul 9, 2013 at 4:03

erickson · Accepted Answer · 2010-03-12 07:20:13Z

8

Written this way, you'll need roughly 6 bytes of memory for every character in the file.

Each character is two bytes. You have the raw input, the substituted output (in the buffer), and you are asking for a third copy when you run out of memory.

If the file is encoded in something like ASCII or ISO-8859-1 (a single-byte character encoding), that means it will be six times larger in memory than on disk.

You could allocate more memory to the process, but a better solution might be to process the input "streamwise"—read, scan, and write the data without loading it all into memory at once.

answered Mar 12, 2010 at 7:20

erickson

271k59 gold badges406 silver badges502 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

dimitarvp Over a year ago

Thumbs up. If your processing is based on line-by-line work, you could very well just use this: BufferedReader rd = new BufferedReader(new FileReader("/path/to/your/file")); and invoke readLine() in a while loop, then do the replaces and do whatever is necessary with the changed line.

bryantsai · Accepted Answer · 2010-03-12 07:26:56Z

6

If your files to be processed are all very large, say more than a few hundred MB, then you really should go with stream processing instead of this "loading all into memory" way, just as @erickson suggested.

Otherwise, there are a few things you could try, all to reduce memory usage as much as possible:

Try properly enlarge your heap size if not yet (when applicable).
Give StringBuffer an initial size same as the lenght of the given String buffer. This should reduce the unnecessary memory usage while expanding the StringBuffer in the process. I assume it is only replacing certain words of the original string and should be more or less the same in length.
If possible, maybe you could return the generated StringBuffer object instead. Calling its toString() only after you get rid of the original String object.

answered Mar 12, 2010 at 7:26

bryantsai

3,4551 gold badge32 silver badges31 bronze badges

1 Comment

Mehul Joisar Over a year ago

enlarging heap size woks.

Mohammed · Accepted Answer · 2010-09-17 07:56:54Z

2

I reckon the problem with StringBuilder.append(). When Matcher appends sequence of characters to the Builder.

As explained in article about OutOfMemoryError with StringBuilder/StringBuffer, it is a known issue that append() will double the capacity if internal buffer chars if the capacity is not sufficient. Go for streams as suggested by Erickson.

answered Sep 17, 2010 at 7:56

Mohammed

4244 silver badges5 bronze badges

Comments

nicerobot · Accepted Answer · 2010-03-12 07:29:34Z

1

I agree with the other responses ... but ... simply because the exception occurs there doesn't necessarily mean it's the problem. You may very well be leaking memory elsewhere and that just happens to be the place that it's revealed. You should run a profiler to examine memory usage and verify exactly what objects aren't being collected.

answered Mar 12, 2010 at 7:29

nicerobot

9,2556 gold badges45 silver badges44 bronze badges

Comments

user464034 · Accepted Answer · 2010-10-01 15:51:53Z

1

Yes! Don't buffer in memory otherwise you'll run out of it specially if you're going over 2MB on I/O.

Recommended link for fixing and appending text: http://java.ittoolbox.com/documents/appending-data-to-a-file-18786

answered Oct 1, 2010 at 15:51

user464034

111 bronze badge

Comments

fastcodejava · Accepted Answer · 2010-03-12 07:09:52Z

0

You could try returning a StringBuffer and setting it to null after use.

answered Mar 12, 2010 at 7:09

fastcodejava

41.3k31 gold badges142 silver badges191 bronze badges

Collectives™ on Stack Overflow

java outOfMemoryError with stringbuilder

6 Answers 6

1 Comment

1 Comment

Comments

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

6 Answers 6

1 Comment

1 Comment

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related