C# Memory Usage Problem

Question

I have an method and it converts pdf text into a list. After the process the memory usage increase too much. For example a 1000 page pdf use 300mb memory and i can't free it. I have readed some LOH articles but have not find a solution.

 public List<string> GetTextFromPdf()
    {
        if (_pdfDoc.Pages == null) return null;
        List<string> ocrList = new List<string>();

        foreach (var words in _pdfDoc.Pages.Select(s => s.Value.WordList))
        {
            ocrList.AddRange(words.Select(word => word.Word).Select(input => Regex.Replace(input, @"[\W]", "")));
        }

        GC.Collect();
        return ocrList;
    }

Don't re-parse the regex every time - use a shared Regex instance — SLaks
– SLaks, Commented Jun 19, 2011 at 13:35

Community · Accepted Answer · 2017-05-23 10:08:06Z

5

This is about normal for a 100 megabyte .pdf. You load the entire thing in memory, that takes double the amount of memory since a character in .NET takes 2 bytes. You will also create a bunch of garbage in the large object heap for the list. Add the typical .NET runtime overhead and 300 megabytes is not an unexpected result.

Check this answer for details on how using the List<>.Capacity property can help reduce the LOH demands.

edited May 23, 2017 at 10:08

CommunityBot

11 silver badge

answered Jun 19, 2011 at 14:14

Hans Passant

946k151 gold badges1.8k silver badges2.6k bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

EricLaw Over a year ago

I'm curious about why you think PDFs aren't already in Unicode?

Hans Passant Over a year ago

@Eric - PDF has been around too long to benefit from Unicode standardization. It has the typical 8-bit encoding zoo, "WinAnsi" is one of them.

Orhan Cinar Over a year ago

i cleared the list and set capacity to 0 but the memory usage is still same

Hans Passant Over a year ago

Yes, it is rare for the Windows memory manager to release virtual memory. The odds that the released memory exactly matches a memory mapping is very low. Not a problem, it is virtual. Minimize your main window to make yourself feel better. The linked answer tried to explain how to reduce LOH usage by not allocating it in the first place.

Henk Holterman Over a year ago

@Orhan : It's not (just) about clearing the list but about allocating it wisely. Like ocrList = new List<string>(_pdfDoc.Pages.Count * OverEstimateWordsPerPage);

Piotr Auguscik · Accepted Answer · 2011-06-19 13:37:22Z

0

Check if your pdf loader is referenced somewhere - so it can not be disposed.

answered Jun 19, 2011 at 13:37

Piotr Auguscik

3,6911 gold badge25 silver badges31 bronze badges

1 Comment

Orhan Cinar Over a year ago

The problem is after adding the words to my ocrList. When i call _pdfDoc.Dispose() the app crashes. Because the pdf is still inside a viewer.

Bob Vale · Accepted Answer · 2011-06-19 13:38:52Z

0

Is your pdf library COM based? You may need to call Marshall.releasecomobject on some of your references when you have finished with them.

answered Jun 19, 2011 at 13:38

Bob Vale

18.5k1 gold badge44 silver badges51 bronze badges

Collectives™ on Stack Overflow

C# Memory Usage Problem

3 Answers 3

5 Comments

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

5 Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related