Out of memory error in regex command loop?

Question

After running a regex, I set the strings to null and force garbage collection, but I'm getting an out-of-memory error.

Manual invocations of the garbage collector aren't doing anything. I use null for empty strings.

int i = 0;

while (i < 40 )
{
    string strFile42 = File.ReadAllText(@"C:\Users\diego\Desktop\finalregex.txt"); 
    string Value3 = @"\n(.*?)&" ;
    string Value4 =string.Format("\n$1#{0}#", i);
    //Force garbage collection.
    GC.Collect();
    strFile42 = Regex.Replace(strFile42, Value3, Value4);
    //Force garbage collection.
    GC.Collect();

    Value4 = null;
    Value3 = null;
    GC.Collect();
    GC.WaitForPendingFinalizers();

    File.WriteAllText(@"C:\Users\diego\Desktop\finalregex.txt", strFile42);
    strFile42 =null;

    GC.Collect();
    GC.WaitForPendingFinalizers();
    i = i + 1;
}

==========================================================

THIS IS MY SOLUTION

      string strFile42 =  File.ReadAllText(@"C:\Users\diego\Desktop\finalregex.txt");


 \\ regex code for replace (PUT REGEX CODE HERE )


    File.WriteAllText(@"C:\Users\diego\Desktop\finalregex.txt", strFile42);


    strFile42 = null ;

  int oldCacheSize = Regex.CacheSize;

Regex.CacheSize         = 0;
  GC.Collect();

    Regex.CacheSize = oldCacheSize; 

  GC.Collect();

 GC.WaitForPendingFinalizers();

Not sure if this is a regex problem, but \n(.*?)& also removes the newline it matches. — user557597
– user557597, Commented Aug 5, 2016 at 23:30
the code works but stop after 20 seg because the memory is enough — user6481216
– user6481216, Commented Aug 5, 2016 at 23:32
@zero625 replacement looks fine. Did you try to debug your code? Are you sure the problem is not about infinite while loop? Try to format your code and insert the whole method — Kovpaev Alexey
– Kovpaev Alexey, Commented Aug 5, 2016 at 23:35
@zero625 also try not to use GC in your code. It works well by itself. Try to follow some naming convensions of c# world so people will more likely try to help you — Kovpaev Alexey
– Kovpaev Alexey, Commented Aug 5, 2016 at 23:39
my code works this stop when i < 40 not infinite the problem is the memory this sum 70mb of memory per loop and stop at 2gb — user6481216
– user6481216, Commented Aug 5, 2016 at 23:43

M.Hassan · Accepted Answer · 2016-08-06 04:32:10Z

You allocate huge size for string when you use: string strFile42 = File.ReadAllText(filename); and doing Regex.Replace on all that huge size at a time represent overhead on memory.

Strings are allocated on the large object heap, which are not collected by GC with other small objects. When you call GC.Collect() to revoke garbage collector, GC CANNOT guarantee the string will be collected immediately.

So, it is best to use BufferedStream with StreamReader and handle one line at a time ,without any overhead on memory.

I measured memory used within execution to monitor memory allocation.

The following class can handle huge size of file without any problems

  public void FileProcess()
    {
        Process proc = Process.GetCurrentProcess();
        int i = 0;
        while (i < 40)
        {
            Console.WriteLine(" i: {0} - Private memory: {1} KB- Memory Used: {2} KB", i,
                proc.PrivateMemorySize64/1024.0, GC.GetTotalMemory(true)/1024.0);    
            var path = "test0.txt";
            var path2 = "test2.txt";
            using (FileStream fs = File.Open(path, FileMode.Open, FileAccess.Read, FileShare.Read))
            using (BufferedStream bs = new BufferedStream(fs))
            using (StreamReader sr = new StreamReader(bs))
            using (StreamWriter outputFile = new StreamWriter(path2))
            {
                string line;
                while ((line = sr.ReadLine()) != null)
                {

                    var text = ProcessLine(line, i);
                    outputFile.WriteLine(text);
                }
            }
            i++;
            // Collect all generations of memory.
            // GC.Collect(); //you need not

        } //while
    }

    private string ProcessLine(string text, int i)
    {
        string Value3 = @"(.*?)&";
        string Value4 = string.Format("$1#{0}#", i);
        var strFile42 = Regex.Replace(text, Value3, Value4);
        return strFile42;
    }
}

Bench Mark Performance Test:

I Generated file with strings with size 72 MB, and used the class to process the file 40 times without any overhead at all. As you see the memory used by application is about 252K and constant all the time without GC collection.

results

     i: 0 - Private memory: 18212 KB- Memory Used: 251.8828125 KB
     i: 1 - Private memory: 18212 KB- Memory Used: 274.5390625 KB
     i: 2 - Private memory: 18212 KB- Memory Used: 274.5390625 KB
     i: 3 - Private memory: 18212 KB- Memory Used: 274.5390625 KB
     i: 4 - Private memory: 18212 KB- Memory Used: 274.5390625 KB
     i: 5 - Private memory: 18212 KB- Memory Used: 274.5390625 KB
     i: 6 - Private memory: 18212 KB- Memory Used: 274.5390625 KB
     i: 7 - Private memory: 18212 KB- Memory Used: 274.48828125 KB
     i: 8 - Private memory: 18212 KB- Memory Used: 274.48828125 KB
     i: 9 - Private memory: 18212 KB- Memory Used: 274.48828125 KB
     i: 10 - Private memory: 18212 KB- Memory Used: 274.48828125 KB
     i: 11 - Private memory: 18212 KB- Memory Used: 274.48828125 KB
     i: 12 - Private memory: 18212 KB- Memory Used: 274.48828125 KB
     i: 13 - Private memory: 18212 KB- Memory Used: 274.48828125 KB
     i: 14 - Private memory: 18212 KB- Memory Used: 274.48828125 KB
     i: 15 - Private memory: 18212 KB- Memory Used: 274.48828125 KB
     i: 16 - Private memory: 18212 KB- Memory Used: 274.48828125 KB
     i: 17 - Private memory: 18212 KB- Memory Used: 274.48828125 KB
     i: 18 - Private memory: 18212 KB- Memory Used: 274.48828125 KB
     i: 19 - Private memory: 18212 KB- Memory Used: 274.48828125 KB
     i: 20 - Private memory: 18212 KB- Memory Used: 274.48828125 KB
     i: 21 - Private memory: 18212 KB- Memory Used: 274.48828125 KB
     i: 22 - Private memory: 18212 KB- Memory Used: 274.48828125 KB
     i: 23 - Private memory: 18212 KB- Memory Used: 274.48828125 KB
     i: 24 - Private memory: 18212 KB- Memory Used: 274.48828125 KB
     i: 25 - Private memory: 18212 KB- Memory Used: 274.48828125 KB
     i: 26 - Private memory: 18212 KB- Memory Used: 274.48828125 KB
     i: 27 - Private memory: 18212 KB- Memory Used: 274.48828125 KB
     i: 28 - Private memory: 18212 KB- Memory Used: 274.48828125 KB
     i: 29 - Private memory: 18212 KB- Memory Used: 274.48828125 KB
     i: 30 - Private memory: 18212 KB- Memory Used: 274.48828125 KB
     i: 31 - Private memory: 18212 KB- Memory Used: 274.48828125 KB
     i: 32 - Private memory: 18212 KB- Memory Used: 274.48828125 KB
     i: 33 - Private memory: 18212 KB- Memory Used: 274.48828125 KB
     i: 34 - Private memory: 18212 KB- Memory Used: 274.48828125 KB
     i: 35 - Private memory: 18212 KB- Memory Used: 274.48828125 KB
     i: 36 - Private memory: 18212 KB- Memory Used: 274.48828125 KB
     i: 37 - Private memory: 18212 KB- Memory Used: 274.48828125 KB
     i: 38 - Private memory: 18212 KB- Memory Used: 274.48828125 KB
     i: 39 - Private memory: 18212 KB- Memory Used: 274.48828125 KB

Community · Accepted Answer · 2017-05-23 12:33:29Z

3

I'm wondering if the cashSize of the Regex isn't the problem.

In the MSDN information, there is a page on Regex

Quote:

The Regex class maintains an internal cache of compiled regular expressions used in static method calls. If the value specified in a set operation is less than the current cache size, cache entries are discarded until the cache size is equal to the specified value. By default, the cache holds 15 compiled static regular expressions. Your application typically will not have to modify the size of the cache. Use the CacheSize property only when you want to turn off caching or when you have an unusually large cache.

Here is a question about CashSize of Regex

edited May 23, 2017 at 12:33

CommunityBot

11 silver badge

answered Aug 6, 2016 at 1:57

Stef Geysels

1,02711 silver badges28 bronze badges

4 Comments

user6481216 Over a year ago

hi friend how i can do this , I dont know

user6481216 Over a year ago

i use this social.msdn.microsoft.com/Forums/en-US/…

Alan Moore Over a year ago

I don't see how the regex cache can be the problem. The code in the question only uses one regex, so the cache size will never get larger than 1.

Stef Geysels Over a year ago

@zero625 all credits for you. You finnaly found the answer. Can you edit your question with your solution? This can be handy for other people.

user6481216 · Accepted Answer · 2016-08-09 00:27:59Z

1

I post my answer here

I use this code for clean memory leak

        string strFile42 =  File.ReadAllText(@"C:\Users\diego\Desktop\finalregex.txt");


  \\ regex code for replace (PUT REGEX CODE HERE )


     File.WriteAllText(@"C:\Users\diego\Desktop\finalregex.txt", strFile42);


      strFile42 = null ;

   int oldCacheSize = Regex.CacheSize;

  Regex.CacheSize         = 0;
   GC.Collect();

        Regex.CacheSize = oldCacheSize; 

    GC.Collect();

GC.WaitForPendingFinalizers();

answered Aug 9, 2016 at 0:27

user6481216

Collectives™ on Stack Overflow

Out of memory error in regex command loop?

3 Answers 3

Comments

4 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

4 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related