Optimize C# file IO

Question

Scenario - 150MB text file which is the exported Inbox of an old email account. Need to parse through and pull out emails from a specific user and writes these to a new, single file. I have code that works, its just dogged slow.

I'm using marker strings to search for where to begin/end the copy from the original file.

Here's the main function:

 StreamReader sr = new StreamReader("c:\\Thunderbird_Inbox.txt");
        string working = string.Empty;
        string mystring = string.Empty;
        while (!sr.EndOfStream)
        {
            while ((mystring = sr.ReadLine()) != null)
            {
                if (mystring == strBeginMarker)
                {
                    writeLog(mystring);

                    //read the next line
                    working = sr.ReadLine();

                        while( !(working.StartsWith(strEndMarker)))
                        {
                            writeLog(working);
                            working = sr.ReadLine();

                        }
                  }
            }

        }
        this.Text = "DONE!!";
        sr.Close();

The function that writes the selected messages to the new file:

  public void writeLog(string sMessage)
    {
            fw = new System.IO.StreamWriter(path, true);
            fw.WriteLine(sMessage);
            fw.Flush();
            fw.Close();
    }

Again, this process works. I get a good output file, it just takes a long time and I'm sure there are ways to make this faster.

BTW - You might want to consider the using statement instead of Close() manually - it's safer in case you hit an exception. My example demonstrates... — Reed Copsey
– Reed Copsey, Commented Jan 20, 2011 at 19:19
while (!sr.EndOfStream) is redundant with while ((mystring = sr.ReadLine()) != null) — Kirk Woll
– Kirk Woll, Commented Jan 20, 2011 at 19:20

Reed Copsey · Accepted Answer · 2011-01-20 19:41:17Z

20

The largest optimization would be to change your writeLog method to open the file once at the beginning of this operation, write to it many times, then close it at the end.

Right now, you're opening and closing the file each iteration where you write, which is going to definitely slow things down.

Try the following:

// Open this once at the beginning!
using(fw = new System.IO.StreamWriter(path, true))
{
    using(StreamReader sr = new StreamReader("c:\\Thunderbird_Inbox.txt"))
    {
        string working;
        string mystring;
        while ((mystring = sr.ReadLine()) != null)
        {
           if (mystring == strBeginMarker)
           {
                writeLog(mystring);

                //read the next line
                working = sr.ReadLine();

                while( !(working.StartsWith(strEndMarker)))
                {
                    fw.WriteLine(working);
                    working = sr.ReadLine();
                }
            }
        }
    }
}
this.Text = "DONE!!";

edited Jan 20, 2011 at 19:41

answered Jan 20, 2011 at 19:13

Reed Copsey

567k80 gold badges1.2k silver badges1.4k bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

paparush Over a year ago

THIS CHANGES EVERYTHING! :-) Fantastic response. What was taking 7 or 8 minutes finished in about 2 seconds. Better still, I learned some valuable coding techniques.

Reed Copsey Over a year ago

@paparush: Glad we could help ;)

acoolaum · Accepted Answer · 2011-01-20 19:24:54Z

2

I think you should:

Open files once.
Load source file in memory.
Break it and use several threads for processing.

answered Jan 20, 2011 at 19:24

acoolaum

2,1503 gold badges15 silver badges24 bronze badges

8 Comments

Reed Copsey Over a year ago

While I like this answer, in theory - in practice, it will probably not help much. The OP is still likely to be completely IO bound on the output (since it's being written to a single output file), so multithreading will likely yield very little benefit. I don't know if it'd be worth the complexity.

Wonko the Sane Over a year ago

I agree with Reed here. How would you break it into threads? You'd still need to have logic to do so. If you broke it into chunks of equal size, you'd need to handle the case where a single message starts in one chunk and ends in another. If you did logic to break it at a marker boundary, you are pre-parsing before adding the overhead of threading, probably making it less efficient. That's not to say it couldn't be done - it's just much more work than it seems worth.

acoolaum Over a year ago

@Wonko the Sane, I think the simplest way is first - break it for equal parts and find begining of the first message in each, second - start thread proccessing from first message of each section.

Wonko the Sane Over a year ago

@acoolaum - but, you aren't handling the case where the break is mid-message (the start marker is in chunk #1, the end marker is in chunk #2). Plus, there is no "processing" needed in this case - the task is, essentially, to take a number of substrings out of a string and plop them in another string.

acoolaum Over a year ago

@Wonko the Sane, the task is clear for me. In first step you have 2 parts: placing pointers of parts to calculated initial position then move pointer of each part to first message in this part.

|

Wonko the Sane · Accepted Answer · 2011-01-20 19:43:37Z

2

I would just do a simple parser. Note that this assumes (as you do in your code above) that the markers are in fact unique.

You may have to play with the formatting a bit of your output, but here is the general idea:

   // Read the entire file and close it
   using (StreamReader sr = new
   StreamReader("c:\\Thunderbird_Inbox.txt");)
   {
       string data = sr.ReadToEnd();   
   }

   string newData = "";   
   int position = data.IndexOf(strBeginMarker);

   while (position > 0)   
   {
      int endPosition = data.IndexOf(endMarker, position);
      int markerLength = position + strBeginMarker.Length;

     newData += data.Substring(markerLength, endPosition - markerLength);

     position = data.IndexOf(strBeginMarker, position+ endStr.Length);   
   }

  writeLog(newData);

(Note that I don't have a 150 MB file to test this on - YMMV depending on the machine you are using).

edited Jan 20, 2011 at 19:43

answered Jan 20, 2011 at 19:38

Wonko the Sane

10.8k8 gold badges72 silver badges97 bronze badges

2 Comments

Gabriel Magana Over a year ago

You should at least warn OP about the potential dangers of loading a 150MB file at once through sr.ReadToEnd()...

Wonko the Sane Over a year ago

Just did (in a very basic way)

Mike · Accepted Answer · 2011-01-20 19:17:52Z

0

I do not have a 150MB text file to test, but if your server has the memory would Reading the hold thing into a string and doing a RegEx pulling out the message work?

answered Jan 20, 2011 at 19:17

Mike

6,0909 gold badges60 silver badges94 bronze badges

1 Comment

Wonko the Sane Over a year ago

RegEx would work, but they can be fairly complex and difficult to get "just right." The parser solution I gave is essentially the same idea, but simpler for an less experienced coder.

Smur · Accepted Answer · 2011-01-20 19:19:56Z

0

You could simply declare the StreamWriter object outside of that while loop and just write the line to it inside the loop.

Like this:

StreamWriter sw = new StreamWriter(path, true);
while
{
    // ...
    while( !(working.StartsWith(strEndMarker)))
    {
        sw.WriteLine(working);
        working = sr.ReadLine();
    }
}

answered Jan 20, 2011 at 19:19

Smur

3,1137 gold badges30 silver badges47 bronze badges

Collectives™ on Stack Overflow

Optimize C# file IO

5 Answers 5

2 Comments

8 Comments

2 Comments

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

2 Comments

8 Comments

2 Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related