optimization of sequential i/o operations on large file sizes

Did you try memory-mapping the file? In my test this was always the fastest way to read large files.

Update: Here's an old, but still accurate description of memory mapped files: http://msdn.microsoft.com/en-us/library/ms810613.aspx

Try this.

hf = CreateFile(..... FILE_FLAG_NO_BUFFERING | FILE_FLAG_OVERLAPPED ...)

Then the reading loop. Minor details omitted as typing on iPad...

int bufsize =4*1024*1024;
CEvent e1;
CEvent e2;
CEvent e3;
CEvent e4;
unsigned char* pbuffer1 = malloc(bufsize);
unsigned char* pbuffer2 = malloc(bufsize);
unsigned char* pbuffer3 = malloc(bufsize);
unsigned char* pbuffer4 = malloc(bufsize);
int CurOffset = 0;

do {
   OVERLAPPED r1;
   memset(&r1, 0, sizeof(OVERLAPPED));
   r1.Offset = CurOffset;
   CurOffset += bufsize;
   r1.hEvent = e1;
   if (! ReadFile(hf, pbuffer1, bufsize, bufsize, &r1)) {
       // check for error AND error_handle_eof (important)
   }

   OVERLAPPED r2;
   memset(&r2, 0, sizeof(OVERLAPPED));
   r2.Offset = CurOffset;
   CurOffset += bufsize;
   r2.hEvent = e2;
   if (! ReadFile(hf, pbuffer2, bufsize, bufsize, &r2)) {
       // check for error AND error_handle_eof (important)
   }

   OVERLAPPED r3;
   memset(&r3, 0, sizeof(OVERLAPPED));
   r3.Offset = CurOffset;
   CurOffset += bufsize;
   r3.hEvent = e3;
   if (! ReadFile(hf, pbuffer3, bufsize, bufsize, &r3)) {
       // check for error AND error_handle_eof (important)
   }

   OVERLAPPED r4;
   memset(&r4, 0, sizeof(OVERLAPPED));
   r4.Offset = CurOffset;
   CurOffset += bufsize;
   r4.hEvent = e4;
   if (! ReadFile(hf, pbuffer1, bufsize, bufsize, &r4)) {
       // check for error AND error_handle_eof (important)
   }

   // wait for events to indicate data present
   // send data to consuming threads
   // allocate new buffer
} while ( not eof, etc )

The above is the bones of what you need. We use this and achieve high I/O throughput rates, but you will need to perhaps improve it slightly to achieve ultimate performance. We found 4 outstanding I/O was best for our use, but this will vary by platform. Reading less than 1Mb per IO was performance negative. Once you have the buffer read, don't ty and consume it in the reading loop, post to another thread, and allocate another buffer (but get them from a reuse queue, dont keep using malloc). The overall intent of the above is to try and keep 4 outstanding IO open to the disk, as soon as you don't have this, overall performance will drop.

Also, this works best on a disk that is only Reading your file. If you start reading/writing different files on the same disk at same time, performance drops quickly, unless you have SSD disks!

Not sure why your readfile is failing for 0.5Mb buffers, just double checked and our live prod code is using 4Mb buffers