Read Multiple Byte Arrays from File

Question

How can I read multiple byte arrays from a file? These byte arrays are images and have the potential to be large.

This is how I'm adding them to the file:

 using (var stream = new FileStream(tempFile, FileMode.Append))
      {
        //convertedImage = one byte array
        stream.Write(convertedImage, 0, convertedImage.Length);
      }

So, now they're in tempFile and I don't know how to retrieve them as individual arrays. Ideally, I'd like to get them as an IEnumerable<byte[]>. Is there a way to split these, maybe?

You can't retrieve them, you don't know how many bytes to read. You'll have to fix the writing code and at a minimum write convertedImage.Length first. — Hans Passant
– Hans Passant, Commented Jun 8, 2017 at 16:26
How would you know where to split? There seems to be no information about that available in file itself. — Evk
– Evk, Commented Jun 8, 2017 at 16:27
@HansPassant...that's what I was afraid of. Where do I split? Can you elaborate on your second sentence, please? If I can manually set the bytes to read for each image, that's doable. — Big Daddy
– Big Daddy, Commented Jun 8, 2017 at 16:32
Well you got 3 answers already. Base idea is simple: first write length of image, then image itself. Repeat for all images. When reading: read length, read specified number of bytes. Repeat until file ends. — Evk
– Evk, Commented Jun 8, 2017 at 16:36

Emma Talbert · Accepted Answer · 2017-06-26 15:31:26Z

2

To retrieve multiple sets of byte arrays, you will need to know the length when reading. The easiest way to do this (if you can change the writing code) is to add a length value:

using (var stream = new FileStream(tempFile, FileMode.Append))
{
    //convertedImage = one byte array
    // All ints are 4-bytes
    stream.Write(BitConverter.GetBytes(convertedImage.Length), 0, 4);
    // now, we can write the buffer
    stream.Write(convertedImage, 0, convertedImage.Length);
}

Reading the data is then

using (var stream = new FileStream(tempFile, FileMode.Open))
{
    // loop until we can't read any more
    while (true)
    {
        byte[] convertedImage;
        // All ints are 4-bytes
        int size;
        byte[] sizeBytes = new byte[4];
        // Read size
        int numRead = stream.Read(sizeBytes, 0, 4);
        if (numRead <= 0) {
            break;
        }
        // Convert to int
        size = BitConverter.ToInt32(sizeBytes, 0);
        // Allocate the buffer
        convertedImage = new byte[size];
        stream.Read(convertedImage, 0, size);
        // Do what you will with the array
        listOfArrays.Add(convertedImage);
    } // end while
}

If all saved images are the same size, then you can eliminate the first read and write call from each, and hard-code size to the size of the arrays.

edited Jun 26, 2017 at 15:31

answered Jun 8, 2017 at 16:31

Emma Talbert

2271 silver badge9 bronze badges

Sign up to request clarification or add additional context in comments.

11 Comments

Big Daddy Over a year ago

Looks promising, but reading the data is really slow, as in minutes, on a 750kb file.

Emma Talbert Over a year ago

Can you tell how many arrays are in that file? Also, if you look in task manager while it's running, what's the disk usage? How about the CPU usage?

Big Daddy Over a year ago

There are 7 arrays/images. CPU is 35% on that process (45% overall) and 0% on disk.

Emma Talbert Over a year ago

Now I'm confused. Are you sure it's in that part of the code that it is hanging/slowing? If not, try wrapping the section it with Debug.WriteLine() calls to see how long this part takes.

Big Daddy Over a year ago

I'm positive. The "size" variable is always 0.

|

Sergey Kalinichenko · Accepted Answer · 2017-06-08 16:28:00Z

Unless you can work out the number of bytes taken by each individual array from the content of these bytes themselves, you need to store the number of images and their individual lengths into the file.

There are many ways to do it: you could write lengths of the individual arrays preceding each byte array, or you could write a "header" describing the rest of the content before writing the "payload" data to the file.

Header may look as follows:

Byte offset    Description
-----------  -------------------
0000...0003 - Number of files, N
0004...000C - Length of file 1
000D...0014 - Length of file 2
...
XXXX...XXXX - Length of file N
XXXX...XXXX - Content of file 1
XXXX...XXXX - Content of file 2
...
XXXX...XXXX - Content of file N

You can use BitConverter methods to produce byte arrays to be written to the header, or you could use BinaryWriter.

Rob · Accepted Answer · 2017-06-08 16:27:15Z

1

When you read back how do you get the number of bytes per image/byte array to read?

You will need to store the length too (i.e. first 4 bytes = encoded 32 bit int byte count, followed by the data bytes.)

To read back, read the first four bytes, un-encode it back to an int, and then read that number of bytes, repeat until eof.

answered Jun 8, 2017 at 16:27

Rob

4644 silver badges10 bronze badges

Collectives™ on Stack Overflow

Read Multiple Byte Arrays from File

3 Answers 3

11 Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

11 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related