How to read binary array directly from the stream?

Question

Here by "directly" I mean without temporary byte[] array.

The problem is, for example I have array of ints or doubles on the disk, so currently I create two arrays -- byte array and int array (in case of ints). The former is just for reading, the latter is the actual output.

Since Stream can read only to byte array I read it to the first array, than copy all the data to the second. It works, but it really hurts me (I am not talking about performance here).

So, how to read the array without temporary array? Using C# unsafe context is fine for me.

So far I tried two approaches: I looked if it is possible to create an array reusing allocated memory and second which looked more promising -- I could get pointer to the result/second array and in unsafe context I could cast it to byte* pointer. Considering my needs it is 100% safe and valid, however byte* pointer is not a byte[] array in C# world and I cannot find the way to cast pointer to array.

Code:

void ReadStuff(Stream stream, double[] data)
{  
  var dataBytes = new byte[data.Length * sizeof(double)];
  stream.Read(dataBytes, 0, dataBytes.Length);
  Buffer.BlockCopy(dataBytes, 0, data, 0, dataBytes.Length);
  // ...
}

No. Use Read() method with number of bytes you need and put into a byte[]. You do not need to read the entire stream to end. You can check the position and length of stream to make sure there is enough data in stream before reading. — jdweng
– jdweng, Commented Sep 18, 2020 at 12:00
@astrowalker your question isn't clear. Stream.Read fills a reusable buffer with just the bytes you asked. So there's no temporary array involved. If you have a lot of concurrent operations and worry that even those buffers can harm performance you could use a MemoryPool to reuse the same buffer across streams. You certainly don't need a byte pointer. If you don't want to copy arrays, you can use Span<byte> — Panagiotis Kanavos
– Panagiotis Kanavos, Commented Sep 18, 2020 at 12:23
@astrowalker you should probably post your code instead of describing it — Panagiotis Kanavos
– Panagiotis Kanavos, Commented Sep 18, 2020 at 12:24
@astrowalker all languages, all runtimes, all OSs work like that. When you use a byte* in C, you also work like that. Someone allocated that array. And since you use the same buffer each time, there's no waste or extra GC involved even with a single stream. MemoryPool is used when you need to conserve memory across thousands of streams, eg when you have a high-traffic web service. So what's the real problem? Why look at a byte*? If you want to avoid copying, you use Stream.Read(Span<byte>) — Panagiotis Kanavos
– Panagiotis Kanavos, Commented Sep 18, 2020 at 12:40
BinaryReader can be used to read integers and bytes from part of stream learn.microsoft.com/en-us/dotnet/api/system.io.binaryreader — Slai
– Slai, Commented Sep 18, 2020 at 12:43

JonasH · Accepted Answer · 2020-09-18 12:35:30Z

1

There is no way I know of to copy data directly from a stream to a typed array. But you can process the data in chunks, limiting your memory overhead to a fixed amount. The memory will be copied twice, but this is unavoidable as far as I know.

For example:

            public static void ReadArrayDataChunked(BinaryReader binaryReader, Array target, Type type, int bufferSize = 4096)
    {
        var buffer = new byte[bufferSize];
        var tSize = Marshal.SizeOf(type) ;

        var remainingBytes = target.Length * tSize;
        var targetPosition = 0;
        while (remainingBytes > 0)
        {
            var toRead = Math.Min(remainingBytes, buffer.Length);
            var bytesRead = binaryReader.Read(buffer, 0, toRead);
            Buffer.BlockCopy(buffer, 0, target, targetPosition, bytesRead);
            targetPosition += bytesRead;
            remainingBytes -= bytesRead;
        }
    }

Note that this only works for primitive types due to the BlockCopy, but this helps improve copy performance. You will need to read other types item by item.

answered Sep 18, 2020 at 12:35

JonasH

38.5k2 gold badges16 silver badges31 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

greenoldman Over a year ago

Yes, thank you, it looks like I end up following this path, because I don't see a glue that connects my "outer" needs (like double array) with stream API needs (byte array).

JonasH Over a year ago

If you have a double[] and want a byte[] or vice versa you could just do a block-Copy directly, no need for a stream to do that.

greenoldman Over a year ago

My data (doubles) are on the disk, so I need stream to read them in the first place.

Collectives™ on Stack Overflow

How to read binary array directly from the stream?

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related