Need help Creating a big array from small byte arrays

Question

i got the following code:

    byte[] myBytes = new byte[10 * 10000];
    for (long i = 0; i < 10000; i++)
    {
        byte[] a1 = BitConverter.GetBytes(i);
        byte[] a2 = BitConverter.GetBytes(true);
        byte[] a3 = BitConverter.GetBytes(false);

        byte[] rv = new byte[10];
        System.Buffer.BlockCopy(a1, 0, rv, 0, a1.Length);
        System.Buffer.BlockCopy(a2, 0, rv, a1.Length, a2.Length);
        System.Buffer.BlockCopy(a3, 0, rv, a1.Length + a2.Length, a3.Length);
    }

everything works as it should. i was trying to convert this code so everything will be written into myBytes but then i realised, that i use a long and if its value will be higher then int.MaxValue casting will fail. how could one solve this?

another question would be, since i dont want to create a very large bytearray in memory, how could i send it directry to my .WriteBytes(path, myBytes); function ?

Marc Gravell · Accepted Answer · 2012-10-12 12:09:31Z

2

If the final destination for this is, as suggested, a file: then write to a file more directly, rather than buffering in memory:

using (var file = File.Create(path)) // or append file FileStream etc
using (var writer = new BinaryWriter(file))
{
    for (long i = 0; i < 10000; i++)
    {
        writer.Write(i);
        writer.Write(true);
        writer.Write(false);
    }
}

Perhaps the ideal way of doing this in your case would be to pass a single BinaryWriter instance to each object in turn as you serialize them (don't open and close the file per-object).

answered Oct 12, 2012 at 12:09

Marc Gravell

1.1m273 gold badges2.6k silver badges3k bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Rafael Herscovici Over a year ago

is BinaryWriter the fastest way to write this? in this example i have 10000 iterations, but in RL it might be 1 Billion, which can take some time.

Marc Gravell Over a year ago

@Demsntic that makes it even more important to not try buffering it it memory. Also: I would not expect BinaryWriter to have the overhead if all those extra gen-zero intermediate arrays. So: yes it'll be fast, and certainly faster than having to allocate a huge array in memory, then think for ages, then do a huge single IO operation.

cirrus · Accepted Answer · 2012-10-12 11:56:37Z

1

Why don't you just Write() the bytes out as you process them rather than converting to a massive buffer, or use a smaller buffer at least?

answered Oct 12, 2012 at 11:56

cirrus

5,6908 gold badges46 silver badges62 bronze badges

1 Comment

Rafael Herscovici Over a year ago

My write function support writing one object at a time, so if i try to .WriteBytes(path, rv) it will Overwrite in each run of the loop.

Collectives™ on Stack Overflow

Need help Creating a big array from small byte arrays

2 Answers 2

2 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related