Writing and loading a large array to/from a file in C++

Question

I'm creating a large 2D array(5000*5000) of data which I need to use later in my program. The process of filling up the array takes 10 seconds but the array data in the end are always the same. So, I was thinking I could avoid these 10 seconds by reading the array from a file each time I run my program.

However, I can't think of an efficient way to write my array into a file. The array consists of floats and writing all the floats with a space between or even a line, I could easily read them later and recreate the array. This obviously turned out to be a bad idea as I end up with a huge txt file of million lines. Reading from that takes even longer and the file takes up way too much space.

How does someone save an array into a file so it can be loaded later?

Edit: You suggested me to try saving in binary format. The tutorials I've found have confused me a bit. Here's my effort:

//2d array[size][size]
vec4** F = new vec4*[size];
for (int i = 0; i < size; i++) {
    F[i] = new vec4[size];
}
// Array is filling up, I won't include this part
//...........
// Array is ready.

//Trying to write the array in a file.
FILE* pFile;
pFile = fopen("myfile.bin", "wb");
fwrite(F, sizeof(vec4*), sizeof(F), pFile);

//Other method
std::ofstream out("filename.data", std::ios_base::binary);
out.write((char*)F, sizeof(vec4)*(size*size));

Both these methods create empty files. Maybe the fact that it's a 2d array complicates fwrite and write?

The creation process takes 10 seconds That seems long for allocating such an array. Are you running release mode code / an optimized build? — drescherjm
– drescherjm, Commented Sep 17, 2020 at 23:30
Your array takes 100 megabytes ( 32b floats ), which is 100 million letters of text at 8b/symbol. — Peri
– Peri, Commented Sep 17, 2020 at 23:30
@drescherjm My bad. I meant evaluating the data and filling up the array. Since the resulting array is always the same, I want to avoid the step of calculating the data and just read the array . — John Katsantas
– John Katsantas, Commented Sep 17, 2020 at 23:33
If 10 seconds is too long to initialize an array, how fast does it need to be for your application? — wallyk
– wallyk, Commented Sep 17, 2020 at 23:36
@JohnKatsantas "How do I create a binary file?" - use std::ofstream with the binary flag enabled. Then you can write() the array data as-is to the stream. — Remy Lebeau
– Remy Lebeau, Commented Sep 17, 2020 at 23:41

Remy Lebeau · Accepted Answer · 2020-09-18 00:50:42Z

3

Simply write out the entire array as-is in its original binary form, not in a textual form.

vec4* F = new vec4[size*size];

// fill and use the array as needed...

std::ofstream out("filename.data", std::ios_base::binary);
out.write((char*)F, sizeof(vec4)*(size*size));

delete[] F;

You can then read the file back into your array as-is:

vec4* F = new vec4[size*size];

std::ifstream in("filename.data", std::ios_base::binary);
if (!in.read((char*)F, sizeof(vec4)*(size*size)))
{
    // generate new values as needed...
}

// use the array as needed...

delete[] F;

Alternatively, if you use a memory-mapped file (CreateFileMapping()/MapViewOfFile() on Windows, mmap() on Linux, etc), then you don't even have to allocate an array and read the file into it, you can just access the file data directly using a memory-mapped data pointer, eg:

vec4 *F = map the file ...; // <-- use platform-specific APIs for this!
bool mapped = (F != nullptr);
if (!mapped)
{
    F = new vec4[size*size];

    // generate new values as needed...

    std::ofstream out("filename.data", std::ios_base::binary);
    out.write((char*)F, sizeof(vec4)*(size*size));
}

// use the array as needed ...

if (mapped)
    unmap the file; // <-- use platform-specific APIs for this...
else
    delete[] F;

UPDATE: if you want to use a 2-dimensional sparse array, you will have to account for that a little differently in your file I/O, eg:

vec4** F = new vec4*[size];
for (int i = 0; i < size; ++i) {
    F[i] = new vec4[size];
}

{
    std::ifstream in("filename.data", std::ios_base::binary);
    if (in)
    {
        for (int i = 0; i < size; ++i) {
            in.read((char*)F[i], sizeof(vec4)*size);
        }
    }

    if (!in)
    {
        // generate new values as needed...
    }
}

// use the array as needed...

{
    std::ofstream out("filename.data", std::ios_base::binary);
    for (int i = 0; i < size; ++i) {
        out.write((char*)F[i], sizeof(vec4)*size);
    }
}

for (int i = 0; i < size; ++i) {
    delete[] F[i];
}
delete[] F;

edited Sep 18, 2020 at 0:50

answered Sep 17, 2020 at 23:40

Remy Lebeau

609k36 gold badges516 silver badges875 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

John Katsantas Over a year ago

Looks like what I needed! Will a 2D array implicate how these functions are used?

Remy Lebeau Over a year ago

I think you mean complicate, and yes it will, a little. I have updated my answer to show that.

Collectives™ on Stack Overflow

Writing and loading a large array to/from a file in C++

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related