Array Memory allocation in nodejs

Question

I am dealing with a huge array,

It contains ~200,000 elements. Basically its a array of strings. Each string being ~50 characters in length. After looking around I found it would take 2 bytes for 1 character, i.e 100 bytes for 1 element.

therefore, the total memory allocation should add up to 200,000 * 100 = ~20 MB

object-sizeof, js-sizeof, sizeof seems to implement same logic.

But consider this snippet,

process.memoryUsage();
const paths = getAllFilePaths();
process.memoryUsage();

Output before getting array,

external:25080
heapTotal:31178752
heapUsed:10427896 //10 MB
rss:51761152

Output after getting array,

external:16888
heapTotal:173539328
heapUsed:134720896 //134 MB
rss:204070912

This is ~124MB addition to heapUsed.

Implementation of getAllFilePaths():

const getAllFilePaths = function (_path, paths = []) {

    fs.readdirSync(_path).forEach(name => {
        const stat = fs.lstatSync(joinPath(_path, name))
        if (stat.isDirectory()) {
            getAllFilePaths(joinPath(_path, name), paths);
            return;
        }

        paths.push(joinPath(_path, name));
    });

    return paths;
};

Why is so much memory being used ? Is this the desired behaviour or somehow getAllFilePaths() function could possibly be leaking memory ?

It could be that the method you use to build this Array of strings, is deep-nested as such, makes use of other resources, etc. — MadaManu
– MadaManu, Commented Jan 6, 2020 at 7:06
Please mention the implementation of your getArray() function — UtkarshPramodGupta
– UtkarshPramodGupta, Commented Jan 6, 2020 at 7:10
getArray() is a recursive function that traverses a directory path and returns all the nested filepaths. — laxman
– laxman, Commented Jan 6, 2020 at 7:10

jmrk · Accepted Answer · 2020-01-06 11:47:59Z

V8 developer here. Two points come to mind to explain the discrepancy between your expectations and measurements:

(1) An array of strings needs more memory than just the strings' characters. In memory, a string object has a header that takes 16 bytes on a 64-bit system (a pointer-sized[1] "shape" pointer plus two 32-bit fields for hash and length). Depending on how exactly the strings are constructed, they might also use different representations internally; header + characters is the simplest form. Additionally, the array itself has a pointer-sized entry for each element, adding at least another 200,000 * 8 bytes = 1.5MB -- dynamically-grown arrays over-allocate when they have to grow so that they don't have to grow for every addition, which can waste space if the array is unlucky enough to stop growing right after having over-allocated.

(2) AFAIK process.memoryUsage() simply returns the current heap usage statistics, which can contain garbage left behind by previous operations. To determine the memory consumption of something, it is advisable to explicitly trigger a full GC cycle before every measurement. Specifically: start Node with --expose-gc and call global.gc() before every process.memoryUsage().

For completeness, I'll mention: strings can take 1 or 2 bytes per character depending on their contents. Per individual string, each character takes the same amount, so a single non-ASCII character forces the entire string to be two-byte. For embedder-provided strings (like file names), the embedder also has to play along to support the one-byte optimization; I don't know whether Node's file API does this.

[1] "pointer-sized" means 64 bits = 8 bytes nowadays; with "pointer-compression" becoming available in V8 8.0 this shrinks to 4 bytes (if you choose to deploy a pointer-compressed build).

MadaManu · Accepted Answer · 2020-01-06 07:43:59Z

1

Did a small test here: Memory Leak Test

This seems to show that 200,000 items of 50 chars hardcoded into an array outputs the following:

{ 
rss: 58232832,
heapTotal: 40378368,
heapUsed: 25490136, // ~25 MB
external: 8272 
}

answered Jan 6, 2020 at 7:43

MadaManu

6764 silver badges10 bronze badges

1 Comment

MadaManu Over a year ago

This is more than likely not quite conclusive, since the string is hardcoded, and could be potentially optimized out, but interesting to see.

Collectives™ on Stack Overflow

Array Memory allocation in nodejs

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related