how many items can a array handle node.js

Question

I am trying to put objects to array based on txt file that has around 500.000 lines or more

I am using require('readline') to handle it, but the processing "pause" for yourself when achieve line 470000(e.g) without errors, warnings, notices...

this is examplo of my code ( the original code fill the dataRow object then it "pauses" when achieve line 411000):

let myList = [];
let lineReader = require('readline').createInterface({
    input: require('fs').createReadStream(filePath).pipe(iconv.decodeStream('latin1'))
});
lineReader.on('line', function (line) {               

    // here there are a lot more fields, but I have to cut off for this example
    let dataRow = JSON.parse('{"Agencia_Cobranca":"","Aliquota_ICMS":""}');

    myList.push(dataRow);

    //this if is only to follow what is happen
    if( myList.length %10000 == 0 || myList.length>420000) {
        console.log(" myList executed: ",myList.length, ' - ', JSON.stringify( myList[myList.length-1] ).length, ' - ' ,new Date() );
    }

}).on('close',function(){   
    console.log('finished');
    process.exit(0);    
});

I am using this command line to execute

node --max-old-space-size=8192 teste

Welll... this is the result, the screen just stay this way when achieve this line... never ends and without errors :(

enter image description here

What operating system? What version of node.js ? What configuration? — Basile Starynkevitch
– Basile Starynkevitch, Commented Sep 5, 2018 at 12:28
I can not see an error relationship with number of lines, I have already read files with more than 1 million lines, test another module, use fs.readfile. — Chance
– Chance, Commented Sep 5, 2018 at 12:37
I don't have problems to read lines, the problem is to store this content in a array of objects — Programador Desenvolvedor
– Programador Desenvolvedor, Commented Sep 5, 2018 at 12:41

Prodigle · Accepted Answer · 2018-09-05 12:29:37Z

1

Your stack/Ram is probably full and erroring out in a weird way. I would recommend if at all possible to make your program more memory efficient, do everything you need to do with a line as you read it and then discard it. Storing it all in memory is never going to be a solution.

answered Sep 5, 2018 at 12:29

Prodigle

1,82715 silver badges25 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Programador Desenvolvedor Over a year ago

I made right now a similar code, but in php, and it worked, but in nodejs it seemed faster than php( considering that moment of "pause" of nodejs ) unfortunately

Franceso Russo · Accepted Answer · 2018-09-05 13:49:12Z

1

In NodeJs (javascript too) maximum size of an array object is 2^32 -1. Just try to execute this in a nodejs application

console.log(new Array(4294967295))

try {
    console.log(new Array(4294967296))
} catch(err){
    console.log(err);
}

answered Sep 5, 2018 at 13:49

Franceso Russo

535 bronze badges

Comments

gege · Accepted Answer · 2018-09-05 13:34:37Z

0

Consider using database if you work with that much data. Store it in a table then query the data you need to work with would be more efficient.

answered Sep 5, 2018 at 13:34

gege

258 bronze badges

Collectives™ on Stack Overflow

how many items can a array handle node.js

3 Answers 3

1 Comment

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related