nodeJs huge array processing throws RangeError: Maximum call stack size exceeded

Question

This is part of code to proceed big number of entries ( originally its works with file system and make some operations with files) . Is there any nice way to bypass the limitation and prevent throwing of RangeError: Maximum call stack size exceeded ( As for now it allows me to iterate about 3000 items )

var async = require('async')
    , _u = require('underscore')

var tifPreview = function (item, callback) {
    console.log(item)
    return callback();
}

var tifQueue = async.queue(tifPreview, 2)

tifQueue.push(_u.range(0, 5000, 1))

georgephillips · Accepted Answer · 2014-09-21 01:35:09Z

26

The problem is that you are making to many function calls. Setting the stack-size to a higher value will only increase the number of items you can handle, not solve the actual problem.

You are calling the next iteration straight from your function, which makes it a recursive function. It's a bit hard to spot since it's going thru async.

This code should work:

var tifPreview = function (item, callback) {
  console.log(item);

  // defer the callback
  setImmediate(callback);
}

Read more about the setImmediate function here: http://nodejs.org/api/timers.html#timers_setimmediate_callback_arg

edited Sep 21, 2014 at 1:35

georgephillips

3,5904 gold badges25 silver badges30 bronze badges

answered Apr 6, 2014 at 9:38

Linus Unnebäck

24.4k16 gold badges79 silver badges91 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

georgephillips Over a year ago

This should definitely be the accepted answer. Actually fixes instead of pushing the bottle neck down.

bevacqua · Accepted Answer · 2013-11-27 13:58:06Z

25

An option could be passing --max-stack-size to node.

node --max-stack-size 32000 app.js

For reference, use node -h

--max-stack-size=val set max v8 stack size (bytes)

Update

Even though help prints it as --max-stack-size, in node -v v0.10.x+ you need to use --stack-size instead.

node --stack-size=32000 app.js

edited Nov 27, 2013 at 13:58

answered Nov 27, 2013 at 13:44

bevacqua

48.8k56 gold badges176 silver badges287 bronze badges

3 Comments

JohnnyHK Over a year ago

In recent builds, it's actually --stack-size even though the help text says otherwise.

Wilk Over a year ago

@Nico: with --stack-size=32000 it exits with no error but it stops the execution anyway. From the Node's man page it seems that is --stack_size instead of --stack-size.

detj Over a year ago

I don't think increasing the stack size is a real solution. It doesn't fix the underlying issue. The code should be re-implemented by using streams or by breaking into multiple sub-processes.

Collectives™ on Stack Overflow

nodeJs huge array processing throws RangeError: Maximum call stack size exceeded

2 Answers 2

1 Comment

Update

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Update

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related