0

I'm trying to insert some data into my MongoDB collection using async's waterfall method, to make sure that everything is done in the correct order. Initially my data was being inserted into MongoDB in the wrong order, which after a while I have realised is because of the fact my insert was running asynchronously so there was no way for me to guarantee that the data would be saved in the correct order.

So here is my async.waterfall:

async.waterfall([
    function(callback) {
      request.get(apiUrl, function(error, response, body) {
        if (!error && response.statusCode == 200) {
          data = JSON.parse(body);
          callback(error, data.items);
        }
      });
    },
    function(data, callback) {
      removeInvalidItems(data);
      var items = convertToObj(data);
      callback(null, items);
    },
    function(items, callback) {
      console.log(items);
      // Insert to collection here
    }
], function (err, result) {
    // do something here
});

So on the last function in the waterfall I have logged items to console, and it is showing the data in the correct order. So now, all I need to do is insert the data to my collection, and in the callback close the connection and move out of the waterfall to the final function. Easy enough right? Well, this is what I am trying right now:

  MongoClient.connect(config.db, function(err, db) {
    if (err) {
      console.log(err);
    } else {
      console.log("Connected correctly to server");

      db.collection('items').insert(items, function(err, result) {
        if (err) {
          console.log(err);
        }
        db.close();
        callback(null, 'done');
      });
    }
  });

The db.close() and callback(null, 'done') should be closing the connection and moving out of the waterfall once they receive the callback. However all I seem to get is RangeError: Maximum call stack size exceeded. I have tried everything that I can think of and I can't get this insert to work with a callback and move out of the waterfall.

Can anyone help?

5
  • You may be looking for an error in the wrong place - is your items variable a plain JSON object? Commented Aug 26, 2015 at 16:11
  • I have made sure that each item in items is stored as a new Item object according to a mongoose Schema. I believe I am doing something wrong with the callback, everything works up until the insert. Commented Aug 26, 2015 at 16:21
  • @germainelol, callback should be called for every task of async.waterfall(). You're calling it ONLY in successful case of your 1st waterfall task... and you're not calling it at all in the 3rd waterfall task. Commented Aug 26, 2015 at 19:27
  • @Ben Could you help to fix that? I've tried to follow async's documentation but apparently not very well. Commented Aug 26, 2015 at 20:19
  • Ok, will put it in answer since it's too long in comment. Commented Aug 26, 2015 at 21:03

3 Answers 3

1

What's the size of this items collection? Can it exceed the callstack size? To get your current size run:

node --v8-options | grep -B0 -A1 stack_size

The default size is 492 kB (for 32-bit) and 984 kB (for 64-bit).

To increase your callstack size try starting node with parameter:

node --max-stack-size={your-desired-size-in-bytes}

Take into consideration that node might not have time to clear callstack between each waterfall callback. Try to run callback on nextTick:

async.waterfall([
    function(callback) {
      request.get(apiUrl, function(error, response, body) {
        if (!error && response.statusCode == 200) {
          data = JSON.parse(body);
          callback(error, data.items);
        }
      });
    },
    function(data, callback) {
      removeInvalidItems(data);
      var items = convertToObj(data);
      process.nextTick(function() {   // callback on next tick
          callback(null, items);
      });
    },
    function(items, callback) {
      console.log(items);
      // Insert to collection here
    }
], function (err, result) {
    // do something here
});
Sign up to request clarification or add additional context in comments.

1 Comment

It's a very small collection of less than 20 small items so nowhere near the maximum in that sense.
1

All callbacks should be called for every task of async.waterfall(). It is called ONLY in successful case of your 1st waterfall task... and it's not called at all in the 3rd waterfall task.

There is no callback for these 2 calls:

removeInvalidItems(data);
items = convertToObj(data);

so I'm assuming they are not asynchronous. And since they are not async, you don't have to put them as a separate task for async.waterfall(). So here is the revised code (I'm assuming that your data conversion is correct):

async.waterfall([
    function(callback) {
        request.get(apiUrl, function(error, response, body) {
            var data;
            if (!error && response.statusCode == 200) {
                data = JSON.parse(body);
                return callback(null, data.items);
            }
            callback(error || response); // error or statusCode !== 200
        });
    },
    function(data, callback) {
        var items;

        // There is no callback for these, so I'm assuming they are not asynchronous
        removeInvalidItems(data);
        items = convertToObj(data);

        MongoClient.connect(config.db, function(err, db) {
            if (err) {
                console.log(err);
                return callback(err);
            }
            console.log("Connected correctly to server");
            db.collection('items').insert(items, function(err, result) {
                if (err || !result) {
                    console.log(err);
                    db.close();
                    return callback(err || {message: 'There is a problem inserting.'});
                }
                db.close();
                callback(null);
            });
        });
    }
], function (err, result) {
    if (err) {
        console.log(err);
    } else {
        console.log('done');
    }
});

3 Comments

Hi @Ben, thanks for helping. I have tried this and I am still receiving a Maximum call stack size exceeded error.
Hey Ben, I have posted my own solution here, but thanks for your answer because you really improved my understanding of how I was using callbacks and async.waterfall!
Glad you found the problem. I knew the error you had was something else. That was the reason why I didn't put the correct callbacks for waterwall as an answer to your question because it didn't... even though the waterfall flow and callbacks need to be corrected.
0

I have found the answer to my own question here. The issue was that when I was bulk inserting data into MongoDB, I was receiving the Maximum call stack size exceeded error. The array of data I was using when running db.collection('something').insert(data, function().... was an array of Users according to a Mongoose Schema called User, so each item in this data array was created as var item = new User({});. Apparently, MongoDB did not like this. I assumed it would be the correct way to insert the data as in production this is how I would be using the data (using User objects).

To rectify the issue, I simply replaced var item = new User({}) with var item = {} so that I was just creating a new JavaScript object. This worked fine and inserted the data into the database, in the correct order, with no errors. Thanks to the people who wrote other answers, I have used those to fix and improve my callbacks!

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.