JavaScript heap out of memory while updating the mongodb

Question

I am trying to synchronize data between to data stores, the source is mssql and the destination is MongoDB. In this syncing process I am getting a memory heap error. I am not sure why this happens and I am fully aware that the following code may not be the best, but for now I am just trying to understand why the allocation error is coming.

I am compiling my code with babel, in development I am just using babel-node.

try {
  const response = await sqlDataStore.findAll({
    attributes: ['id', 'Name'],
  });
  /* eslint no-restricted-syntax: 0 */
  for (const item of response) {
    /* eslint no-await-in-loop: 0 */
    await this.Model.updateOne({}, item, { upsert: true });
  }
} catch (err) {
  console.log(err);
}

If I understand correctly the heap error is caused by the for loop, so that would mean that every await statement is cached in the memory. I would have expected that every await statement is cleared from the memory because I am not assigning it to any variable.

Updated:

Gladly I found already a solution due to another post: Bulk upsert in MongoDB using mongoose

My Code:

  const response = await sqlDataStore.findAll({
    attributes: ['id', 'Name'],
  });

  const bulkUpdate = response.map(doc => ({
    updateOne: {
      filter: { _id: doc.id },
      update: doc.dataValues,
      upsert: true,
    },
  }));

  this.Model.collection.bulkWrite(bulkUpdate);

If someone is using this solution it should be kept in mind that this also could crash for lots amount of data. The solution provided in the other posts suggests that the data should be processed in buckets of 1000 until every document is updated/inserted.

Just for interest and technical understanding I would appreciate an explanation of what exactly I did wrong in my first code.

Just to be sure, does the code run fine with just the first SQL query? I.e., it's not that it's just too huge an amount of data being brought in? — jameslol
– jameslol, Commented May 3, 2019 at 5:25
yea it is definitely fine for the sql query and the amount of data is also not that big there are just 2401 rows. — Matt
– Matt, Commented May 3, 2019 at 5:58
Hi I understand that the operations are delayed but that is not my question, the question is if they are indeed stored in the memory and if that is expected behavior even though I am not storing the response in any variable. — Matt
– Matt, Commented May 3, 2019 at 6:27
I guess this is because your execution is completely blocked the next execution and will be in memory until your entire execution is complete. Check the link I have shared. — Atul Agrawal
– Atul Agrawal, Commented May 3, 2019 at 6:28

Atul Agrawal · Accepted Answer · 2019-05-03 07:23:14Z

2

You are getting this because your function call stack is not getting free since they are waiting for other calls to complete its execution.

Since all the call stacks blocked your stack memory and hence after some executions you are getting out of memory exception.

Check this link: https://eslint.org/docs/rules/no-await-in-loop

As you can see your await call blocked in memory for other await to finish and they are returning you the value in one go which is bad in your code.

It is actually you are making a sync call and each sync call waiting for other sync calls to finish and finally, your sync calls get pile up in stack memory and you are getting the exception .

edited May 3, 2019 at 7:23

answered May 3, 2019 at 6:36

Atul Agrawal

1,5387 gold badges24 silver badges48 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

JavaScript heap out of memory while updating the mongodb

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related