Making multiple mongoose calls in parallel using async

Question

Been struggling with this for a while, and I seem to be missing something quite obvious. So as an example. Lets say that I want to make three calls in parallel. To make it an easy example, or easy to test, lets say we want a script that asked the following information from mongodb.

We want to find the following information: - We want to list ns of each index - We want to list users that have an _id - We want to list ns of each index that have a key (Yes, you are right, this is a ridiculous set of queries, but they return something on EVERY mongo database.)

So a simple script that would do this on a mongo db is:

var mongoose = require("mongoose");
var morgan = require('morgan');             // log requests to the console (express4)
var async = require('async');
mongoose.connect('mongodb://localhost/admin');



var Indexes = mongoose.model('system.indexes', {
    name: String,
    ns: String
});
var Users = mongoose.model('system.users', {
    user: String
})


        Indexes.find().exec(function(err, docs){
            for( var doc in docs ){
                console.log("Got Index " + docs[doc].ns);
            }
        })
        Users.find({ _id: {$ne: null}}).exec(function(err, docs){
            for (var doc in docs){
                console.log("And user " + docs[doc].user );
            }
        })
        Indexes.find({ key : {$ne : null}}).exec(function(err, docs){
            for (var doc in docs) {
                console.log("And Index with a key " + docs[doc].ns);
            }
        })

But how would I go about it if I wanted to use async. So tried the following to run them in parallel:

var mongoose = require("mongoose");
var morgan = require('morgan');             // log requests to the console (express4)
var async = require('async');
mongoose.connect('mongodb://localhost/admin');



var Indexes = mongoose.model('system.indexes', {
    name: String,
    ns: String
});
var Users = mongoose.model('system.users', {
    user: String
})


/// So lets do it with async Map instead
var queries = [];
queries.push(Indexes.find().exec());
queries.push(Users.find({ _id: {$ne: null}}).exec());
queries.push(Indexes.find({ key : {$ne : null}}).exec());

async.parallel(queries, function(err, docs){
    console.log("I've done some stuff");
})

This gives me an error statement of

task(_restParam(function (err, args) { ^ TypeError: object is not a function

But is there a simple way around this. How do I go about making multiple Find using either async map or parallel.

===== Following Jigars excellent reply, we are closer to the answer ===== Jigars excellent answer now woks, keeping the stuff below If someone has the same issue

/// So lets do it with async Map instead
var queries = [];
queries.push(function (cb) {
    try {
        result = Indexes.find().exec();
        cb(null, result);
    } catch(e) {
        cb(e);
    }
})

queries.push(function (cb) {
    try {
        result = Users.find({ _id: {$ne: null}}).exec();
        cb(null, result);
    } catch(e) {
        cb(e);
    }
})

queries.push(function (cb) {
    try {
        result = Indexes.find({ key : {$ne : null}}).exec();
        cb(null, result);
    } catch(e) {
        cb(e);
    }
})

async.parallel(queries, function(err, docs) {
    // if any query fails
    if (err) {
        throw err;
    }

    var res1 = docs[0]; // result of queries[0]
    for (var opts in res1){
        console.log("In res1 we got " + res1[opts])
    }
    var res2 = docs[1]; // result of queries[1]
    for (var opts in res2){
        console.log("In res2 we got " + res2[opts])
    }
    var res3 = docs[2]; // result of queries[2]
    for (var opts in res3){
        console.log("In res3 we got " + res3[opts])
    }
})

So the parallel works now, just not returning what I wanted. The result in res1, res2 and res3 looks like js code:

Jigar Jain · Accepted Answer · 2015-08-11 11:40:50Z

5

Your queries.push command should be in following way

queries.push(function (cb) {
    Indexes.find().exec(function (err, docs) {
        if (err) {
            throw cb(err);
        }

        // do some stuff with docs & pass or directly pass it
        cb(null, docs);
    });
})    

queries.push(function (cb) {
    Users.find({ _id: {$ne: null}}).exec(function (err, docs) {
        if (err) {
            throw cb(err);
        }

        // do some stuff with docs & pass or directly pass it
        cb(null, docs);
    });
})

queries.push(function (cb) {
    Indexes.find({ key : {$ne : null}}).exec(function (err, docs){
        if (err) {
            throw cb(err);
        }

        // do some stuff with docs & pass or directly pass it
        cb(null, docs);
    });
})

async.parallel(queries, function(err, docs) {
    // if any query fails
    if (err) {
        throw err;
    }

    var res1 = docs[0]; // result of queries[0]
    var res2 = docs[1]; // result of queries[1]
    var res3 = docs[2]; // result of queries[2]
})

async takes multiple functions with callback function as argument. You need to call the cb once every function is completed. Also you need to pass the result value of each query. The same will given to you back in array form when all the queries are finished executing in parallel

edited Aug 11, 2015 at 11:40

answered Aug 11, 2015 at 10:44

Jigar Jain

1,4472 gold badges15 silver badges38 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

vrghost Over a year ago

Really nice stuff, it is a lot closer to working than before. However, the res1,2 and 3 contains java code not the responses. Took your example and uptdated it in my question. And something comes back but it seems (I may be wrong) that it believes that the code is the complete product from the function rather than the return.

Jigar Jain Over a year ago

Updated the answer based on your edit. Idea is to complete all the async functions & pass the final output through cb(). You can go through async documentation which has various nice examples.

vrghost Over a year ago

So now a sneaky follow up, how on earth do I send a variable into the push. Tried with bind but getting errors...

Arvind Pal · Accepted Answer · 2021-07-02 13:13:54Z

0

We can user async/await for this as below code.

let queries = [];
    
      userIds.forEach(userId => {
        queries.push(async()=>await Users.find({_id:userId}))
      });
    
      Promise.all( queries).then(function(result) {
          // result is an array of responses here
          console.log("result", result);
          return result;
      }).catch(function(err) {
          console.log(err);
          return ;
      });

answered Jul 2, 2021 at 13:13

Arvind Pal

5213 silver badges16 bronze badges

Collectives™ on Stack Overflow

Making multiple mongoose calls in parallel using async

2 Answers 2

3 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related