mongoexport with parameters + node.js + child process

Question

I am trying to export a csv using node.js from mongodb. For this i started with this code:

app.get('/export', function(req, res) {
 var spawn = require('child_process').spawn,
 ls = spawn('mongoexport');
res.sendfile('/home/database.csv'); 
});

And this works fine. Then for making it more usable i tried to code below with mongoexport using arguments:

 app.get('/export', function(req, res) {
 var spawn = require('child_process').spawn,
 ls = spawn('mongoexport --db lms --collection databases --fields firstname,lastname,email,daytimePhone,addressOne,city,state,postalCode,areaOfStudy,currentEducationLevel,company --csv --out /home/database.csv');
res.sendfile('/home/database.csv') 

});

This throws a exception:

events.js:72
        throw er; // Unhandled 'error' event
              ^
Error: spawn ENOENT
    at errnoException (child_process.js:980:11)
    at Process.ChildProcess._handle.onexit (child_process.js:771:34)

Then i tried using only one argument but it gives the same error :(

I tried this also to see if arguments work this way but the same error:

spawn('mongoexport',['--csv']);

Ben · Accepted Answer · 2013-11-30 16:22:35Z

5

The syntax for spawn is:

spawn(<command>, [array of arguments]);

For example, doing a ls command with -l /home options would look like this:

ls = spawn('ls', ['-l', '/home'];

So your spawn('mongoexport',['--csv']); is heading in the right direction but mongoexport --csv is not valid. That's why you're getting error. mongoexport needs more than just --csv. Like what you've done above, you, for instance, need to specify database name (-d "lms"), collection name (-c "databases"), field names (--fields firstname,lastname), and etc.

In your case, it should be something like this:

 var spawn = require('child_process').spawn;
 app.get('/export', function(req, res) {
     var mongoExport = spawn('mongoexport', [ 
         '--db', 'lms', '--collection', 'databases', 
         '--fields',
         'firstname,lastname,email,daytimePhone,addressOne,city,state,postalCode,areaOfStudy,currentEducationLevel,company',   
         '--csv'
     ]);

     res.set('Content-Type', 'text/plain');
     mongoExport.stdout.on('data', function (data) {
         if (data) {
             // You can change or add something else here to the
             // reponse if you like before returning it.  Count
             // number of entries returned by mongoexport for example
             res.send(data.toString());
         } else {
             res.send('mongoexport returns no data');
         }
     });
}

edited Nov 30, 2013 at 16:22

answered Nov 24, 2013 at 15:54

Ben

5,0842 gold badges21 silver badges23 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

beNerd Over a year ago

ok the csv gets generated correctly...but i guess this line gets executed before the generation has finished: res.sendfile('/home/database.csv'). Any callback for this?

beNerd Over a year ago

This just gives me a download of a file called "export" with no parameters applied :(

Ben Over a year ago

beNerd, I currently don't have the environment to test the code you have above and see what it's doing in the debugger. I'll do that when I have some cycle. In the mean time, try setting Content-Type before calling res.senfile(). Something like this: res.set('Content-Type', 'text/plain');

Ben Over a year ago

@beNerd, as robertklep said, creating /home/database.csv would introduce race condition. So avoid it is good. One way is to pipe the output to res like robertklep suggested. Another way is to add event handler to 'data' event. This approach allows you to add something before after the output in the response. You can event do something like altering the result, count number of entries and etc. Since robertklep already put down the pipe solution. I'll modify my answer to handle 'data' event.

beNerd Over a year ago

this goes in the non-terminating condition and nginx timesout!

|

robertklep · Accepted Answer · 2013-11-24 16:18:15Z

0

Your code has a race condition, where /home/database.csv could get truncated/overwritten by multiple requests for /export that come in (almost) simultaneously.

Using the code that Ben provided, I would pipe the output of mongoexport directly to the response object, thus not needing to use a temporary file:

app.get('/export', function(req, res) {
  // Set correct content-type
  res.set('Content-Type', 'text/csv');
  // Export collection and pipe to `res`
  spawn('mongoexport', [ 
    '--db', 'lms', '--collection', 'databases', 
    '--fields', 'firstname,lastname,email,daytimePhone,addressOne,city,state,postalCode,areaOfStudy,currentEducationLevel,company', 
    '--csv'
  ]).stdout.pipe(res);
});

Personally I would generate the CSV directly from Node using a MongoDB driver and csv.

answered Nov 24, 2013 at 16:18

robertklep

205k38 gold badges421 silver badges412 bronze badges

5 Comments

beNerd Over a year ago

may i know why i should not take this approach and generate the csv directly as suggested by you?

robertklep Over a year ago

@beNerd I'm not suggesting you shouldn't, just that I probably wouldn't do it myself :) (spawning new processes for a request takes quite some resources, but if it's not being called that often that probably doesn't matter)

beNerd Over a year ago

i tried the method of ben but it just gives me a download of a file called "export" with no parameters applied :(

robertklep Over a year ago

@beNerd start with a simple script first, see this gist. See what that does.

beNerd Over a year ago

it just says "connected to 127.0.0.1" in the terminal and nothing else

Collectives™ on Stack Overflow

mongoexport with parameters + node.js + child process

2 Answers 2

6 Comments

5 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

6 Comments

5 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related