5

For some user requests, I need to do some heavy computations (taking about 100ms of time); and, of course, I do not want to perform these in node's main event loop so that it will prevent other requests from serving.

The most obvious, but definitely not the cleanest way to solve this problem would be to offload the computations to another program and wait for the results asynchronously.

Is there any way to do it without leaving the node process (and implementing inter-process communications)? E.g. something like this:

var compute = function (input) {
    var i,
        result = input;
    for (i = 0; i < 1000000; i++) {
        result = md5(result);
    }
}

var controller = function (req, res) {
    offload(compute, req.params.input, function(result) {
        res.send(result);
    });
}
6
  • 1
    github.com/xk/node-threads-a-gogo seems to be what I'm looking for, except for threads-a-gogo being only available on *nix (while I'm looking for a way to do the same on Windows). Commented May 14, 2012 at 10:22
  • 1
    Use nodejs.org/api/child_process.html? That I believe still does IPC, which I think all solutions will have to do Commented May 15, 2012 at 5:43
  • Yes, i'm aware of the child_process - basically it's the only way to instantiate external processes, so i don't understand why did you put "still" in your message. I'd like to do it without implementing IPC and writing another program though (especially considering the computation results might be partially cached); basically, the only way to do it right with the IPC is to run another node cluster with a lot of workers (more than # of cores) and to do IPC-over-http. Commented May 15, 2012 at 5:52
  • With child_process you would use IPC for processes to communicate with each other? Commented May 15, 2012 at 5:54
  • I must somehow transmit the input data to the child process and read the output data after it finished processing. It involves some means of serialization, which i would like to avoid. Also, in order to enable caching of the intermediate results, I must set up the child process to be the ever-running node http server (or another cluster). Commented May 15, 2012 at 6:04

2 Answers 2

4

I found a compute-cluster module, which seems to be designed exactly for my goals. It spawns children processes, and the communication is made with IPC (no sockets wasted).

Sign up to request clarification or add additional context in comments.

3 Comments

Consider updating your question to reflect that you are OK using child processes and IPC through a sufficiently convenient library.
Well the question title asks about "threads" and the body specifically asks "is there any way to do it without leaving the node process...". So it might lead people to believe that separate processes and IPC are not considered in this discussion. Maybe change the title to "offload heavy computations from the node event-loop" or similar?
Fixed the title. AFAIK, the only difference between threads and processes in Node besides the API difference is lower memory requirements and startup time for threads (because javascript does not support shared memory approaches).
1

The JavaScript language has no support for threads and node.js provides no interface to them either so you'll need to rely on a library to provide threading functionality. See the node-fibers and node-threads-a-go-go libraries for possible solutions.

If you're looking for a way to make IPC a little easier in node.js then consider using the zeromq library (node.js bindings). This somewhat-related-answer might be useful as well.

1 Comment

Unfortunately, node-threads-a-go-go are not supported on Windows. It seems that isolates would do a job, but they won't be there even in v0.8. I'll give zeromq a try, but it doesn't seem to make IPC significantly easier

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.