Async python script from node js

Question

I have a node.js program that runs around 50 different python script instances. I would like to be able to throttle the phase - such that at any one time, only 4 processes will run parallel.

I tried a simple loop that calls a function that runs 4 instances of the python script. I used 60 seconds delay as this is the average time takes the script to run.

function startPythonScraping(){
  for(var i = 0; i < finalList.length; i++){
    setTimeout(function(){
        runFourProccesses(finalList[i]);
    }, i*60*1000);
   }
}


function runFourProccesses(stockSymbol){
    console.log("working on " + stockSymbol);
    for(var j = 0; j < 4; j++){
        if(j == 0){
            trueOrFalse = "y"
        } else {
            trueOrFalse = "n"
        }
    let secondPythonProcess = spawn('python',     ["/Users/nybgwrn/Desktop/AlphaSecWebsite/getAllData.py",     stockSymbol, (j*10).toString(), "10", trueOrFalse]);
  secondPythonProcess.stdout.on('data', (data) => {
    let messegeFromPython = JSON.stringify(data.toString('utf8')).replace("\\n", "");
            console.log(messegeFromPython + " with stock " + stockSymbol);


            
            if(messegeFromPython != "something went wrong"){
                //console.log("created file for " + symbol);
            } else {
                //console.log("couldn't create file for " + symbol + "_" + (j*10).toString() + "-" + (j*10 + 10).toString());
            }
        });
    }  
}

It doesn't work because somehow the index i begin with 50 instead of 0, and also I want a better solution as I want to be SURE that only 4 instances are running.

fafl · Accepted Answer · 2018-09-17 11:47:31Z

2

This is usually done using semaphores. A semaphore is basically a pool of locks, so that multiple locks can be held at the same time.

In your case you could use this package and wrap each python process spawning with sem.take and sem.leave.

answered Sep 17, 2018 at 11:47

fafl

7,5074 gold badges32 silver badges54 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Async python script from node js

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related