0

I have the following code:

for (var i = 0; i < files.length; i++) {
    var data = fs.readFileSync(files[i]);
    fs.appendFileSync("file_part", data);
}

Let's say I have 1000 files. Is it the for loop async? Or My application will wait until it will finish the process?

If it sync, how I make it async? so each iterate will be async?

Edit:

This code is part of a server code, using express. We get a lot of https request, and serves the client. One of the operation is to appends files of 1MB to one big file. The server should return 200 if the operation success, 500 otherwise.

The problem is that if the code above (the for loop) is executed, The express module stop serving other clients, and it "busy" with this request. I try to use async.eachSeries (because the order of the files has a meaning, so I have to use Series method), but it still stucks the server.

Improve this question
9
  • 1
    It's synchronous because you're using the synchronous methods. If you want to make it async, read the documentation and use the asynchronous methods instead. Commented Feb 23, 2015 at 19:59
  • If you want to make it asynchronous, do you really want to open 1000 files in parallel? Or do you want them to execute sequentially? If so, what's wrong with the solution that you already have? Commented Feb 23, 2015 at 20:01
  • 2
    You seem to have a stray ) there. Does this code even run? Commented Feb 23, 2015 at 20:03
  • similar: stackoverflow.com/questions/28087624/… Commented Feb 23, 2015 at 21:04
  • @Bergi: I edit the question and explain what wrong with my solution. Commented Feb 24, 2015 at 12:06

3 Answers 3

Reset to default
1

All control structures in JS are executed synchronously. Since you have used no asynchronous functions in the loop body (but the explicitly synchronous counterparts), it will wait for them to finish their tasks.

To make the snippet non-blocking, you need to use the regular asynchronous fs methods, and chain them appropriately. I would recommend promises, but if you are already using , then it should look like this:

async.eachSeries(files, function(file, cb) {
    fs.readFile(file, function(err, data) {
        if (err) return cb(err);
        fs.appendFile("file_part", data, cb);
    });
}, function(err) {
    // call back to express here with 500 if err and 200 otherwise
});
Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

My problem is that I need to append 1000 files of 1 MB to one file of size 1GB. The code in my question "stuck" my server, until all the appending process is finished. So my problem is the I need to read the file synchronicity (Because the order of the bytes is important, but I don't want to block it). I try to use async.eachSeries, but it also stuck my server.
@MiddleWare: Have you used async.eachSeries with the synchronous functions?
0

If all you want to do is to concat a list of files together asynchronously you can just do it in the callback to fs.readFile:

function concatFiles (source_array, destination) {
    var source = source_array.shift();

    fs.readFile(source,function(err,data){
        fs.appendFile(destination,data,function(err){
            if (source_array.length) {
                concatFiles(source_array, destination);
            }
        });
    });
}

// Usage:
concatFiles(files,"file_part");

Note that for clarity I didn't handle errors in the code above. In production code you should check for errors and handle them appropriately.

It's fairly obvious what's going on. Pop off the first file in the source list. Then read that file asynchronously. Then when that's done write to the destination file asynchronously. Repeat until source list is empty.

Improve this answer

Comments

0

There are several problems in both the question and what you're trying to do. You may want to either clarify the question or clarify what you really want to do.

The short answer to your question: No. A for loop is not, itself, asynchronous.

A slightly longer answer: It isn't clear you need the loop itself to be async, but you may just want the contents to be async. So instead of calling readFileSync you may want to use readFile which takes a callback. In this callback, you can append to the file. This will "fire off" 1000 read files, which will call their callback when ready.

But there is still a problem. If you read the 1000 files asynchronously, and save them to the same file as you read them in, you won't get the file contents in any order and may even have file content interleaving. I can't imagine why you actually want to do this.

Update: Your updated question is still a little vague - if you need to wait until everything is done to send the code 200 reply... then you're going to have to wait until all the files are appended. There really isn't any other solution.

If, however, you can return code 200 immediately to indicate you're working on the file appends, and then do the appending in the background, you can execute the loop as part of an asynchronous task. You can do this inside a process.nextTick() call or in a promise or in any other way that makes sense to your flow.

Improve this answer

3 Comments

My problem is that I need to append 1000 files of 1 MB to one file of size 1GB. The code in my question "stuck" my server, until all the appending process is finished. So my problem is the I need to read the file synchronicity (Because the order of the bytes is important, but I don't want to block it). I try to use async.eachSeries, but it also stuck my server.
Can you clarify, possibly in the original question, if this file needs to be returned as part of the process and why having the server pause to do this work is unreasonable?
updated answer to reflect what you need to consider regarding replying to the user.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.