Read multiple files from directory with Node.js using Promises/Async/Await

Question

I'm trying to scan a local directory for all of the text files within it, then read each of those files. I want to do it properly using promises, async, and await, but something is going wrong and am stuck trying to figure it out.

I am able to read the directory contents and output all of the filenames correctly, but when I try to read those files themselves using the map method, I get a Promise { <pending> } log for each of the files. Also, I get a UnhandledPromiseRejection error at the end.

I can read a single file fine outside of the map function, but can't loop through all of the files and read them without getting an error.

import fs from 'fs';
import path from 'path';
import { fileURLToPath } from 'url';

const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);

const textFileDir = `${__dirname}/files/`;

const readDirPro = (dir) => {
  return new Promise((resolve, reject) => {
    fs.readdir(dir, (error, files) => {
      if (error) reject('Could not find directory');
      resolve(files);
    });
  });
};

const readFilePro = (file) => {
  return new Promise((resolve, reject) => {
    fs.readFile(file, 'utf-8', (err, data) => {
      if (err) reject('Could not find file');
      resolve(data);
    });
  });
};

const readAllFiles = async () => {
  try {
    const dirFilesArr = await readDirPro(textFileDir);
    // correctly outputs array of file names in directory
    // console.log('dirFilesArr: ', dirFilesArr);

    // THIS IS THE PROBLEM:
    await dirFilesArr.map((file) => {
      // console.log(file);
      // correctly outputs filenames
      const fileContent = readFilePro(file);
      // console.log(fileContent);
      // incorrectly outputs "Promise { <pending> }" logs for each file
    });
  } catch (err) {
    console.log(`catch (err): ${err}`);

    throw err;
  }
  return '2: final return';
};

(async () => {
  try {
    console.log('1: readAllFiles!');
    const x = await readAllFiles();
    console.log(x);
    console.log('3: Done!');
  } catch (err) {
    console.log(`IIFE catch ERROR 💥: ${err}`);
  }
})();

Here's is the full console output for that javascript, with the `Promise { } logs commented out:

1: readAllFiles!
2: final return
3: Done!
node:internal/process/promises:289
            triggerUncaughtException(err, true /* fromPromise */);
            ^

[UnhandledPromiseRejection: This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). The promise rejected with the reason "Could not find file".] {
  code: 'ERR_UNHANDLED_REJECTION'
}

Node.js v19.0.1

I would prefer to keep the working solution as close to this code as possible, because it's is actually one piece of a larger puzzle, and I need to be able to continue utilizing this structure.

My ultimate goal is to read all of those files, pull certain data from them, then write that data into a new JSON file.

UPDATE: Check this answer below for the most exact solution to the question: https://stackoverflow.com/a/75206022/3787666

readFilePro() returns a pending promise. That's what it does. The caller has to use await or .then() on that promise to know when it's done and get the result. — jfriend00
– jfriend00, Commented Jan 22, 2023 at 6:39
Do you realize that promise versions already exist for the fs module? See fsPromises.readFile() in the doc. — jfriend00
– jfriend00, Commented Jan 22, 2023 at 6:51
thanks, @jfriend00, but where does that await go? I have one before the map method here, and if I try it like this const fileContent = await readFilePro(file);, then I get an "Unexpected reserved word" error — david
– david, Commented Jan 22, 2023 at 7:01
As for fsPromises.readFile(), are you saying I could use that instead of the readFilePro function here? — david
– david, Commented Jan 22, 2023 at 7:04
Yes, it's built into the fs module. You don't have to make your own promisified versions. — jfriend00
– jfriend00, Commented Jan 22, 2023 at 7:11

RAllen · Accepted Answer · 2023-01-22 08:35:07Z

NodeJS has promisified version of fs, so technically you can achieve the desired results with the following code:

import fs from 'fs';
import path from 'path';
import { fileURLToPath } from 'url';

const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);

const textFileDir = `${__dirname}/files/`;

(async () => {
  try {
    console.log('1: readAllFiles!');
    const dirFilesArr = await fs.promises.readdir(textFileDir);
    const files = await Promise.all(dirFilesArr.map((file) => {
        const fullPath = path.join(textFileDir, file);
        return fs.promises.readFile(fullPath, { encoding: 'utf-8' });
    }));
    console.log(files);
    console.log('3: Done!');
  } catch (err) {
    console.log(`IIFE catch ERROR 💥: ${err}`);
  }
})();

If you prefer to fix your original code, you will need to make a few changes. First change is needed because readdir doesn't return file names with full pathes and as a result, you need to add a path to each file to read a file. Therefore, your readFilePro should look something like so:

const readFilePro = (file) => {
  return new Promise((resolve, reject) => {
    fs.readFile(path.join(textFileDir, file), 'utf-8', (err, data) => {
      if (err) reject('Could not find file');
      resolve(data);
    });
  });
};

The second change is that your readAllFiles needs to be changed like so:

const readAllFiles = async () => {
  try {
    const dirFilesArr = await readDirPro(textFileDir);
    return Promise.all(dirFilesArr.map((file) => readFilePro(file)));
  } catch (err) {
    console.log(`catch (err): ${err}`);

    throw err;
  }
  return '2: final return';
};

In your original code, you run readFilePro for each file and retrieve a promise that you never wait. The approach above fixes the problem.

Please let me know if this helps.

Thank you so much! Providing two solutions even. I was just looking at handling the map method with a Promise.all and that turns out to be a key part of the solution.

david · Accepted Answer · 2023-01-24 02:57:22Z

0

Building on the accepted answer above, here's a supplemental answer, which addresses a key aspect of my question. Rather than returning all of the data in in one big chunk from the map, I really needed to be able to manipulate data from each file as soon as it is read (why I had the const fileContent in my question above). So, this is how I handle that now, by using an async/await for the callback inside of the map method. I still need the Promise.all method with the return if I want to pass const files data to a function following the Promise.all.

const files = await Promise.all(
      dirFilesArr.map(async (file, i) => {
        const fullPath = path.join(textFileDir, file);
        const fileContent = await fs.promises.readFile(fullPath, {
          encoding: 'utf-8',
        });
        //console.log(fileContent); // can now view file data individually
        await externalFunction(fileContent, i); // manipulating file data
        // return fs.promises.readFile(fullPath, { encoding: 'utf-8' });
      })
    );

edited Jan 24, 2023 at 2:57

answered Jan 23, 2023 at 5:50

david

2735 silver badges22 bronze badges

1 Comment

david Over a year ago

@RAllen Your super helpful answer got me to this point. Thanks again

Collectives™ on Stack Overflow

Read multiple files from directory with Node.js using Promises/Async/Await

2 Answers 2

1 Comment

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related