amazon-s3 upload fails for upload files in loop

Question

Trying to upload 40k files to s3 from nodejs and aws-sdk I do promisify every thing code works fine upto 5800 records but after that it fails rest of files are not uploaded can anyone help why this happen so or i have to use some another approch for this or do i need to updation in my code?

Here is my code

public static convertClobAndUploadToS3(allClobAttachments: Array<RfAttachment>):
  Promise<FileConvertS3UploadResponse> {
  return new Promise(async (resolve, reject) => {
  const fileConvertResp: FileConvertS3UploadResponse = { processed: [], unprocessed: [] };
  for (let i = 0; i < allClobAttachments.length; i++) {
    const rfAttachment = allClobAttachments[i];
    const mainFileBufferFormat: Buffer = Buffer.from(rfAttachment.CLOB);
    const path = CONST.S3.PATH.MAIN_FILE;
    const fileName = rfAttachment.RF_ID + CONST.S3.EXT.MAIN_FILE;
     // upload only single file to s3 at a time and returns url
    const url = await GFIUtils.uploadFileToS3(config, fileName, mainFileBufferFormat, path);
    url ? fileConvertResp.processed.push({ RFID: rfAttachment.RF_ID, url, rfAttachment })
    : fileConvertResp.unprocessed.push(rfAttachment.RF_ID);
    if (fileConvertResp.processed.length === allClobAttachments.length) {
      logger.info(`CLOB STAGE::: All clob successfully uploaded!!! TOTAL CLOB:::${allClobAttachments.length}`);
      return resolve(fileConvertResp);
    } else if (fileConvertResp.unprocessed.length + fileConvertResp.processed.length
       === allClobAttachments.length) {
      logger.info(`allClobAttachments::: ${allClobAttachments.length}`);
      logger.info(`processed::: ${fileConvertResp.processed.length}`);
      logger.info(`unprocessed::: ${fileConvertResp.unprocessed.length}`);
      return reject(fileConvertResp);
    } else { continue; }
  }
  });
 }

that's not the right way to upload so many records. Could you tell us more if getting any error? — Sandeep Patel
– Sandeep Patel, Commented Feb 20, 2020 at 14:31
Also, you should not load all these files in memory. Use streams, process files in batch. Also, check what are all the limitations AWS s3 put when you upload multiple files. — Sandeep Patel
– Sandeep Patel, Commented Feb 20, 2020 at 14:36

rohanmehto2 · Accepted Answer · 2020-02-20 14:44:05Z

3

Instead of doing this take a more safer approach and open up a subprocess using aws cli.

Use aws s3 sync or copy command to do this.

It would be lot faster as well as more fail proof then a brute force approach like calls inside a for loop.

In case if all the files are not in a single or some limited number of directories. You should generate a list of paths of the file you want to upload. Write that text file to disk. Use aws cli to upload or download using the paths from the text file.

answered Feb 20, 2020 at 14:44

rohanmehto2

9601 gold badge7 silver badges19 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

amazon-s3 upload fails for upload files in loop

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related