aws-sdk Multipart Upload to s3 with node.js

Question

I am trying to upload files to a s3 bucket using the node js aws-sdk V3.

I know I am supposed to be using the commmands: CreateMultipartUploadCommand,UploadPartCommandand so forth. But I can't find any working example of a full multipart upload.

Can anyone share any code samples?

Thanks in advance

kbanman · Accepted Answer · 2021-04-10 03:38:59Z

3

I put together a bit of code for this: https://gist.github.com/kbanman/0aa36ffe415cdc6c44293bc3ddb6448e

The idea is to upload a part to S3 whenever we receive a chunk of data in the stream, and then finalize the upload when the stream is finished.

Complicating things is S3's minimum part size of 5MB on all but the last part in the series. This means that we need to buffer data until we can form those 5MB chunks. I accomplished this using a transformer that adds back-pressure on the content stream between each chunk upload.

Parallelization is also made difficult by the fact that S3 insists on receiving the parts in order (despite asking for the parts to be numbered).

answered Apr 10, 2021 at 3:38

kbanman

4,2716 gold badges34 silver badges40 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

aws-sdk Multipart Upload to s3 with node.js

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related