How to copy files in S3 from folder to another recursively using bash script?

Question

I have a folder structure like this in my AWS S3 bucket named myfiles:

mainfolder/
|-- current
`-- error
    `-- files
        |-- files1
        `-- files2

I'm trying to copy all the files under files1 and files2 folders to current, so that resulting structure shows like this:

mainfolder/
|-- current
|   |-- files1
|   `-- files2
`-- error
    `-- files
        |-- files1
        `-- files2

I'm trying to do this using simple bash for loop like:

BUCKET=myfiles
for dir in $(aws s3 ls s3://$BUCKET/mainfolder/error/files/ | awk '{print $2}'); do
    aws s3 cp $i s3://$BUCKET/mainfolder/current/$i
done

but I'm getting errors:

The user-provided path 0 does not exist.

How this could be done?

Try aws s3 sync s3://bkt/mainfolder/error/files/ s3://bkt/mainfolder/current/ — jarmod
– jarmod, Commented Nov 6, 2018 at 17:35

John Rotenstein · Accepted Answer · 2018-11-06 20:04:17Z

2

You can use:

aws s3 sync s3://my-bucket/mainfolder/error/files/ s3://my-bucket/mainfolder/current/

or:

aws s3 cp --recursive s3://my-bucket/mainfolder/error/files/* s3://my-bucket/mainfolder/current/

See: sync — AWS CLI Command Reference

answered Nov 6, 2018 at 20:04

John Rotenstein

273k28 gold badges456 silver badges541 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to copy files in S3 from folder to another recursively using bash script?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related