How to get query s3 bucket by specific file size

Question

at the moment there's some files being uploaded where they are getting corrupted. They'll have a filesize of 0 bytes. May I ask how do I query my s3 bucket and filter by specific size, i'm trying to query when byte is 0?

At the moment I have two queries.

First one list all the files recursively in the bucket but no sorting.

aws s3 ls s3://testbucketname --recursive --summarize --human-readable

Second one sorts but only when provided a prefix, in my case the prefix is the folder name. My current bucket structure is as followed {accountId}/{filename}

aws s3api list-objects-v2 --max-items 10 --bucket testbucketname --prefix "30265"  --query "sort_by(Contents,&Size)"

30265 is the accountId/folder name. When the prefix isn't provided, the sort doesn't quite work.

Any help would be greatly appreciated.

This query works well for filtering the name which is a string

aws s3api list-objects --bucket testbucketname --query "Contents[?contains(Key, '.jpg')]"

Unfortunately I couldn't use contains for Size and there isn't a equals.

I would recommend using S3 inventory and query it with Athena. — jellycsc
– jellycsc, Commented May 9, 2022 at 14:20
You could also potentially use Lambda to alert you in real time to objects being uploaded with zero size (which is inherently an indication that your client sent zero bytes). — jarmod
– jarmod, Commented May 9, 2022 at 14:42
Do you just mean something like aws s3api list-objects-v2 --bucket example-bucket --query 'Contents[?Size==`0`].Key' (Use " instead of ' on Windows)? — Anon Coward
– Anon Coward, Commented May 9, 2022 at 14:59
@AnonCoward exactly that, TY! Did you want to post an answer — Master
– Master, Commented May 9, 2022 at 15:26

Anon Coward · Accepted Answer · 2022-05-09 16:09:18Z

4

You can use the --query logic to filter the list objects locally to only those that are zero-byte big:

aws s3api list-objects-v2 --bucket example-bucket --query 'Contents[?Size==`0`]'

Or, if you just want to see the list of keys without other meta-data, you can further filter the list:

aws s3api list-objects-v2 --bucket example-bucket --query 'Contents[?Size==`0`].Key'

(For both of these, replace the outer ' with " when running on Windows.)

Further, if the goal is the remove these objects, you can use jq and a subshell to construct a query that deletes the targeted objects:

aws s3api delete-objects --bucket example-bucket --delete \
"$(aws s3api list-objects-v2 --bucket example-bucket --query 'Contents[?Size==`0`].Key' |\
 jq '{"Objects": map({"Key":.})}')"

There isn't a direct way to do this same sort of construct with Windows's command interpreter.

edited May 9, 2022 at 16:09

answered May 9, 2022 at 16:02

Anon Coward

11k3 gold badges31 silver badges42 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to get query s3 bucket by specific file size

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related