Using S3A client in EMR serverless

Question

I am using a S3 compatible object store (CloudFlare R2) and trying to get EMR serverless to connect to it. R2 requires that you use the correct endpoint and pass the secret key and access key.

In the local machine, running spark works perfectly fine. But when running it on EMR serverless it does not respect the params I try passing to it. To be precise, fs.s3a.access.key and fs.s3a.secret.key.

I've tried setting it in the application, as part of job from the console and as --conf in spark parameters, but it does not work.

Any ideas on how to go about this is really appreciated, thanks!

stevel · Accepted Answer · 2023-01-31 12:40:50Z

0

use the EMR s3 connector -it's the one they support

answered Jan 31, 2023 at 12:40

stevel

13.6k1 gold badge41 silver badges54 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Using S3A client in EMR serverless

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related