1

I'm new with EMR-serverless and I want to know how to pass, in a spark application, jar and packages as for example:

spark-submit --deploy-mode client --jars /usr/lib/hudi/hudi-spark3.3-bundle_2.12-0.11.1-amzn-0.jar,/usr/lib/hudi/hudi-utilities-bundle_2.12-0.11.1-amzn-0.jar ...

I want to set when I submit a job but I cannot find a way about how to do it.

Can someone help me with this, please?

Improve this question

1 Answer 1

Reset to default
2

When submitting a job to EMR Serverless in the console and you want to provide additional options to spark-submit, you can use the "Spark properties" section. Instead of --jars, you can use the spark.jars key and set the value appropriately.

Your Spark application will be a Python script or JAR file on S3 provided as the "Script location" aka entrypoint.

Also note that Hudi is available on the EMR Serverless image and there's some documentation on using Hudi with EMR Serverless.

Screenshot of Serverless console with Spark properties

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.