AWS EMR 4.0 - How can I add a custom JAR step to run shell commands

Question

I am trying to run shell commands using steps on EMR 4.0.0 and used this link for reference - http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-hadoop-script.html

But I want to know from where to put 'command-runner.jar' in 'JAR location' field https://i.sstatic.net/CRicz.png

I kept 'command-runner.jar' in AWS s3 and tried to load it from that location and in 'Arguments' gave s3 location of my 'example.sh' file and after adding step it failed giving this exception

Exception in thread "main" java.lang.RuntimeException: java.io.IOException: Cannot run program "s3://<path>/example.sh" (in directory "."): error=2, No such file or directory
    at com.amazonaws.emr.command.runner.ProcessRunner.exec(ProcessRunner.java:139)
    at com.amazonaws.emr.command.runner.CommandRunner.main(CommandRunner.java:13)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.io.IOException: Cannot run program "s3://<path>/example.sh" (in directory "."): error=2, No such file or directory
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047)
    at com.amazonaws.emr.command.runner.ProcessRunner.exec(ProcessRunner.java:92)
    ... 7 more
Caused by: java.io.IOException: error=2, No such file or directory
    at java.lang.UNIXProcess.forkAndExec(Native Method)
    at java.lang.UNIXProcess.<init>(UNIXProcess.java:187)
    at java.lang.ProcessImpl.start(ProcessImpl.java:130)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1028)
    ... 8 more

so you point from the Jar Location ? did you try the CLI from the document you refer in your question ? — Frederic Henri
– Frederic Henri, Commented Dec 8, 2015 at 8:57
no it should not matter, but if console does not work, its good to see if CLI works so if its general error or an error with the use of console — Frederic Henri
– Frederic Henri, Commented Dec 8, 2015 at 10:06
@Frédéric It didn't work with CLI as well!!!! Are you sure I am supposed to locate command-runner.jar from s3? — Sohan Malegaonkar
– Sohan Malegaonkar, Commented Dec 8, 2015 at 12:01

Mauricio De Diana · Accepted Answer · 2016-03-02 23:11:55Z

1

To run a shell script stored in S3, use script-runner.jar, which is located at s3://elasticmapreduce/libs/script-runner/script-runner.jar.

command-runner.jar will work for local files only.

answered Mar 2, 2016 at 23:11

Mauricio De Diana

1595 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Sohan Malegaonkar Over a year ago

Thank you Mauricio De Diana, I have already followed aws documentation to Run a Script in a Cluster link but it did not work out for me. So I created my own script-runner jar and placed it on S3 and executed my job. Thank you again

Collectives™ on Stack Overflow

AWS EMR 4.0 - How can I add a custom JAR step to run shell commands

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related