Apache beam on google cloud dataflow

Question

I'm trying to run a job on Google Cloud Dataflow but can't get the deployment to work. The job runs fine using DirectRunner but as soon as I switch over to dataflow-runner I get the following exception:

[WARNING]
java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: No Runner was specified and the DirectRunner was not found on the classpath.
Specify a runner by either:
    Explicitly specifying a runner by providing the 'runner' property
    Adding the DirectRunner to the classpath
    Calling 'PipelineOptions.setRunner(PipelineRunner)' directly
        at org.apache.beam.sdk.options.PipelineOptions$DirectRunner.create(PipelineOptions.java:286)
        at org.apache.beam.sdk.options.PipelineOptions$DirectRunner.create(PipelineOptions.java:276)
        at org.apache.beam.sdk.options.ProxyInvocationHandler.returnDefaultHelper(ProxyInvocationHandler.java:575)
        at org.apache.beam.sdk.options.ProxyInvocationHandler.getDefault(ProxyInvocationHandler.java:516)
        at org.apache.beam.sdk.options.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:155)
        at org.apache.beam.sdk.options.PipelineOptionsValidator.validate(PipelineOptionsValidator.java:70)
        at org.apache.beam.sdk.runners.PipelineRunner.fromOptions(PipelineRunner.java:44)
        at org.apache.beam.sdk.Pipeline.create(Pipeline.java:138)
        at my.package.SalesTransactions.main(SalesTransactions.java:218)

My command to run:

mvn compile exec:java -Dexec.mainClass=my.package.SalesTransactions -Dexec.args="--runner=DataflowRunner --project=my-project --tempLocation=gs://my-project/tmp" -Pdataflow-runner

Can you share your pom.xml or confirm that you have added a dependency on the Dataflow runner? — Ben Chambers
– Ben Chambers, Commented Apr 5, 2017 at 18:50
I've added a pom.xml extract showing the maven profile used. — jimmy
– jimmy, Commented Apr 5, 2017 at 22:13

Jimmy · Accepted Answer · 2017-04-06 14:10:30Z

2

Found my mistake. When switching over from DirectRunner to DataFlowRunner I had to add the dependency in my pom.xml instead of just having it as a profile runtime dependency.

<dependency>
      <groupId>org.apache.beam</groupId>
      <artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
      <version>${beam.version}</version>
    </dependency>

answered Apr 6, 2017 at 14:10

Jimmy

1751 gold badge1 silver badge13 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Apache beam on google cloud dataflow

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related