5

I am getting below error while running dataflow job. I am trying to update my existing beam version to 2.11.0 but I am getting below error at run time.

java.lang.IncompatibleClassChangeError: Class org.apache.beam.model.pipeline.v1.RunnerApi$StandardPTransforms$Primitives does not implement the requested interface com.google.protobuf.ProtocolMessageEnum at org.apache.beam.runners.core.construction.BeamUrns.getUrn(BeamUrns.java:27) at org.apache.beam.runners.core.construction.PTransformTranslation.(PTransformTranslation.java:58) at org.apache.beam.runners.core.construction.UnconsumedReads$1.visitValue(UnconsumedReads.java:49) at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:666) at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:649) at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:649) at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:649) at org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:311) at org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:245) at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:458) at org.apache.beam.runners.core.construction.UnconsumedReads.ensureAllReadsConsumed(UnconsumedReads.java:40) at org.apache.beam.runners.dataflow.DataflowRunner.replaceTransforms(DataflowRunner.java:868) at org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:660) at org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:173) at org.apache.beam.sdk.Pipeline.run(Pipeline.java:313) at org.apache.beam.sdk.Pipeline.run(Pipeline.java:299)

Improve this question
9
  • Do you have the same exception when running on direct runner? Commented Mar 13, 2019 at 21:03
  • Hi Rui, yes its running fine with direct runner. I am getting this error only while running with DataflowRunner Commented Mar 14, 2019 at 17:46
  • Hi Rui, just adding one more point here - my code is working fine till Beam version 2.5.0. It breaks as soon as i change it to 2.6.0 or next versions with same error. Commented Mar 14, 2019 at 18:09
  • Tagging @RuiWang for notification Commented Mar 14, 2019 at 20:04
  • Clarification - when you say "update your pipeline" you mean an in-place update of the running pipeline, yes? Commented Mar 25, 2019 at 17:53

3 Answers 3

Reset to default
2

I don't have enough rep to leave a comment, but I ran into this issue and later figured out that my problem was that I had different beam versions in my pom.xml. Some had 2.19 and some had 2.20.

I would do a quick search of your versions in the pom or gradle file to make sure they are all the same.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

2

This may be caused by incompatible dependencies. I successfully upgraded beam from 2.2.0 to 2.20.0 by upgrading the dependencies at the same time.

beam.version: 2.20.0
guava.version: 29.0-jre
bigquery.version: v2-rev20191211-1.30.9
google-api-client.version: 1.30.9
google-http-client.version: 1.34.0
pubsub.version: v1-rev20200312-1.30.9
Improve this answer

Comments

1

This usually means that the version of com.google.protobuf:protobuf-java that Beam was built with does not match the version at runtime. Does your pipeline code also depend on protocol buffers?

UPDATE: I have filed https://issues.apache.org/jira/browse/BEAM-6839 to track this. It is not expected.

Improve this answer

3 Comments

No my pipeline code doesn't have any protocol buffers.
See update. I also notice that 2.7 and beyond have protobuf updated compared to 2.5 and below. Did you recompile or are you running the same binaries with new dependencies?
Can you reply on the bug with some steps to reproduce if you can share them?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.