1

we got error response {"errors":["Internal server error: null"]} when uploading jar to Flink 1.5.0 server using java REST client. The same code work properly in Flink 1.4.2. In fact we can see the jar has been uploaded from Flink GUI. But the wrong status break logic. Any advice please?

        HttpPost uploadFile = new HttpPost(flinkJobManagerUrl + "/jars/upload");
        MultipartEntityBuilder builder = MultipartEntityBuilder.create();
        builder.addBinaryBody(
                "jarfile",
                new FileInputStream(f),
                ContentType.create("application/x-java-archive"),
                f.getName()
        );

        HttpEntity multipart = builder.build();
        uploadFile.setEntity(multipart);
        CloseableHttpResponse response = restClient.execute(uploadFile);
        rd = new BufferedReader(new InputStreamReader(response.getEntity().getContent()));

==========================

Error from JM log:

018-06-01 16:05:46,514 WARN org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Unhandled exception org.apache.flink.shaded.netty4.io.netty.handler.codec.http.multipart.HttpPostRequestDecoder$EndOfDataDecoderException at org.apache.flink.shaded.netty4.io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.hasNext(HttpPostMultipartRequestDecoder.java:366) at org.apache.flink.shaded.netty4.io.netty.handler.codec.http.multipart.HttpPostRequestDecoder.hasNext(HttpPostRequestDecoder.java:241) at org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:92) at org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:51) at org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) at org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242) at org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:147) at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) at org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:847) at org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) at java.lang.Thread.run(Thread.java:748)


But after awhile you will see the jar uploaded from GUI. But then when run it it will give another exception:

2018-06-01 16:10:06,752 ERROR org.apache.flink.runtime.webmonitor.handlers.JarRunHandler - Exception occurred in REST handler. org.apache.flink.runtime.rest.handler.RestHandlerException: Expected only one value [--KAFKA_IN [email protected]:9092, 192.168.56.121:9092, 192.168.56.122:9092/a_O_124 --ZK_SESSION_TIMEOUT 60000 --ZK_KEEP_CONN_ALIVE 1]. at org.apache.flink.runtime.rest.handler.util.HandlerRequestUtils.getQueryParameter(HandlerRequestUtils.java:56) at org.apache.flink.runtime.rest.handler.util.HandlerRequestUtils.getQueryParameter(HandlerRequestUtils.java:44) at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.handleRequest(JarRunHandler.java:102) at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.handleRequest(JarRunHandler.java:68) at org.apache.flink.runtime.rest.handler.AbstractRestHandler.respondToRequest(AbstractRestHandler.java:77) at org.apache.flink.runtime.rest.AbstractHandler.respondAsLeader(AbstractHandler.java:168) at org.apache.flink.runtime.rest.handler.RedirectHandler.lambda$null$0(RedirectHandler.java:139) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442) at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357) at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) at org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) at java.lang.Thread.run(Thread.java:748)

2
  • Could you share the cluster entrypoint/jobmanager logs with us. They should contain the problem. Commented Jun 1, 2018 at 15:52
  • error updated above, any clue @TillRohrmann? thank you Commented Jun 1, 2018 at 16:14

2 Answers 2

1

The first exception is due to a bug in the FileUploadHandler; it doesn't properly handle EmptyLastHttpContent messages. See https://issues.apache.org/jira/browse/FLINK-9500.

The second exception is caused by having spaces within the program-arguments, which as of 1.5 is not supported anymore.

Sign up to request clarification or add additional context in comments.

4 Comments

Thanks @ChesnaySchepler, any idea when the bug will be fixed?
1.5.1 i suppose, but we don't have a date yet. The jar should still be uploaded correctly, so you should be able to safely ignore it.
but according to 1.5.0 doc, the program-arguments looks no change :( section "From the command line arguments" ci.apache.org/projects/flink/flink-docs-release-1.5/dev/…
That's for the CLI. The submission fails because the REST API parses query parameters differently now since it actually cares about commas (allowing multiple values for a single query parameter).
0

about the second exception, you can put your programArgs with comma into the post request body like

{
  "programArgs":"--test=a,b"
}

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.