Flink Job submission throws java.nio.file.NoSuchFileException while the file actually exists

Question

I tried to submit a flink job that is already packaged in a JAR. Basically it consumes a kafka topic protected by SASL authentication, thus it requires a .jks file which I already include them in JAR and read in the code as:

try(InputStream resourceStream = loader.getResourceAsStream(configFile)){
        properties.load(resourceStream);
        properties.setProperty("ssl.truststore.location",
            loader.getResource(properties.getProperty("ssl.truststore.location")).toURI().getPath());
      }

catch(Exception e){
        System.out.println("Failed to load config");
      }

I tried to submit the job on two different (different VM specs) standalone server for the sake of testing. One server runs succesfully, but another throw a java.nio.file.NoSuchFileException, saying that my .jks file is not found. Can someone please point out the possible issue on it?

Here, the flink is deployed on a standalone cluster mode with the following version:

Flink version: 1.14.0
Java version: 11.0.13

Please include the stacktrace that shows the pathname that is being used to >open< the .jks file. (My guess would be that the pathname is relative, and your application is in the wrong directory to open it.) — Stephen C
– Stephen C, Commented Dec 13, 2021 at 0:46
Sorry. Yeah, you're correct stephen. I actually thought that using .toUri().getPath() will resolve to the correct path, but instead it returns null. Thus the config being used is instead the one loaded from the original configFile, thats why is resolved to relative path. Sorry again for this stupid question. — hilmialf
– hilmialf, Commented Dec 13, 2021 at 2:10

hilmialf · Accepted Answer · 2021-12-13 02:41:38Z

I realize my question was really silly. This part actually returns null and trigger exception.

loader.getResource(properties.getProperty("ssl.truststore.location")).toURI().getPath()

The problem was that I submit the job through web UI thus I couldn't see the printed message. Thus, the filename resolves to the original one stored under the configFile, which is a relative path. Why one machine works and another one doesn't? Cause I previously somehow has the .jks on my homedir for another testing :).

For others to not jump into this mistake, here is the summary of what will .getResource() resolve if run from IDE (gradle run task) and jar, respectively.

//      file:home/gradle-demo/build/resources/main/kafka-client.truststore.jks
//      jar:file:home/gradle-demo/build/libs/gradle-demo-1.0-SNAPSHOT.jar!/kafka-client.truststore.jks
      System.out.println(loader.getResource("kafka-client.trustore.jks").toString());

//      home/gradle-demo/build/resources/main/kafka-client.truststore.jks
//      file:home/gradle-demo/build/libs/gradle-demo-1.0-SNAPSHOT.jar!/kafka-client.truststore.jks
      System.out.println(loader.getResource("kafka-client.trustore.jks").getPath());

//      home/gradle-demo/build/resources/main/kafka-client.truststore.jks
//      null
      System.out.println(loader.getResource("kafka-client.trustore.jks").toURI().getPath());

//      file:home/gradle-demo/build/resources/main/kafka-client.truststore.jks
//      jar:file:home/gradle-demo/build/libs/gradle-demo-1.0-SNAPSHOT.jar!/kafka-client.truststore.jks
      System.out.println(loader.getResource("kafka-client.trustore.jks").toURI());

Leo · Accepted Answer · 2022-04-20 06:52:21Z

0

kafka-client:2.4.1 org.apache.kafka.common.security.ssl.SslEngineBuilder#285

try (InputStream in = Files.newInputStream(Paths.get(path))) {
    KeyStore ks = KeyStore.getInstance(type);
    // If a password is not set access to the truststore is still available, but integrity checking is disabled.
    char[] passwordChars = password != null ? password.value().toCharArray() : null;
    ks.load(in, passwordChars);
    return ks;
  } catch (GeneralSecurityException | IOException e) {
    throw new KafkaException("Failed to load SSL keystore " + path + " of type " + type, e);
}

It looks like we should put jks file in file system(nfs or hdfs) where task manager can access by absolute path.

answered Apr 20, 2022 at 6:52

Leo

1

1 Comment

Community Over a year ago

As it’s currently written, your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center.

Collectives™ on Stack Overflow

Flink Job submission throws java.nio.file.NoSuchFileException while the file actually exists

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related