4

Update April 2020: I have accepted the answer below because it proposes a very good and simple solution to my problem but I never got my code to work! If anyone has already built something similar please contact me!


I have a simple script my-script.py that does some calculations very fast using a specific library written in Python:

import sys
import special-library

def fun(x):
  do something

fun(sys.argv[1])

In my Java code, I use/call this script a lot in different part of the code, using the ProcessBuilder approach, which means that I simply run the command python my-script.py argument. This means that every time I call this script, I have to execute the import commands which is the most time-consuming thing in this (the calculation part is actually faster).

Is there a solution so that I call the import just once? I looked a little bit about Jython - would be possible to write a class or something that would be initiated and do the import once and then calling a function of it every time I want to do the calculations part (fun)? Has anyone done something similar or have any other solution to this?

First Attempt at Implementation

I've tried to write a 'pipe' Java Class that will execute the python script once. Now the script reads continuously from the stdin and when it gets an input argument, it does the calculations and returns the result:

import sys
import special-library

def fun(x):
  do something
  print('END')

for arg in sys.stdin:
  fun(arg)

And this works of course when I test it from the command line and supplying it with input arguments.

The Java Class is as follows:

package my.package;
import java.io.*;

public class PythonScriptExecuter {

    private static BufferedReader pythonToJavaReader;
    private static PrintWriter javaToPythonWriter;

    public PythonScriptExecuter() throws Exception {
        String MPBNScriptFile = "fullPathToScriptFile";
        ProcessBuilder pb = new ProcessBuilder("python", MPBNScriptFile);
        pb.redirectErrorStream(true);

        // Start script
        Process p = pb.start();

        pythonToJavaReader = getOutputReader(p); // from python to java
        javaToPythonWriter = getInputWriter(p); // from java to python
    }

     // Python => Java
    private static BufferedReader getOutputReader(Process p) {
        return new BufferedReader(new InputStreamReader(p.getInputStream()));
    }

    // Java => Python
    private static PrintWriter getInputWriter(Process p) {
        return new PrintWriter(new OutputStreamWriter(p.getOutputStream()));
    }

    public static Arraylist<String> getResults(String argument) throws IOException {
        // send argument to python script
        javaToPythonWriter.println(argument);
        //javaToPythonWriter.flush();

        // get results back one by one
        ArrayList<String> results = new ArrayList<>();

        int count = 0;
        String line;
        while (!(line = pythonToJavaReader.readLine()).equals("END")) {
            // process `line` string 
            results.add(line);
        }

        return results;
    }
}

So, what I do is that somewhere in the initialization part of my code I have this simple line to start the process:

new MPBNPythonScriptExecuter();

and later when I want to get some results back, I use:

String arg = "something"
Arraylist<String> res = PythonScriptExecuter.getResults(arg);

and the whole thing hangs on the reading-the-output-from-the-script line:

while (!(line = pythonToJavaReader.readLine()).equals("END"))

Any ideas what is going wrong here?

2
  • Did you verify that your python script did not have an error? Need to catch the error also with something like this while ((ligne = error.readLine()) != null) { System.out.println("err: "+ ligne); } Commented Apr 13, 2020 at 6:37
  • No, it's more like it does not read properly somehow (at the right time maybe)... when I kill the java program, sometimes I can see the output inside the while loop (I print the line) - but only then - like it was stucked or something... Commented Apr 13, 2020 at 13:09

2 Answers 2

7

You could communicate between java and Python with pipes.

You run your Python as you do now (without command line args)

Python script

you write an infinite loop in python that will

  1. read data from the standard input (it will be your arg for the function)

  2. You call your function

  3. you write the answer to the standard output

Java

  1. Write a method for sending args to python

  2. write to the pipe the arg of your function

  3. read the answer from the pipe

You're done.

Here is some snippet

Here how you create your tube

        Process p = Runtime.getRuntime().exec(commande);
        BufferedReader output = getOutput(p); //from python to java
        BufferedReader error = getError(p); //from python to java
        PrintWriter input  = getInput(p); //from java to python

private static BufferedReader getOutput(Process p) {
    return new BufferedReader(new InputStreamReader(p.getInputStream()));
}
private static BufferedReader getError(Process p) {
    return new BufferedReader(new InputStreamReader(p.getErrorStream()));
}    
private static PrintWriter getInput(Process p){
    return new PrintWriter (new OutputStreamWriter(p.getOutputStream()));
}
Sign up to request clarification or add additional context in comments.

6 Comments

That sounds really promising! I will work on a solution based on that and will come back at you!
You will also avoid .exec() it will be done only one time, like import
There cannot be parallel requests for this function of the script in this case, right? E.g. two process call the function at the same time (of the same process p) and since the script writes to stdout, you don't know which results will belong to which. Maybe synchronized in Java will help in this case, will check.
Hi Pascal! I ran into a problem and editted the question with my current approach. Can you check it if possible?
If you need a parallel call, it will then be better to make a client/server approach, but I think it will be to much complicated for your need I think. The solutions with an IO pipe is a sequential you need to make it safe and add synchronized to your getResults method
|
1

Try the following in your Python script:

import sys

def fun(x):
  print(x)
  print('END')
  sys.stdout.flush()

while 1:
    try:
        line = sys.stdin.readline()
    except KeyboardInterrupt:
        break

    if not line:
        break

    fun(line)

Your Java code should be more or less correct.

1 Comment

Thanks for the effort! The problem still persists and it mostly has to do with the Java part I think.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.