2

I have a script that I load via:

python = new PythonInterpreter();
String script = "script I loaded from somewhere"    
    
try {
    python.exec(script);
} catch (Exception e) {
    AtiWarning.msg(host, "eaPY", "Error parsing script: " + e, e);
}

This seems to "compile" the script.

The script contains a function that I then call many times for a given host via:

try {
    python.set("host", host);
    python.set("tableName", tableName);
    python.set("state", order);
    python.set("index", index);
            python.set("values", values);

    python.exec("row(host, tableName, state, index, values)");  

} catch (Exception e) {
    AtiWarning.msg(host, "eaPY", "Error in row: " + e, e);
}

This entire process may be repeated for other hosts. Multiple hosts may need to execute the script concurrently.

After the first exec(script), is there any way to save some kind of object (such as PyCode?) so when another host comes along I do not have to call exec(script) again?

Due to concurrency, I cannot simply hold on to the PythonInterpreter instance and reuse it. I would like to create a new instance of PythonInterpeter and give it some object rather than having it parse the script again.

3
  • I've added the java tag as it seems appropriate. Commented Nov 6 at 14:29
  • is the script being executed by the "host" what gives access to your local variables (host, tableName etc.)? i.e. if you are sharing the function does it need to be called before you set locals? Commented Nov 6 at 15:06
  • was my answer below sufficient? If so please mark as the accepted answer to help others @Robert Commented Nov 10 at 20:17

1 Answer 1

0

Underneath your code snippet demonstrating PythonInterpreter('some string...') you say

This seems to "compile" the script.

No, this executes the script. Which is why you haven't been able to reuse the instance.


Per the docs: PythonInterpreter.compile('some string...')

Compiles a string of Python source as either an expression (if possible) or a module.

Your question has two parts. Focusing on the easier question, without getting too into the specifics of your setup:

The script contains a function that I then call many times [...] Is there a way to save some kind of object (PyCode?) so when another "host" comes along I do not have to call exec(script) again?


Yes. Based on the Jython docs, you should be first compiling the string into your PyCode object via: PyCode my_func= new PythonInterpreter.compile('some string expression or module') that you can then pass around to local threads. Which brings us to the second and slightly more difficult question

This entire process may be repeated for other hosts. Multiple hosts may need to execute the script concurrently. [...] Due to concurrency, I cannot simply hold on to the PythonInterpreter instance and reuse it. I would like to create a new instance of PythonInterpeter and give it some object rather than having it parse the script again.

You will need to manage local threads for each host via threadLocalStateInterpreter -- then simply replace your python.set('env_var',env_var) with a thread-safe python.setLocals(). Something like

import org.python.core.*;
import org.python.util.PythonInterpreter;
import java.util.concurrent.*;

public class JythonRemoteExecutor {
    
    // shared, thread-safe, compiled Python function
    private static PyCode compiledFunction;
    
    // thread-local interpreter for each connecting host
    private static ThreadLocal<PythonInterpreter> threadLocalStateInterpreter = 
        ThreadLocal.withInitial(() -> {
            PythonInterpreter interp = new PythonInterpreter();
            return interp;
        });

    // compile your Python function once at startup
    static {
        String pythonCode = "def your_function():...";
        PythonInterpreter compiler = new PythonInterpreter();
        compiledFunction = compiler.compile(pythonCode);
        compiler.close();
    }
    
    /* execute the shared function for a specific host */
    public static PyObject executeForHost(String hostId) {

        // this is where you begin to take advantage of that precompiled 
        // code from earlier
        PythonInterpreter interp = threadLocalStateInterpreter.get();
        // execute the compiled function in this thread's namespace,
        // this will allow you to use `your_function()` locally,
        // independent of other hosts who may be executing their
        // instance of your function
        interp.exec(compiledFunction);
        
        // set local variables specific to this host using setLocals()
        interp.setLocals(new PyStringMap() {{
            __setitem__("host_id", new PyString(hostId));
            __setitem__("host", host);
            __setitem__("tableName", tableName);
            __setitem__("state", state);
            __setitem__("index", index);
            __setitem__("values", values);
        }});
        
        interp.exec("row(host, tableName, state, index, values)");
    }
    
    /**
     * example driver program simulating multiple hosts connecting and executing
     */
    public static void main(String[] args) throws InterruptedException {
        ExecutorService executor = Executors.newFixedThreadPool(3); // set as needed
        Callable<String> host1Task = () -> {
            PyObject result = executeForHost("Host-1");
            return "Host-1 finished executing."
        };
        
        Callable<String> host2Task = () -> {
            PyObject result = executeForHost("Host-2");
            return "Host-2 finished executing."
        };
        
        Callable<String> host3Task = () -> {
            PyObject result = executeForHost("Host-3");
            return "Host-3 finished executing."
        };
        
        // execute all host connections concurrently
        Future<String> future1 = executor.submit(host1Task);
        Future<String> future2 = executor.submit(host2Task);
        Future<String> future3 = executor.submit(host3Task);
        
        // print results -- should see all hosts finish executing
        try {
            System.out.println(future1.get());
            System.out.println(future2.get());
            System.out.println(future3.get());
        } catch (ExecutionException e) {
            e.printStackTrace();
        }
        
        executor.shutdown();
    }
}

Sign up to request clarification or add additional context in comments.

4 Comments

Thank you. Question: If each host has an instance of PythonInterpreter where I called exec(compiledFunction), why do I need to use ThreadLocal? The reality is that I only have a single thread. It is making asynchronous calls to all of the hosts walking through the table on each host. The hosts answer when they answer. So only a single instance of the interpreter is ever being executed at one time. Eventually all of the tables finish their rows.
"If each host has an instance of PythonInterpreter where I called exec(compiledFunction) ... " the PythonInterpreter instance inside of each host all point to the same object; they share the interpreter that is managed by a thread executor, ensuring that each host can respond in their own time and that they never attempt to access the interpreter while it is being used by another host. this way you are only ever getting that compiled function once, and then sharing access to its usage between each host
think of it like this: when your program starts, it grabs your interpreter. your program then waits for a host to ask for access to that global interpreter, which already contains the function you "loaded from elsewhere". the host then asks for the global interpreter to execute that function which has now been brought into their local namespace (so to avoid writing their data to another host's output). the host then tells your interpreter they're done and disconnects.
Are you saying that no matter how many times I call new PythonInterpreter(); it is always the same instance? My assumption is that each "new" creates an instance with its own local namespace. Each namespace can take the same instance of the previously compiled code and run it with no interference to each other.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.