Several ways for java to call python files

The opportunity for java to call python comes from a project that needs to use an algorithm, but the python written by the algorithm engineers, so there is a need for the java backend to call python scripts, and many problems have been encountered in the middle, so I will record and sort it out once.

1. What are the ways of calling python from java

1.1 Method 1: jpython

The class library developed specifically for calling python2 from java, but because the python3 version is not supported, and the syntax between python2 and 3 is not compatible, the jpython library is not particularly universal. Someone on github asked when the python3 version of the library will be released. The official answer is that it is feasible but very difficult (as of August 2022, the jpython official has not yet developed a class library that supports python3)

The syntax of jpython is very simple, and python files can be easily manipulated by using PythonIntercepter.

1.1.1 Import jar package

 <dependency>
  <groupId>org.python</groupId>
  <artifactId>jython-standalone</artifactId>
  <version>2.7.0</version>
</dependency>

1.1.2 Call the method1() method in the python script

 PythonInterpreter interpreter = new PythonInterpreter();
interpreter.execfile("C:\\Users\\Dick\\Desktop\\demo.py");
// 调用demo.py中的method1方法
PyFunction func = interpreter.get("method1",PyFunction.class);
Integer a = 10;
Integer b = 10;
PyObject pyobj = func.__call__(new PyInteger(a), new PyInteger(b));
System.out.println("获得方法的返回值 = " + pyobj.toString());

Note: If there is no return value, just execute the interpreter.execfile() method

1.2 Method 2: ProcessBuilder

ProcessBuilder is a script execution tool class provided by jdk. Whether it is a python file, a shell script or other instructions, it can be executed through this class. Let's see how it calls a python script.

1.2.1 First we put the python file under resource

1.2.2 The next step is to execute the script

 /**
 * 执行python脚本
 * @param fileName 脚本文件名称
 * @param params 脚本参数
 * @throws IOException
 */
public static void execPythonFile(String fileName, String params) throws IOException {
 
  // 获取python文件所在目录地址
  String windowsPath = ClassUtils.getDefaultClassLoader().getResource("").getPath().substring(1) + "py/"; 
 
  // windows执行脚本需要使用 cmd.exe /c 才能正确执行脚本
  Process process = new ProcessBuilder("cmd.exe", "/c", "python", windowsPath + fileName, params).start();
 
  logger.info("读取python文件 开始 fileName={}", fileName);
  BufferedReader errorReader = null;
  // 脚本执行异常时的输出信息
  errorReader = new BufferedReader(new InputStreamReader(process.getErrorStream()));
  List<String> errorString = read(fileName, errorReader);
  logger.info("读取python文件 异常 fileName={}&errorString={}", fileName, errorString);
 
  // 脚本执行正常时的输出信息
  BufferedReader inputReader = null;
  inputReader = new BufferedReader(new InputStreamReader(process.getInputStream()));
  List<String> returnString = read(fileName, inputReader);
  logger.info("读取python文件 fileName={}&returnString={}", fileName, returnString);
 
  try {
      logger.info("读取python文件 wait fileName={}", fileName);
      process.waitFor();
  } catch (InterruptedException e) {
      logger.error("读取python文件 fileName="+fileName+" 等待结果返回异常", e);
  }
  logger.info("读取python文件 fileName={} == 结束 ==", fileName);
}
private static List<String> read(String fileName, BufferedReader reader) {
        List<String> resultList =  Lists.newArrayList();
        String res = "";
        while (true) {
            try {
                if (!((res = reader.readLine()) != null)) break;
            } catch (IOException e) {
                logger.error("读取python文件 fileName=" + fileName + " 读取结果异常", e);
            }
            resultList.add(res);
        }
        return resultList;
}

The above code only considers windows, but the situation will be more complicated in Linux.

1.2.3 Existing problems in executing python in Linux

We know that the conventional project deployment is to package the project into a jar package, and then directly put it into Linux or deploy it through a container such as docker. At this time, the py file under resources is in the jar package, but when we execute the python script, we use :

The location of the python3 script file,

At this time, the python script is in the jar package and cannot be accessed through the jar path /BOOT-INF/classes/py/xxx.py [I have tested for a while and found that python3 (python instructions do not work) instructions cannot call scripts in the jar] , so the solution I can think of is to put the python script file directly into a folder on the server for subsequent access. If it is a docker deployment, you only need to add a COPY command to the dockerfile to put the py file in the specified directory:

1.2.4 Executing python files in Linux

The following code will be compatible with windows and linux to call the py file [Linux executes the py file using python or python3 according to the actual py environment variable configuration to choose]

 /**
* 执行python文件
* @param fileName python文件地址
* @param params 参数 其实可以改成传入多个参数 一个个放入ProcessBuilder中的
* @throws IOException
*/
public static void execPythonFile(String fileName, String params) throws IOException {
  // ① 当前系统类型
  String os = System.getProperty("os.name");
 
  // ② 获取python文件所在目录地址
  String windowsPath = ClassUtils.getDefaultClassLoader().getResource("").getPath().substring(1) + "py/";  
  String linuxPath = "/ai/egcc/";
 
  logger.info("读取python文件 init fileName={}&path={}", fileName);
  Process process;
  if (os.startsWith("Windows")){
      // windows执行脚本需要使用 cmd.exe /c 才能正确执行脚本
      process = new ProcessBuilder("cmd.exe", "/c", "python", windowsPath + fileName, params).start();
  }else {
      // linux执行脚本一般是使用python3 + 文件所在路径
      process = new ProcessBuilder("python3", linuxPath + fileName, params).start();
  }
 
  logger.info("读取python文件 开始 fileName={}", fileName);
  BufferedReader errorReader = null;
  // 脚本执行异常时的输出信息
  errorReader = new BufferedReader(new InputStreamReader(process.getErrorStream()));
  List<String> errorString = read(fileName, errorReader);
  logger.info("读取python文件 异常 fileName={}&errorString={}", fileName, errorString);
 
  // 脚本执行正常时的输出信息
  BufferedReader inputReader = null;
  inputReader = new BufferedReader(new InputStreamReader(process.getInputStream()));
  List<String> returnString = read(fileName, inputReader);
  logger.info("读取python文件 fileName={}&returnString={}", fileName, returnString);
 
  try {
      logger.info("读取python文件 wait fileName={}", fileName);
      process.waitFor();
  } catch (InterruptedException e) {
      logger.error("读取python文件 fileName="+fileName+" 等待结果返回异常", e);
  }
  logger.info("读取python文件 fileName={} == 结束 ==", fileName);
}
private static List<String> read(String fileName, BufferedReader reader) {
    List<String> resultList =  Lists.newArrayList();
    String res = "";
    while (true) {
        try {
            if (!((res = reader.readLine()) != null)) break;
        } catch (IOException e) {
            logger.error("读取python文件 fileName=" + fileName + " 读取结果异常", e);
        }
        resultList.add(res);
    }
    return resultList;
}

Do you think it’s over? In fact, there are some problems with the process.waitFor() method. If it goes online, it may cause accidents. For details, please refer to: java calls exe program to use process.waitFor() deadlock

Then let's try to use the thread pool to solve the deadlock problem.

1.2.5 Solve the implicit problem of java calling script file

The following is the ultimate version code:

 private static ExecutorService taskPool = new ThreadPoolExecutor(8, 32
        ,200L,TimeUnit.MILLISECONDS, new LinkedBlockingQueue<Runnable>(600)
        ,new ThreadFactoryBuilder()
        .setNameFormat("thread-自定义线程名-runner-%d").build());
/**
* 执行python文件
* @param fileName python文件地址
* @param params 参数 多个直接逗号隔开
* @throws IOException
*/
public static void execPythonFile(String fileName, String params) throws IOException {
    // ① 当前系统类型
    String os = System.getProperty("os.name");
  
    // ② 获取python文件所在目录地址
    String windowsPath = ClassUtils.getDefaultClassLoader().getResource("").getPath().substring(1) + "py/";  
    String linuxPath = "/ai/egcc/";
  
    logger.info("读取python文件 init fileName={}&path={}", fileName);
    Process process;
    if (os.startsWith("Windows")){
        // windows执行脚本需要使用 cmd.exe /c 才能正确执行脚本
        process = new ProcessBuilder("cmd.exe", "/c", "python", windowsPath + fileName, params).start();
    }else {
        // linux执行脚本一般是使用python3 + 文件所在路径
        process = new ProcessBuilder("python3", linuxPath + fileName, params).start();
    }
 
    taskPool.submit(() -> {
        logger.info("读取python文件 开始 fileName={}", fileName);
        BufferedReader errorReader = null;
        // 脚本执行异常时的输出信息
        errorReader = new BufferedReader(new InputStreamReader(process.getErrorStream()));
        List<String> errorString = read(fileName, errorReader);
        logger.info("读取python文件 异常 fileName={}&errorString={}", fileName, errorString);
    });
 
    taskPool.submit(() -> {
        // 脚本执行正常时的输出信息
        BufferedReader inputReader = null;
        inputReader = new BufferedReader(new InputStreamReader(process.getInputStream()));
        List<String> returnString = read(fileName, inputReader);
        logger.info("读取python文件 fileName={}&returnString={}", fileName, returnString);
    });
    
    try {
        logger.info("读取python文件 wait fileName={}", fileName);
        process.waitFor();
    } catch (InterruptedException e) {
        logger.error("读取python文件 fileName="+fileName+" 等待结果返回异常", e);
    }
    logger.info("读取python文件 fileName={} == 结束 ==", fileName);
}
private static List<String> read(String fileName, BufferedReader reader) {
    List<String> resultList =  Lists.newArrayList();
    String res = "";
    while (true) {
        try {
            if (!((res = reader.readLine()) != null)) break;
        } catch (IOException e) {
            logger.error("读取python文件 fileName=" + fileName + " 读取结果异常", e);
        }
        resultList.add(res);
    }
    return resultList;
}

Well, the above code can already call the python script correctly, but the blogger still has some problems to solve: for example, how to call the py file inside the java jar package? The py file in the jar package on windows can be called successfully [I have tested the jar package locally on windows], but the jar in the docker container cannot be called successfully. What is the reason?

If anyone encounters problems, please leave a message and discuss in the comment area

1.2.6 The ultimate python execution tool class [recommended use]

 import com.google.common.collect.Lists;
import com.google.common.util.concurrent.ThreadFactoryBuilder;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Component;
import org.springframework.util.ClassUtils;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.util.List;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
 
/**
 * java调用python的执行器
 */
@Component
public class PythonExecutor {
    private static final Logger logger = LoggerFactory.getLogger(PythonExecutor.class);
    private static final String OS = System.getProperty("os.name");
 
    private static final String WINDOWS_PATH = ClassUtils.getDefaultClassLoader().getResource("").getPath().substring(1) + "py/automl/";  // windows为获取项目根路径即可
    private static final String LINUX_PATH = "/ai/xx";// linux为python文件所在目录
 
    private static ExecutorService taskPool = new ThreadPoolExecutor(8, 16
            , 200L, TimeUnit.MILLISECONDS, new LinkedBlockingQueue<Runnable>(600)
            , new ThreadFactoryBuilder()
            .setNameFormat("thread-自定义线程名-runner-%d").build());
 
    /**
     * 执行python文件【异步 无需等待py文件执行完毕】
     *
     * @param fileName python文件地址
     * @param params   参数
     * @throws IOException
     */
    public static void execPythonFile(String fileName, String params) {
        taskPool.submit(() -> {
            try {
                exec(fileName, params);
            } catch (IOException e) {
                logger.error("读取python文件 fileName=" + fileName + " 异常", e);
            }
        });
 
    }
 
    /**
     * 执行python文件 【同步 会等待py执行完毕】
     *
     * @param fileName python文件地址
     * @param params   参数
     * @throws IOException
     */
    public static void execPythonFileSync(String fileName, String params) {
        try {
            execSync(fileName, params);
        } catch (IOException e) {
            logger.error("读取python文件 fileName=" + fileName + " 异常", e);
        }
    }
 
    private static void exec(String fileName, String params) throws IOException {
        logger.info("读取python文件 init fileName={}&path={}", fileName, WINDOWS_PATH);
        Process process;
        if (OS.startsWith("Windows")) {
            // windows执行脚本需要使用 cmd.exe /c 才能正确执行脚本
            process = new ProcessBuilder("cmd.exe", "/c", "python", WINDOWS_PATH + fileName, params).start();
        } else {
            // linux执行脚本一般是使用python3 + 文件所在路径
            process = new ProcessBuilder("python3", LINUX_PATH + fileName, params).start();
        }
 
        new Thread(() -> {
            logger.info("读取python文件 开始 fileName={}", fileName);
            BufferedReader errorReader = null;
            // 脚本执行异常时的输出信息
            errorReader = new BufferedReader(new InputStreamReader(process.getErrorStream()));
            List<String> errorString = read(fileName, errorReader);
            logger.info("读取python文件 异常 fileName={}&errorString={}", fileName, errorString);
        }).start();
 
        new Thread(() -> {
            // 脚本执行正常时的输出信息
            BufferedReader inputReader = null;
            inputReader = new BufferedReader(new InputStreamReader(process.getInputStream()));
            List<String> returnString = read(fileName, inputReader);
            logger.info("读取python文件 fileName={}&returnString={}", fileName, returnString);
        }).start();
 
        try {
            logger.info("读取python文件 wait fileName={}", fileName);
            process.waitFor();
        } catch (InterruptedException e) {
            logger.error("读取python文件 fileName=" + fileName + " 等待结果返回异常", e);
        }
        logger.info("读取python文件 fileName={} == 结束 ==", fileName);
    }
 
    private static void execSync(String fileName, String params) throws IOException {
        logger.info("同步读取python文件 init fileName={}&path={}", fileName, WINDOWS_PATH);
        Process process;
        if (OS.startsWith("Windows")) {
            // windows执行脚本需要使用 cmd.exe /c 才能正确执行脚本
            process = new ProcessBuilder("cmd.exe", "/c", "python", WINDOWS_PATH + fileName, params).start();
        } else {
            // linux执行脚本一般是使用python3 + 文件所在路径
            process = new ProcessBuilder("python3", LINUX_PATH + fileName, params).start();
        }
 
        taskPool.submit(() -> {
            logger.info("读取python文件 开始 fileName={}", fileName);
            BufferedReader errorReader = null;
            // 脚本执行异常时的输出信息
            errorReader = new BufferedReader(new InputStreamReader(process.getErrorStream()));
            List<String> errorString = read(fileName, errorReader);
            logger.info("读取python文件 异常 fileName={}&errorString={}", fileName, errorString);
        });
 
        taskPool.submit(() -> {
            // 脚本执行正常时的输出信息
            BufferedReader inputReader = null;
            inputReader = new BufferedReader(new InputStreamReader(process.getInputStream()));
            List<String> returnString = read(fileName, inputReader);
            logger.info("读取python文件 fileName={}&returnString={}", fileName, returnString);
        });
 
        try {
            logger.info("同步读取python文件 wait fileName={}", fileName);
            process.waitFor();
        } catch (InterruptedException e) {
            logger.error("同步读取python文件 fileName=" + fileName + " 等待结果返回异常", e);
        }
        logger.info("同步读取python文件 fileName={} == 结束 ==", fileName);
    }
 
    private static List<String> read(String fileName, BufferedReader reader) {
        List<String> resultList = Lists.newArrayList();
        String res = "";
        while (true) {
            try {
                if (!((res = reader.readLine()) != null)) break;
            } catch (IOException e) {
                logger.error("读取python文件 fileName=" + fileName + " 读取结果异常", e);
            }
            resultList.add(res);
        }
        return resultList;
    }
 
}

Runtime.getRuntime().exec()

In fact, the bottom layer of the above script also uses the ProcessBuilder object, so it is the same.