The previous article introduced the interactive code of jdbc interpreter and interpreter module. This article mainly analyzes the code of Zengine calling Interpreter module.
After introducing this article, we can string together the process of paragraph run (the whole process will be discussed later)
Similarly, look at this test class
zeppelin-zengine/src/test/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreterTest.java
@Test
public void testFIFOScheduler() throws InterruptedException, InterpreterException {
LOGGER.info("===testFIFOScheduler====");
interpreterSetting.getOption().setPerUser(InterpreterOption.SHARED);
// by default SleepInterpreter would use FIFOScheduler
LOGGER.info("===getInterpreter====");
final Interpreter interpreter1 = interpreterSetting.getInterpreter("user1", note1Id, "sleep");
LOGGER.info("===createDummyInterpreterContext====");
final InterpreterContext context1 = createDummyInterpreterContext();
// run this dummy interpret method first to launch the RemoteInterpreterProcess to avoid the
// time overhead of launching the process.
LOGGER.info("111");
LOGGER.info("=====name:{}=======",interpreter1.getClassName());
System.out.println(interpreter1.getClassName());
interpreter1.interpret("10101", context1);
LOGGER.info("222");
Thread thread1 = new Thread() {
@Override
public void run() {
try {
assertEquals(Code.SUCCESS, interpreter1.interpret("100", context1).code());
} catch (InterpreterException e) {
e.printStackTrace();
fail();
}
}
};
Thread thread2 = new Thread() {
@Override
public void run() {
try {
assertEquals(Code.SUCCESS, interpreter1.interpret("100", context1).code());
} catch (InterpreterException e) {
e.printStackTrace();
fail();
}
}
};
long start = System.currentTimeMillis();
thread1.start();
thread2.start();
thread1.join();
thread2.join();
long end = System.currentTimeMillis();
assertTrue((end - start) >= 200);
}You can take a look at this test method, and add some logs here
RemoteInterpreterTest inherits the abstract class in AbstractInterpreterTest, and will first execute the setUp method to initialize the interpreter for reading configuration file information
The core is mainly to execute the interpret method in RemoteInterpreter,
@Override
public InterpreterResult interpret(final String st, final InterpreterContext context)
throws InterpreterException {
LOGGER.info("st:\n{}", st);
if (LOGGER.isDebugEnabled()) {
LOGGER.debug("st:\n{}", st);
}
final FormType form = getFormType();
RemoteInterpreterProcess interpreterProcess = null;
try {
interpreterProcess = getOrCreateInterpreterProcess();
} catch (IOException e) {
throw new InterpreterException(e);
}
if (!interpreterProcess.isRunning()) {
return new InterpreterResult(InterpreterResult.Code.ERROR,
"Interpreter process is not running\n" + interpreterProcess.getErrorMessage());
}
return interpreterProcess.callRemoteFunction(client -> {
RemoteInterpreterResult remoteResult = client.interpret(
sessionId, className, st, convert(context));
Map<String, Object> remoteConfig = (Map<String, Object>) GSON.fromJson(
remoteResult.getConfig(), new TypeToken<Map<String, Object>>() {
}.getType());
context.getConfig().clear();
if (remoteConfig != null) {
context.getConfig().putAll(remoteConfig);
}
GUI currentGUI = context.getGui();
GUI currentNoteGUI = context.getNoteGui();
if (form == FormType.NATIVE) {
GUI remoteGui = GUI.fromJson(remoteResult.getGui());
GUI remoteNoteGui = GUI.fromJson(remoteResult.getNoteGui());
currentGUI.clear();
currentGUI.setParams(remoteGui.getParams());
currentGUI.setForms(remoteGui.getForms());
currentNoteGUI.setParams(remoteNoteGui.getParams());
currentNoteGUI.setForms(remoteNoteGui.getForms());
} else if (form == FormType.SIMPLE) {
final Map<String, Input> currentForms = currentGUI.getForms();
final Map<String, Object> currentParams = currentGUI.getParams();
final GUI remoteGUI = GUI.fromJson(remoteResult.getGui());
final Map<String, Input> remoteForms = remoteGUI.getForms();
final Map<String, Object> remoteParams = remoteGUI.getParams();
currentForms.putAll(remoteForms);
currentParams.putAll(remoteParams);
}
return convert(remoteResult);
}
);
}Among them, getOrCreateInterpreterProcess() goes all the way to call the start method in zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/remote/ExecRemoteInterpreterProcess.java, and execute the shell or cmd script through the commons-exec command ( bin/interpreter.sh) starts an independent process, and the specific execution class in the shell script (org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer), echoes the analysis of the interpreter principle in the previous article
@Override
public void start(String userName) throws IOException {
// start server process
CommandLine cmdLine = CommandLine.parse(interpreterRunner);
cmdLine.addArgument("-d", false);
cmdLine.addArgument(getInterpreterDir(), false);
cmdLine.addArgument("-c", false);
cmdLine.addArgument(getIntpEventServerHost(), false);
cmdLine.addArgument("-p", false);
cmdLine.addArgument(String.valueOf(intpEventServerPort), false);
cmdLine.addArgument("-r", false);
cmdLine.addArgument(getInterpreterPortRange(), false);
cmdLine.addArgument("-i", false);
cmdLine.addArgument(getInterpreterGroupId(), false);
if (isUserImpersonated() && !userName.equals("anonymous")) {
cmdLine.addArgument("-u", false);
cmdLine.addArgument(userName, false);
}
cmdLine.addArgument("-l", false);
cmdLine.addArgument(getLocalRepoDir(), false);
cmdLine.addArgument("-g", false);
cmdLine.addArgument(getInterpreterSettingName(), false);
interpreterProcessLauncher = new InterpreterProcessLauncher(cmdLine, getEnv());
interpreterProcessLauncher.launch();
interpreterProcessLauncher.waitForReady(getConnectTimeout());
if (interpreterProcessLauncher.isLaunchTimeout()) {
throw new IOException(
String.format("Interpreter Process creation is time out in %d seconds", getConnectTimeout() / 1000) + "\n"
+ "You can increase timeout threshold via "
+ "setting zeppelin.interpreter.connect.timeout of this interpreter.\n"
+ interpreterProcessLauncher.getErrorMessage());
}
if (!interpreterProcessLauncher.isRunning()) {
throw new IOException("Fail to launch interpreter process:\n" + interpreterProcessLauncher.getErrorMessage());
}
if (isHadoopClientAvailable()) {
String launchOutput = interpreterProcessLauncher.getProcessLaunchOutput();
Matcher m = YARN_APP_PATTER.matcher(launchOutput);
if (m.find()) {
String appId = m.group(1);
LOGGER.info("Detected yarn app: {}, add it to YarnAppMonitor", appId);
YarnAppMonitor.get().addYarnApp(ConverterUtils.toApplicationId(appId), this);
}
}
}And the client-side code that actually calls the thrift server-side service
The above picture is the log of the code running, which can help us locate the running order of the code
({FIFO-RemoteInterpreter-python-shared_process-shared_session-1} ProcessLauncher.java[launch]:96) - Process is launched: [.\\bin\interpreter.cmd, -d, ./\interpreter/python, -c, 10.4.144.223, -p, 52945, -r, :, -i, python-shared_process, -l, ./\local-repo\python, -g, python]
({FIFO-RemoteInterpreter-md-shared_process-shared_session-1} ProcessLauncher.java[launch]:96) - Process is launched: [.\\bin\interpreter.cmd, -d, ./\interpreter/md, -c, 10.4.144.223, -p, 52945, -r, :, -i, md-shared_process, -l, ./\local-repo\md, -g, md]
({FIFO-RemoteInterpreter-jdbc-shared_process-shared_session-1} ProcessLauncher.java[launch]:96) - Process is launched: [.\\bin\interpreter.cmd, -d, ./\interpreter/jdbc, -c, 10.4.144.223, -p, 52945, -r, :, -i, jdbc-shared_process, -l, ./\local-repo\jdbc, -g, jdbc]
reference
The Gospel of Programmers - Apache Commons Exec - Programmer Sought
Detailed Explanation of Apache Thrift Series (1) - Overview and Getting Started - Nuggets