How does python start a new process and get the output of the program.

How does python start a new process and get the output of the program.

The first, more disgusting way, will block the main process, and if you don't read it, you will often freeze the log.

import shlex
import subprocess
 
if __name__ == '__main__':
    shell_cmd = 'python3 subprogram.py'
    cmd = shlex.split(shell_cmd)
    p = subprocess.Popen(cmd, shell=False, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
    while p.poll() is None:  # 这里死循环了.
        line = p.stdout.readline() # readlines(100)
        line = line.strip()
        if line:
            print('Subprogram output: [{}]'.format(line))
    if p.returncode == 0:
        print('Subprogram success')
    else:
        print('Subprogram failed')

When using subprocess.PIPE in subprocess.Popen, use readlines() to read the contents of the pipeline, and the program will block. In this case, you only need to set the parameter n in readlines(n).
For example: p=subprocess.Popen(['nethogs','-t','-d'],stdout=subprocess.PIPE)
x = p.stdout.readlines(100)
If you want to take out all the content in the pipeline, You only need to set the parameter n very large, after all, the maximum capacity of subprocess.PIPE is only 64kb.

The second type: It is also the simplest and most elegant way of writing. This way of writing will write the output to a file.
This way is relatively stable and does not require a while loop to execute. There is no problem of blocking the main thread. The disadvantage is that the output cannot be obtained in real time. , You must read the log file yourself.

import subprocess

Logfilename = str(time.time())+"std.log"
with io.open(Logfilename, 'w',encoding='utf-8' ) as writer:
   self.proc = subprocess.Popen(
                [
                "python.exe", 
                "./UNet/pdseg/train.py",
                "--use_gpu",
                "--cfg=./static/dataset/optic_disc_seg/_UNetAIModels/M1/unet_optic.yaml",
                ],
               shell = True,
               stdout=writer
           )

The third method is to try to write a wrapper class to wrap the output to the file function. Intercept the Write operation in the middle . Later, it turns out that this idea is wrong. Because no matter how you wrap it, the output in the subprocess is not Will be automatically sent to the main process. Two objects are completely independent in memory. The log output operation in the child process is directly operated by the file handle passed in the past. The basic operation of the operating system is used, which is basically the same as that of the parent process. The python code above is irrelevant, so the packaging is useless.

The fourth is to create a new thread to read the pipeline output of the subprocess in a loop . On the basis of the first method, set the parameter stdout to subprocess.PIPE, so that the main thread will not be stuck. The reading is faster. Just feel It's not so beautiful, it needs more resources.

#读取日志的线程
class ReadLogThread(threading.Thread):
    def __init__(self, _pipe, onout):
        threading.Thread.__init__(self)
        self._pipe = _pipe
        self.logList = []
        self.onout = onout
    def run(self):
        while True:
            try:
                line = self._pipe.readline()
                if line == "":
                    break
                self.logList.append(line)
                if(self.onout!=None):
                    self.onout(line)
                # print("Thread=>",line)
            except Exception as eee:
                traceback.print_exc()
                # self.logList.append(str(eee))
                
 
    def get_log(self):
        return "".join(self.logList)
  

def main():
    writer = io.StringIO(initial_value='',NEWLINE='\n')
        self.proc = subprocess.Popen(
                arg,
                # [
                # "python.exe", 
                # "./UNet/pdseg/train.py",
                # "--use_gpu",
                # "--cfg=./static/dataset/optic_disc_seg/_UNetAIModels/M1/unet_optic.yaml",
                # ],
                shell = True,
                stdout=subprocess.PIPE
            )
 
        
         循环输出标准输出内容，避免阻塞
         self.stdoutThread = ReadLogThread(self.proc.stdout,onStdoutCallBack)
         self.stdoutThread.start()
        
         self.stderrThread = ReadLogThread(self.proc.stderr,onStderrCallBack)
         self.stderrThread.start()
         

The fifth method is based on the second method of outputting to a file. Read the log file output by the child process in a loop. Then parse the output content by yourself. The disadvantage is that it must be read from the beginning each time, which is slow. The
best is also Read in sub-threads. The disk operations are relatively large.

Summary:
Personally, from the current point of view, it is necessary to be both real-time and stable. The best practice is the fourth method, which is to open a child thread to read the pipeline information in a loop. And process the output log in the child thread.