How can I capture real time command line output of x265.exe with Python?

LeeRoermond :

I would like to write a GUI for x265.exe which presents a better (more humanized) real time progress .

Here's the code I used to capture subprocess's output:

import subprocess

cmd = r'ping www.baidu.com -n 4'
popen = subprocess.Popen(cmd, stdout = subprocess.PIPE ,stderr=subprocess.STDOUT ,shell=True)
while True:
    next_line = popen.stdout.readline()
    if next_line == b'' and popen.poll() != None:
        break
    else:
        print(next_line.decode('ascii').replace('\r\n','\n') , end='')

It performs perfectly with 'ping'.

However ,when I swiched to 'x265' command ,thing goes to wired.

For example, If I replaced string variable 'cmd' into "x265 --y4m --crf 21 --output output.hevc input.y4m" in the preceding code.Theoretically , it should gives out the following output in lines arranged in order of time :

y4m  [info]: 1920x1080 fps 24000/1001 i420p10 frames 0 - 100 of 101
x265 [info]: Using preset ultrafast & tune none
raw  [info]: output file: C:\temp\output.hevc
x265 [info]: Main 10 profile, Level-4 (Main tier)
x265 [info]: Thread pool created using 16 threads
x265 [info]: Slices                              : 1
x265 [info]: frame threads / pool features       : 4 / wpp(34 rows)
x265 [info]: Coding QT: max CU size, min CU size : 32 / 16
x265 [info]: Residual QT: max TU size, max depth : 32 / 1 inter / 1 intra
x265 [info]: ME / range / subpel / merge         : dia / 57 / 0 / 2
x265 [info]: Keyframe min / max / scenecut / bias: 23 / 250 / 0 / 5.00
x265 [info]: Lookahead / bframes / badapt        : 5 / 3 / 0
x265 [info]: AQ: mode / str / qg-size / cu-tree  : 1 / 0.0 / 32 / 1
x265 [info]: Rate Control / qCompress            : CRF-21.0 / 0.60
x265 [info]: tools: strong-intra-smoothing lslices=6 deblock

[1.0%] 1/101 frames, 6.289 fps, 7217.8 kb/s
[25.7%] 26/101 frames, 59.23 fps, 299.23 kb/s
[45.5%] 46/101 frames, 66.76 fps, 322.81 kb/s  
[69.3%] 70/101 frames, 73.30 fps, 224.53 kb/s
[93.1%] 94/101 frames, 77.05 fps, 173.67 kb/s   

x265 [info]: frame I:      1, Avg QP:23.45  kb/s: 7098.44 
x265 [info]: frame P:     25, Avg QP:25.71  kb/s: 311.24  
x265 [info]: frame B:     75, Avg QP:28.33  kb/s: 23.89   
x265 [info]: consecutive B-frames: 3.8% 0.0% 0.0% 96.2% 

encoded 101 frames in 1.22s (82.58 fps), 165.06 kb/s, Avg QP:27.64

But the truth is, those output block in the middle part which indicates the real time progress will not be captured every specific it updated . popen.stdout.readline() command will be blocked until progress goes to 100% and then output altogether. Obviously that's not what I want.

( ↓ I mean by this part)

[1.0%] 1/101 frames, 6.289 fps, 7217.8 kb/s
[25.7%] 26/101 frames, 59.23 fps, 299.23 kb/s
[45.5%] 46/101 frames, 66.76 fps, 322.81 kb/s  
[69.3%] 70/101 frames, 73.30 fps, 224.53 kb/s
[93.1%] 94/101 frames, 77.05 fps, 173.67 kb/s   

Could anyone help me fiture out what's going on and how to fix it to achieve my goal?

Thanks a lot.

syfluqs :

My guess would be, that since the command output is updating the text in-place, there is no newline character at the end, instead, the output only has a carriage return to return the cursor at the beginning of the same line and overwriting the previous line. So using readline will not achieve what you want. You will have to write your own custom logic to read the output and split the output into lines.

Following is my implementation

import re
import subprocess

cmd = r'ping www.baidu.com -n 4'
popen = subprocess.Popen(cmd, stdout = subprocess.PIPE ,stderr=subprocess.STDOUT ,shell=True)

read_chunk_size = 20
next_lines = ['']
while True:
    # read the stdout upto read_chunk_size bytes
    chunk = popen.stdout.read(read_chunk_size)
    # checking if process has finished
    if not chunk and popen.poll() != None:
        break
    # splitting the read string of charachters by either \n, \r or \n\r
    split_chunk = re.split('\n|\r|\n\r', chunk.decode('utf-8'))
    # combining the first of the split strings to the last of the strings
    # which were read in the last iteration. since we are reading upto only 
    # a fixed length, we may get incomplete lines
    next_lines = [next_lines[-1] + split_chunk[0]] + split_chunk[1:]
    # iterate upto the second last item in next_lines, since the last item may
    # still be incomplete which will be read in the next iteration
    for line in next_lines[:-1]:
        # read your lines here
        print(line)