In the multi-process task in airflow, there is a problem that ERROR is not thrown in the child process

In the airflow of Python 2.7 (version 1.10.1), when using multiprocessing, an error occurs in the child process , and no error is thrown for each running node in the tree view on the page. There is no result information printed in the view log as shown in the figure :

The pool in multiprocessing used, after investigation, you can use the get() function in pool

    pool = Pool(processes=4)
    rst=[]
    for sql in sql_list:
        # 获取并行处理结果
        rst.append(pool.apply_async(func,(t[0],t[1])))
    pool.close()
    pool.join()
    for r in rst:
        print(r.get())

The get() function also raises error when implementing function raise error

class multiprocessing.pool.AsyncResult

The class of the result returned by Pool.apply_async() and Pool.map_async().

get([timeout])

Return the result when it arrives. If timeout is not None and the result does not arrive within timeout seconds then multiprocessing.TimeoutError is raised. If the remote call raised an exception then that exception will be reraised by get().

https://docs.python.org/2/library/multiprocessing.html#multiprocessing.pool.AsyncResult.get

You can also refer to: How to make python throw errors when calling multiprocessing? -TraderJay's answer-Zhihu https://www.zhihu.com/question/54644474/answer/140391537

https://stackoverflow.com/questions/22094852/how-to-catch-exceptions-in-workers-in-multiprocessing