在Python2.7版的airflow(1.10.1版)中,使用multiprocessing时,子进程出现错误,页面上tree view中各个运行节点无错误抛出,如图view log中也没有结果信息打印出来:
使用的multiprocessing中的pool,经查,可以使用pool中的get()函数
pool = Pool(processes=4)
rst=[]
for sql in sql_list:
# 获取并行处理结果
rst.append(pool.apply_async(func,(t[0],t[1])))
pool.close()
pool.join()
for r in rst:
print(r.get())
get()函数在实现 function raise error 时自己也 raise error
class
multiprocessing.pool.
AsyncResult
The class of the result returned by
Pool.apply_async()
andPool.map_async()
.
get
([timeout])Return the result when it arrives. If timeout is not
None
and the result does not arrive within timeout seconds thenmultiprocessing.TimeoutError
is raised. If the remote call raised an exception then that exception will be reraised byget()
.https://docs.python.org/2/library/multiprocessing.html#multiprocessing.pool.AsyncResult.get
也可以参考:python调用multiprocessing时如何让其抛出错误? - TraderJay's的回答 - 知乎 https://www.zhihu.com/question/54644474/answer/140391537
https://stackoverflow.com/questions/22094852/how-to-catch-exceptions-in-workers-in-multiprocessing