In learning python multi-process, method of operation, the process receives a number of parameters and encountered problems when more than one result, and now after learning summarize here
Pool.map () multi-parameter task
in passing with a method to map multiple parameters methods can not achieve the desired results, like the following
def job(x ,y):
return x * y
if __name__ == "__main__":
pool = multiprocessing.Pool()
res = pool.map(job, 2, 3)
print resTherefore, the method can only be encapsulated by a plurality of parameters, the method of operating the package follows in the process
def job(x ,y):
return x * y
def job1(z):
return job(z[0], z[1])
if __name__ == "__main__":
pool = multiprocessing.Pool()
res = pool.map(job1, [(2, 3), (3, 4)])
print resThus we can achieve a plurality of transmission parameters
ps: If you need to obtain a plurality of results in a list of the plurality of incoming tuple
Pool.apply_async () output a plurality of results of the iteration
when the method of receiving a plurality of parameters using apply_async () method, the normal parameters defining a plurality of tasks in the method, the parameters can be passed in the form of tuples
but to apply_async () method of transmission will obtain a plurality of values being given the results of a plurality of iterations, because the method can only receive a value, it is possible to put the method into a list in the formula below
def job(x):
return x * x
if __name__ == "__main__":
pool multiprocessing.Pool()
res = [pool.apply_async(target=job, (i,)) for i in range(3)]
print [r.get() for r in res]When you want to improve the efficiency of a task, we can split the task, to split the task into multiple sub-tasks, then use asynchronous multi-process execution, namely processed simultaneously, shortening overall mission time. In the python's multiprocessing package, there are two methods can be constructed process tasks executed asynchronously, apply_async () and map_async (), both of which can add tasks separately, and then execute multiple processes simultaneously. But the two have significant differences, described below.
For apply_async (func, args), func function name of the task to be performed, args is a list of tuples or objects such as may be iterative, which contains the parameters to be passed to func, for a plurality of subtasks, a plurality of times to each call apply_async () to add eleven, but this can be resolved to achieve through the list to allow multiple processes result in a return to save the list. For map_async (func, iterable, chunksize), if multiple sub-tasks are performed by the same function, but with different parameters, you can specify the number of processes and parameters parameters by chunksize split in a list by iterable passed, (in fact, chunksize here represents the number of iterable split, but preferably allowed equal number of processes), so that you do not need to add eleven.
These are just two slightly different, more important difference is that, if it goes through apply_async () method, because it is a manual process and add the specified tasks are independent so the results between each process will be saved separately, so that the benefits that, although a few of which may process an error has occurred, an exception is thrown, but does not lead to other processes also fails, other processes will not be affected, and when the result of the acquisition process of throwing an exception, will return exception information; but if map_async () method, its mandate is not independent sub-parameters, if one of these sub-parameter task throws an exception, but also lead to other sub-parameters mission to stop, that is, and not a sub-parameters to perform different tasks by a separate thread.
Through the above comparison, it was found when a split task to improve the efficiency, by parsing the list using apply_async () method adds a child task, is to perform multiple processes independent () method is better than map_async, so in this case, apply_async () is the best choice.