python multiprocessing multiprocessing Pool-related issues

python multi-process presumably most people have used before, can take full advantage of multi-core CPU make more efficient code efficiency.

We look at the official use of multiprocessing.pool.Pool.map

map(func, iterable[, chunksize])
A parallel equivalent of the map() built-in function (it supports only one iterable argument though). It blocks until the result is ready.

This method chops the iterable into a number of chunks which it submits to the process pool as separate tasks. The (approximate) size of these chunks can be specified by setting chunksize to a positive integer.

 

First, how to become a multi-parameter passing parameters

usage map, a function func parameter can only pass into iteration.

How to do it if we need to pass multiple parameters,

One method is to put a plurality of parameters into tuples in the list or passed as a parameter in func

Another is the use of partial function, partial function (Partial function) by the parameter part of a pre-binding function of some value to obtain a new variable function with less parameters. In Python, partial functions may be implemented by the function of the partial functools higher-order functions. Partial source of partial functions as follows:

def partial(func, *args, **keywords):
    """New function with partial application of the given arguments
    and keywords.
    """
    if hasattr(func, 'func'):
        args = func.args + args
        tmpkw = func.keywords.copy()
        tmpkw.update(keywords)
        keywords = tmpkw
        del tmpkw
        func = func.func

    def newfunc(*fargs, **fkeywords):
        newkeywords = keywords.copy()
        newkeywords.update(fkeywords)
        return func(*(args + fargs), **newkeywords)
    newfunc.func = func
    newfunc.args = args
    newfunc.keywords = keywords
    return newfunc

Is also very simple to use, for example, we have a function func, which should an incoming texts, lock, data of three parameters, but we want to use multi-process the data are passed into the calculation, then we can first use partial function, the texts and lock secured to the first function in the formation of a new function, then a new function parameter passing data on it

from functools Import partial
 DEF FUNC (Texts, Lock, data): 
    ...... 


pt = partial (FUNC, Tests, Lock)   

# new function pt only need to pass a parameter data

This we can apply the function pt pool.map function and only in passing in a parameter data.

 

Second, and more inter-process communication

There is another situation, communication between multiple processes to each other, for example, I have the results of each process into the texts in this list. Of course, take this as a parameter passed to the function texts inside, but the general list and can not be shared to all processes, we need to use multiprocessing.Manager (). List () can be used to establish a list of inter-process communication, conflict prevention , but also on texts plus lock to prevent operation of conflict. Note multiprocessing.Lock () to create the lock can not be passed, require the use of multiprocessing.Manager (). Lock () to create. multiprocessing.Manager () to create a dictionary, you can also create list, lock, it creates a variable that can be used across multiple delivery process will not go wrong. For example, the following code:

texts = multiprocessing.Manager().list()
lock = multiprocessing.Manager().Lock()
pool = multiprocessing.Pool(processes=4)
data = list(range(20))
pt = partial(func, texts, lock)
pool.map(pt, data)
pool.close()
pool.join()