Python Multiprocessing multi-process, multi-core CPU to calculate and display a progress bar using tqdm

1. Background

 　  When run some python, calculation function relatively high degree of complexity, the server-side of the single-core CPU time consuming, requiring the use of multi-process multi-CPU speed

2. The function requires

　 I use it is: pathos.multiprocessing library, progress bar display tqdm library, the installation method:

pip install pathos

　　After the installation is complete

from pathos.multiprocessing import ProcessingPool as Pool
from tqdm import tqdm

　Causes used here because pathos, Pool multiprocessing library function only supports a single parameter input, for example, f (x) = x ** 2 , and can not process f (x, y) = x + y of such a function 
　is more Needless to say, some functions require parameters, for example: F (x, alpha = 0.5 , gamma = 0.1) this.

3. Code

　　Define a function F. [X], where X is the input on the first iteration Array dimension, size: [num_X, len], iterate a first dimension num_X.

def F(X,lamda=10,weight=0.05):

    res={}

    res.update(F_1(X,lamda=lamda,weight=weight))
    res.update(F_2(X,lamda=lamda,weight=weight))
    return res

F x is output, a dict (dictionary format) 
here two parameters lamda function and weight over time, although the value of each call is the same, but still need to put an array for each iteration.

zip_lamda = [lamda for i in range(len(X)) ]
zip_weight = [weight for i in range(len(X)) ]

with tqdm(total=len(cold_sequences)) as t:
        for i, x in enumerate(pool.imap(F,X,zip_lamda,zip_weight)):
            X[i,:] = [x[key] for key in x.keys()]
            Y[i,] = 0
            t.update()

    pool.close()
    pool.join()

4. Results

mutiprocess accelerated ago

 

After mutiprocess accelerate