python thread pool --threadpool

At the time reptiles, sometimes resolved to get a lot of pictures or video address, if a download is complete go download another one, so the efficiency is too slow, then you can use the thread pool threadpool, using the basic steps are as follows:

1. scheduled task function

2. Create a thread pool, define the number of threads task_pool = threadpool.ThreadPool (n), n is the number of threads

3. Create a task threadpool.makeRequests thread (task function, parameter list), the main parameters must be iterative, such as list, task list and threadpool.makeRequests return the form to list all available extend () function to get all the tasks, see below example

4. Create a list of tasks put to the thread pool, task.putRequest (req), using the for loop iterates the task list or map (task.putRequest, task_list)

5. perform tasks, wait for all tasks finished threadpool.wait () to perform this step, the task really started

Code Example:

Import ThreadPool
 Import Time 


DEF foo (n-):
     Print (n-) 
    the time.sleep ( 2 ) 

T1 = the time.time () 
task_pool = threadpool.ThreadPool (10)   # Create a thread pool threads 10 
task_list = []                           # Task List 
= DATA_LIST [i for i in the Range (10)]      # create a list of parameters 
task_list.extend (threadpool.makeRequests (foo, DATA_LIST))     # create a thread task 
for REQ in task_list: 
    task_pool.putRequest (REQ)            #The task put into a pool thread 
Print ( " Starting ................... " ) 
task_pool.wait ()                         # start the task 
T2 = time.time () 

Print ( T1-T2)    　　　　　　　　　　　　　　# calculate elapsed time

Results of the:

starting...................0162
73
4
5





8
9
2.06499981880188

From the results:

1. task_pool.wait () really began when the task execution

2. As a result of the disorder, indicating that indeed perform tasks at the same time

3.foo function in each pause two seconds, if time and again in order to perform, requires 2 * 10 seconds, using multi-threaded took only 2.06 seconds ( actually I have tried 100 tasks, 100 thread execution time of 2.05 seconds ) , indicating that the effect is obvious thread pool .

python thread pool --threadpool

Guess you like