python multi-threaded (thread pool)

Reference a custom thread pool: https://www.cnblogs.com/zhang293/p/7954353.html

 

 

Get the best number of threads:

1, the user slowly incrementing to pressure measurement performance, observed QPS (i.e., in response to requests per second, that is, the maximum throughput capacity.), A response time

2, is calculated according to the formula: Number of best server thread = ((thread thread cpu time latency +) / threads of cpu time) * Number cpu

3, single-user pressure test to check the CPU consumption, and then directly by the percentage, then the pressure test, usually in the vicinity of this value should be the optimal number of threads.

 

Recommended way :( 1)

ThreadPoolExecutor returns future (T)

'''
Created on 2019年10月11日

@author: sea
'''
# -*- coding: utf-8 -*-
from concurrent.futures import ThreadPoolExecutor
import time
from com.sea.email.MailUtils import send
'''
ThreadPoolExecutor中的submit()
<T> Future<T> submit(Callable<T> callable);
<T> Future<T> submit(Runnable var1, T result);
Future<?> submit(Runnable runnable);
'''
print(" 线程池 ThreadPoolExecutor 的使用")
def sayhello(a):
    print("hello  "+a)
#     time.sleep(1)
    return "结果 nihao : "+a



def test1():#submit()
    pool = ThreadPoolExecutor(3)
    submit1 = pool.submit(sayhello,("sea"))  #方法名 ，参数（）
#     submit1 = pool.submit(sayhello,"sea")  #方法名 ，参数（）

    submit2 = pool.submit(sayhello,("sea2"))
    submit3 = pool.submit(sayhello,("sea3"))
    submit4 = pool.submit(sayhello,(" Sea4 " )) 
    
    Print (submit4.result ()) # print the return value should be blocked - until returning results 
    Print (submit1.result ())
     Print (submit2.result ())
     Print (submit3.result ())
     Print ( " over " ) 

DEF test2 (): # Map () 
    SEED = [ " A " , " B " , " C " ] 
    the pool = the ThreadPoolExecutor (. 3 ) 
    rsultList = pool.map (sayHello, SEED) # returns a value of the result set list
    for result in rsultList:
        print(result)#打印应返回值
    print("over")

#     with ThreadPoolExecutor(3) as executor1:
#         executor1.map(sayhello,seed)
if __name__ == '__main__':
#     test1()
    test2()
    
    
  

 

 

concurrent.futures.ThreadPoolExecutor, at the time of submission of the task, there are two ways, one is to submit () function, the other is the map () function, the main difference is that:

2.1, map can ensure sequentially output, are output sequentially submit chaos

2.2, if you want to submit a function of the task is the same, it can be simplified into a map. But if submitted by the task function is not the same, or possibly abnormal process of the implementation of the (course map will identify problems in the implementation of direct throw an error) is necessary to use submit ()

2.3, submit and map parameters are different, submit always need to submit an objective function and the corresponding parameters, map only need to submit one objective function, parameter objective function in an iterator (lists, dictionaries) where you can .

3. Now?

Here we consider a problem, to achieve the above two thread pool are packaged, can only add a task, then, suppose I now have such a demand when the thread pool is initialized, it is necessary runtime thread pool, and then further inside add a new task (note that the new task, a new thread is not), then how to do?

 

 

 

 

 

Mode 2: python3 of vthread library

You can try to vthread python3 library
 Import vthread 

@ vthread.pool ( . 6 )
 DEF some (A, B, C):
     Import Time; the time.sleep (. 1 )
     Print (A + B + C) 

for I in Range (10 ): 
    some (i, i, i) 

packet thread pool 
Import vthread 

@ vthread.pool ( . 6 )
 DEF some1 (A, B, C):
     Import Time; the time.sleep (. 1 )
     Print ( " some1 " , A + B + C) 

vthread.pool @ ( 3, 1 )
 DEFsome2 (A, B, C):
     Import Time; the time.sleep (. 1 )
     Print ( " some2 " , A * B * C) 

for I in Range (10 ): 
    some1 (I, I, I) 
    some2 (I, i, i) 

a locking or other operation, help () at substantially able to understand.

 

 

 

Mode 3 :( older)

DETAILED module using threadpool used as follows:

#! /usr/bin/env python
# -*- coding: utf-8 -*-

import threadpool
import time

def sayhello (a):
    print("hello: "+a)
    time.sleep(2)

def main():
    global result
    seed=["a","b","c"]
    start=time.time()
    task_pool=threadpool.ThreadPool(5)
    requests=threadpool.makeRequests(sayhello,seed)
    for req in requests:
        task_pool.putRequest(req)
    task_pool.wait()
    end=time.time()
    time_m = end-start
    print("time: "+str(time_m))
    start1=time.time()
    for each in seed:
        sayhello(each)
    end1=time.time()
    print("time1: "+str(end1-start1))

if __name__== ' __main__ ' : 
    main ()