The system starts a new thread cost is relatively high because it involves interaction with the operating system. In this case, use the thread pool may well improve performance, especially the need to create a large number of short-lived when the program thread, the more you should consider using thread pool.
Thread pool is to create a large number of idle threads at system startup, the program will be submitted as long as the function to a thread pool thread pool will be free to start a thread to execute it. After the end of the execution of the function, the thread does not die, but returned again to the thread pool becomes idle, waiting for the next function.
In addition, thread pool can be effectively number of concurrent threads in a control system. When the system contains a large number of concurrent threads, it will lead to a sharp decline in system performance, even lead Python interpreter to crash, while the maximum number of threads in the thread pool parameters to control the number of concurrent threads the system does not exceed this number.
Use the thread pool
Base class thread pool is concurrent.futures module Executor, Executor provides two sub-categories, namely ThreadPoolExecutor and ProcessPoolExecutor, which ThreadPoolExecutor used to create a thread pool, and ProcessPoolExecutor process used to create the pool.
If you use the thread pool / pools to manage concurrent programming process, as long as the corresponding task will be submitted to the thread pool function / process pool, rest of the things to get the thread pool / process pool.
Exectuor provides the following common methods:
1.submit (fn, * args, ** kwargs): will be presented to the thread pool fn function. * Args parameter to fn represents the function, * kwargs representatives in the form of keyword arguments passed to fn function parameters.
2.map (func, * iterables, timeout = None, chunksize = 1): This function is similar to a global function map (func, * iterables), but the function will start multiple threads to execute asynchronously map immediately iterables deal with.
3.shutdown (wait = True): close the thread pool.
The program will submit task function (submit) to the thread pool, submit method returns a Future object, Future class is primarily used to obtain a thread task function's return value. As the thread executes asynchronously task in the new thread, so thread executing the function of the equivalent of a "future perfect" job, so Python using Future to represent.
Future provides the following methods:
cancel (): cancel the Future represents the thread task. If the task is being performed, irrevocable, the method returns False; otherwise, the program will cancel the task and return True.
cancelled (): Returns the Future represents the thread whether the task was successfully canceled.
running (): If the Future represents the thread is executing the task, can not be canceled, the method returns True.
done (): If threaded tasks on behalf of the Funture are canceled or executed successfully completed, the method returns True.
result (timeout = None): Gets the Future represents the result of the last-threaded task returned. If the representative of the Future threaded task has not been completed, which will block the current thread, in which the blocking timeout parameter specifies a maximum number of seconds.
exception (timeout = None): Gets the Future represents the thread task was thrown. If the task is completed without exception, then the method returns None.
add_done_callback (fn): threaded tasks for the Future on behalf of a registered "callback function", when the task is completed successfully, the program will automatically trigger the fn function.
Once you've used a thread pool, you should call the shutdown of the thread pool () method, which will initiate a shutdown sequence of the thread pool. After calling shutdown () method thread pool is no longer receiving new tasks, but will have to submit all the previous task execution is completed. When all tasks are executed to complete the thread pool, all threads of the thread pool of death.
To use the thread pool threads to perform the following tasks:
1. Call ThreadPoolExecutor class constructor to create a thread pool.
2. Define a common function as a thread task.
3. Call ThreadPoolExecutor object's submit () method to submit threaded tasks.
4. When the time did not want to submit any task, calling ThreadPoolExecutor object shutdown () method to close the thread pool.
from concurrent.futures import ThreadPoolExecutor import threading import time # 定义一个准备作为线程任务的函数 def action(max): my_sum = 0 for i in range(max): print(threading.current_thread().name + ' ' + str(i)) my_sum += i return my_sum # 创建一个包含2条线程的线程池 pool = ThreadPoolExecutor(max_workers=2) # 向线程池提交一个task, 50会作为action()函数的参数 future1 = pool.submit(action, 50) # Submit a task and then to the thread pool, 100 will be as action () function parameters future2 = pool.submit (Action, 100 ) # judge future1 Representative's mandate has ended Print (future1.done ()) the time.sleep ( 3 ) # judge future2 representative's mandate has ended Print (future2.done ()) # View future1 tasks on behalf of returned results Print (future1.result ()) # View future2 tasks on behalf of returned results Print (future2.result () ) # close the thread pool pool.shutdown ()
When the program submission action () function to the thread pool, Submit () method returns the Future object corresponding to the task, the program immediately determines done futurel () method returns False (indicating that it is the task of further undone). Then the main program pause for 3 seconds, then the judge future2 done () method, if the task has been completed at this time, then the method returns True.
Programs last two asynchronous tasks to get the results returned by the Future of the result () method. When the program uses the Future of the result () method to obtain the results, which will block the current thread, if no timeout parameter, the current thread stays blocked until the task on behalf of the Future returns.
Get the results
In front of the program calls the result Future () method to obtain a thread shipped back to the task of value, but the current method will block the main thread, only to wait until after the task is completed Qian Cheng, blocking will result () method is released.
If the program does not want to directly call result () method blocks the thread, you can add a callback function Future of add_done_callback () method, the callback function of the form fn (future). When the thread task is completed, the program will automatically trigger the callback function and the corresponding Future object as a parameter passed to the callback function.
from concurrent.futures Import the ThreadPoolExecutor Import Threading Import Time # define a thread is prepared as a function of task DEF Action (max): my_sum = 0 for I in Range (max): Print . (threading.current_thread () name + ' ' + STR (i)) my_sum + = i return my_sum # create a two threads in a thread pool with ThreadPoolExecutor (max_workers = 2 ) aS the pool: # submit a task to the thread pool, 50 will be as action () function parameters future1 = pool.submit(action, 50) # 向线程池再提交一个task, 100会作为action()函数的参数 future2 = pool.submit(action, 100) def get_result(future): print(future.result()) # 为future1添加线程完成的回调函数 future1.add_done_callback(get_result) # 为future2添加线程完成的回调函数 future2.add_done_callback(get_result) print('--------------')
Above were added to the main program with a callback function future1, future2, the callback function will get its return value at the end of the thread task.
The last line of code in the main program prints a horizontal line. Because the program does not directly call future1, future2 the result () method, so the main thread is not blocked, you can immediately see the output of the main thread prints out the horizontal line. Next you will see two new concurrent execution threads, a thread when the task execution is completed, get_result () function is triggered, the return value of the output thread task.
Further, since the thread pool implementation management protocol context (Context Manage Protocol), and therefore, it can be used with the program statements to manage the thread pool, so avoiding manual close to the thread pool, as shown in the above procedure.
In addition, Exectuor also provides a method, a function which is similar to the global function map (), except that the thread pool map () method will start a thread for each element of iterables concurrent manner to perform the function func. This mode is equivalent to start len (iterables) threads, wells to collect the results of each thread. map(func, *iterables, timeout=None, chunksize=1)
from concurrent.futures Import the ThreadPoolExecutor Import Threading Import Time # define a thread is prepared as a function of task DEF Action (max): my_sum = 0 for I in Range (max): Print . (threading.current_thread () name + ' ' + STR (I)) my_sum + = I return my_sum # Create a thread of the thread pool 4 with the ThreadPoolExecutor (= max_workers. 4 ) the pool AS: # using a thread of execution calculation map # 后面元组有3个元素，因此程序启动3条线程来执行action函数 results = pool.map(action, (50, 100, 150)) print('--------------') for r in results: print(r)
上面程序使用 map() 方法来启动 3 个线程（该程序的线程池包含 4 个线程，如果继续使用只包含两个线程的线程池，此时将有一个任务处于等待状态，必须等其中一个任务完成，线程空闲出来才会获得执行的机会），map() 方法的返回值将会收集每个线程任务的返回结果。
运行上面程序，同样可以看到 3 个线程并发执行的结果，最后通过 results 可以看到 3 个线程任务的返回结果。
通过上面程序可以看出，使用 map() 方法来启动线程，并收集线程的执行结果，不仅具有代码简单的优点，而且虽然程序会以并发方式来执行 action() 函数，但最后收集的 action() 函数的执行结果，依然与传入参数的结果保持一致。也就是说，上面 results 的第一个元素是 action(50) 的结果，第二个元素是 action(100) 的结果，第三个元素是 action(150) 的结果。