Summary:
- Process pool and thread pool
- Synchronous calls and asynchronous calls
- Callback
- Coroutine
First, the process pool and thread pool:
1, the pool concept:
Whether thread or process can not go on unlimited, always consuming and resource-intensive.
In other words, the hardware load capacity is limited, to ensure high efficiency while working should also need to ensure that the hardware resource consumption, so it is necessary to set an upper limit to the hardware to reduce the pressure on hardware, so there will be a pool concept .
2, with the use of process pool thread pool thread :( process of creating similar, so the process pools and thread pools use basically the same)
from concurrent.futures Import ProcessPoolExecutor # import process cell module from concurrent.futures Import ThreadPoolExecutor # import thread pool module Import os Import Time Import Random # Below an example process pool, thread pool just use import module is not the same, and nothing more. DEF Task (name): Print ( ' name: [% s] | processes: [% s] is running ' % (name, os.getpid ())) the time.sleep (random.randint ( 1, 3)) # simulation run time-consuming process. # The need for this step: When creating process, the code will be imported from start to finish in a modular fashion to perform load it again #(So if you do not create a thread in the main written inside, all the code inside the py file will execute it again from beginning to end loader # would lead to an infinite loop in the creation process.) IF __name__ == ' __main__ ' : the pool = ProcessPoolExecutor (4) # set the thread pool size, default is equal to the number of cpu core. for I in Range (10 ): pool.submit (Task, ' process S% ' % I) # submitted asynchronously (without waiting for submission) pool.shutdown (the wait = True) # Process cell inlet no longer submitted while waiting all finished running process pool. (Similar to the join method) Print ( ' primary ' ) # before it is complete statement identifies the primary process
# Run the process and results: name: [Process 0] | processes: [4080 ] running name: [Process 1] | processes: [18336 ] running name: [Process 2] | processes: [19864 ] running name: [process 3] | processes: [25604 ] running name: [process 4] | processes: [4080 ] running name: [process 5] | processes: [18336 ] running name: [process 6] | processes: [ 4080 ] is running name: [process 7] | processes: [19864 ] running name: [process 8] | processes: [25604 ] running name: [process 9] | processes: [18336 ] running main
Second, synchronous calls, asynchronous calls
Synchronous call: submit the task, still waiting for the task is finished, get the result and then perform the next task, resulting in a serial program execution!
from concurrent.futures Import ProcessPoolExecutor # import process cell module from concurrent.futures Import ThreadPoolExecutor # import thread pool module Import os Import Time Import Random DEF Task (name): Print ( ' name: [% s] | process [% s] is run ... ' % (name, os.getpid ())) the time.sleep (the random.randint ( . 1,. 3 )) return ' get [s%] | process results for% s ... ' % (name , os.getpid ()) IF the __name__ == '__main__ ' : the pool = ProcessPoolExecutor (4 ) the Result = [] # Create an empty list to collect the results for i in the Range (10 ): RES = pool.submit (Task, ' the process of S% ' % i) .Result () # use .result () method to get results every time, synchronous call result.append (RES) pool.shutdown (the wait = True) for J in the result: Print (J) Print ( ' primary process ' )
# Execution results: name: [Process 0] | processes [3376 ] running ... name: [Process 1] | processes [27124 ] running ... name: [Process 2] | processes [10176 ] is running. .. name: [process 3] | processes [28636 ] running ... name: [process 4] | processes [3376 ] running ... name: [process 5] | processes [27124 ] running ... name: [process 6] | processes [10176 ] running ... name: [process 7] | processes [28636 ] running ... name: [process 8] | processes [3376 ] running ... name: [process 9] | processes [27124 ] running ... to get [process 0] | results 3376 of process ... to get [process 1] | result of the process 27124 ... get [process 2] |The process results 10176 ... get [Process 3] | result of the process 28636 ... get [Process 4] | results 3376 of process ... to get [Process 5] | result of the process 27124 ... take to [process 6] | result of the process 10176 ... get [process 7] | result of the process 28636 ... get [process 8] | results 3376 of process ... to get [the process 9] | processes 27124 the result ... the main process
Asynchronous call: submit the task, and so the results do not, continue.
from concurrent.futures import ProcessPoolExecutor import os import random import time def task(name): time.sleep(random.randint(1, 3)) print('name: %s 进程[%s]运行...' % (name, os.getpid())) if __name__ == '__main__': pool = ProcessPoolExecutor(4) for i in range(10): pool.submit(task, '进程%s'I%) # asynchronous call, do not wait for the results after the submission, continue to execute code pool.shutdown (the wait = True) Print ( ' primary process ' )
name: Process 3 Process [10016 ] running ... name: Process 0 Process [ 12736 ] running ... name: Process 1 Process [ 4488 ] running ... name: Process 2 Process [ 3920 ] running ... name: process 5 process [ 12736 ] running ... name: process 6 process [ 4488 ] running ... name: process 4 process [ 10016 ] running ... name: process 9 process [ 4488 ] running ... name: process 8 process [ 12736 ] running ... name: process 7 process [ 3920 ] running ... the main process
Third, the callback function:
The above demonstrates the asynchronous call us at the time, said that the author does not wait for the task execution results, continue to implement the code down, then the result of the execution of how we get it?
Each process can be a process or thread pool and thread pool to bind a function that automatically trigger the task and to receive the return value as a parameter in the task execution process or thread is completed, this function is a callback function.
from concurrent.futures Import ThreadPoolExecutor Import Time Import Random Import Requests DEF Task (url): Print ( ' access to the Web [% s] information ' % url) the Response = requests.get (url) # download page time.sleep (random.randint (. 1,. 3 )) return { ' URL ' : URL, ' content ' : response.text} # returns: page address and page content Futures = [] DEF Back (RES): RESRes.result = () # taken to submit the results of the task (fixed wording callback function) RES = ' sites [% s] Content-Length: S% ' % (res.get ( ' URL ' ), len (res.get ( ' Content ' ))) futures.append (RES) return Futures IF the __name__ == ' __main__ ' : URLs = [ ' http://www.baidu.com ' , ' http://www.dgtle.com/ ' , ' https://www.bilibili.com/ ' ] the pool = the ThreadPoolExecutor (. 4 ) Futures = [] for I in URLs: pool.submit (Task, I) .add_done_callback (Back) # After executing thread callback function pool.shutdown (the wait = True) for J in Futures: Print (J)
Access to the Web [http: // www.baidu.com] information access to the Web [HTTP: //www.dgtle.com/ ] information access to the Web [HTTPS: //www.bilibili.com/ ] information website [HTTP: // www.dgtle.com/] content-length: 39360 website [HTTPS: //www.bilibili.com/] content-length: 69377 website [HTTP: //www.baidu.com] content-length: 2381