A use management subprocess subprocess module, and controls the input and output streams
1, Popen is configured to start the process, communicate information output method for reading subprocess
proc=subprocess.Popen(['echo','Hello'],stdout=subprocess.PIPE)
out,err=proc.communicate()
print(out.decode('utf-8'))
2, the child process will operate independently of the parent process, the parent process here refers to the Python interpreter
3, can pass parameters to communicate timeout methods, to avoid process deadlock or loss of response, a period of time does not automatically throw an exception response
Second, the use of a blocking thread to execute I / O module is simpler than asyncio
1, blocking I / O response is relatively slow, it should use the thread execution points out, to avoid jamming program within a certain time interval, in response interval, the CPU can perform some operations calculated
Third, the use Lock to prevent data races
1, thread suspension of the operation could lead to confusion values, even the simplest of the increment operator, will be split into three steps, the middle step assignment conversion, if the conversion of the assignment process, the thread is interrupted, then it is possible to assign a new value into the other threads, and then convert it back again when they assigned the old value, resulting in data loss
2, Lock Python-threading module built, corresponding mutex, protected by a mutex value, so that the same time a plurality of threads can have access to the value
with lock():
count+=1
Fourth, the use Queue to coordinate work among threads
1, the function pipeline, each task will handle the return phase value is transferred to the next stage, it involves blocking I / O task is appropriate, it is easy to assign such tasks to a plurality of threads or processes to
2, for example: a processing system such as a photograph, the camera acquires picture start (a), and then treated to adjust its size (B), and finally uploaded to the network album (c), the functions are written in three stages, and then by a line of stitching concurrent processing, it can be produced in this way - consumption queue modeling
3, to achieve their own pipeline has many defects, such as a difference every stage of execution speed, a stage of the process is too slow, resulting in some stage there has been no task can be handled, and possibly some stage process too quickly, resulting in increasing the capacity of the queue , running out of memory and then crash
4, using the Queue class to make up for their flaws write queue
queue=Queue(1) #参数表示队列中最大任务数量
and use queue.get(), queue.put()to get into and mission
5, also can track the progress of work tasks by task_done method Queue class
Fifth, consider coroutine to run multiple functions
1, thread drawbacks:
- Complex multi-threaded code will make the project become difficult to maintain
- Thread overhead is relatively large, it will slow down the program execution speed itself
- Thread would take a lot of memory, each thread executing approximately 8MB, a dozen threads can afford, but hundreds of thousands of functions with a thread of execution will be a problem
2, Python coroutine avoid the above problems and achieve coroutine actually a generator extension function call overhead required to start the generator similar to coroutines, is active coroutine only about less than 1KB of memory
3, the working principle of the coroutine: execution whenever the generator function to the expression yield and consumes a code generator to a value generated by the send method, generator treats it as the execution result yield expression, while the right variable yield, as a pop-value, as the return value of the send method
#统计最小值
def minimize():
current=yield
while True:
value=yield current
current=min(value,current)
it=minimize()
# 在调用send前，先调用next()
next(it)# 将生成器推进到第一条yield表达式位置
print(it.send(10))
print(it.send(5))
4, coroutine with a similar thread, is an independent function, consumed by an external incoming data and generates the output response is different thread, coroutine will pause at each yield expression, the outside world until the call again after the send method will be performed to the next yield expression
5, the output may utilize the generated value generator, the other generator functions to advance, connecting a plurality of function generators can simulate the behavior of concurrent Python threads, so that looks like a program run multiple functions
Having in mind true parallel computing (multi-processor) with concurrent.futures
1, using the built concurrent.futures multiprocessing module with another built-in module, which in practice will form sub-process, a plurality of parallel running interpreter, using multi-core CPU to enhance execution speed, since the sub master interpreter separation processes and, so they GIL global Interpreter lock is also independent of each other
pool=ProcessPoolExecutor(max_workers=2)# 最大工作数应与CPU核心数相同
#将函数映射到数据列表上去
results=list(pool.map(func,alist))
2, ProcessPoolExecutor based multiprocessing module by the use of the underlying mechanism is provided, following the completion of the gradual
- The alist for each data to map
- A pickle serialization module data into binary form
- , The sequence of the data transmitted from the master interpreter interpreter process to the child process by a local socket (local socket)
- Child process database pickle deserialization, reduced to python objects
- Function func injection module comprising
- Input data for each of the sub-processes in parallel, run the function func
- The results of the run serialize
- The serialized copy the results to the primary process through the socket
- Main deserialization process, reducing python objects
- The main process results obtained by each sub-process of merging into one list and return
3, multiprocessing high overhead because of the serialization and deserialization operation, to other portions of the program need not run the function shared state, and the main process and the sub process requires only a small amount of data transfer, a number of tasks to complete operations speaking, this program is more appropriate. If the operation to be performed do not meet the above characteristics, then the cost may not be generated by parallel programming to enhance the computing speed, in this case, to turn to high-level mechanism multiprocessing, such as shared memory, cross-process lock, queues, agents, etc. but with these characteristics is very complex, it is recommended less touch
4, supplement, causing performance bottlenecks and high call rates and high performance requirements of section uses C language extensions to write, but a larger workload, but also may lead BUG