The application of a daemon process: In fact, the concept of a daemon process is added to our producer and consumer model, so that our process can exit normally after the task is executed.
import time import random from multiprocessing import Process,JoinableQueue #We import a joinableQueue module here, def consumer(name,q): while True: res=q.get() if res is None:break time.sleep(random.randint(1,3)) print( ' \033[46m Consumer ==="%s ate %s\033[0m ' % (name,res)) q.task_done() def producer(name,q,food): for i in range(5): time.sleep (random.randint ( 1 , 2 )) res='%s%s' %(food,i) q.put(res) print( ' \033[45m Producer ==="%s produced %s\033[0m ' % (name,res)) if __name__ == ' __main__ ' : q=JoinableQueue() p1=Process(target=producer,args=('egon',q,'包子')) p2 =Process(target=producer,args=( ' Liu Qingzheng ' ,q, ' swill ' )) p3 =Process(target=producer,args=( ' Yang Jun ' ,q, ' rice ' )) c1=Process(target=consumer,args=('alex',q)) c2=Process(target=consumer,args=('梁书东',q)) c1.daemon=True c2.daemon=True p1.start() p2.start() p3.start() c1.start() c2.start() # Make sure the producer has indeed finished producing p1.join() p2.join() p3.join() # After the producer completes production, get the total number of elements in the queue, and then the q.join() line of code will not run until the total number of elements becomes 0 q.join() # q.join()# Once it ends, it means that the queue is indeed emptied, and the consumer has indeed fetched all the data. print( ' main process ends ' ) print('main process ends')
will be executed after all child processes end
The difference between multi-process and multi-threaded daemons:
#Guardian : The difference between processes and threads. : as long as there is no executable code in this process. Then the daemon process ends. #In the process, there is only one main thread in the main process we talked about earlier, so after the main process (that is, after the main thread code is executed) is completed, # the daemon process ends. #Then in the thread: After the main thread ends, there are non-guardian child threads, the code needs to be run, and the main thread will wait for the end of the operation of the child thread to end, # That is to say, there are still child threads in this process that have not finished running. So the daemon process will not end until all non-daemon child process processes have ended.
Two process pools:
In the process of using Python, multi-process is one of the means for us to achieve concurrency, but there are a few problems to pay attention to:
- It is obvious that the tasks that need to be executed concurrently are usually much larger than the number of cores
- It is impossible for an operating system to open processes indefinitely. Usually, there are several cores to open several processes.
- Too many processes are opened, and the efficiency will be reduced (opening processes needs to occupy system resources, and processes with excess number of cores cannot be parallelized)
For example, when the number of objects to be operated is not large, you can directly use the Process in multiprocessing to dynamically generate multiple processes. A dozen or so are fine, but if there are hundreds or thousands. . . It is too cumbersome to limit the number of processes manually, and the effect of the process pool can be exerted at this time.
We can control the number of processes by maintaining a process pool, such as the process mode of httpd, specifying the minimum number of processes and the maximum number of processes...
ps: For advanced applications of remote procedure calls, a process pool should be used. Pool can Provide a specified number of processes for the user to call. When a new request is submitted to the pool, if the pool is not full, a new process will be created to execute the request; but if the number of processes in the pool has reached If the maximum value is specified, the request will wait until a process in the pool ends, and the process in the process pool will be reused.
The class that creates the process pool: If you specify numprocess as 3, the process pool will create three processes from scratch, and then use these three processes to perform all tasks from beginning to end, and will not start other processes
Three multithreading:
Then we need to know what is a thread: it refers to the process of a pipeline work.
In fact, the process we mentioned earlier is not an execution unit. A process is actually a resource unit, and what really executes is the thread in it. A process has its own thread, which by default has a main thread. A thread is the unit of execution on the CPU. For example, in our factory, each opening of a workshop is equivalent to opening a process. The resources between workshops are independent of each other, so the resources between multiple assembly lines in the same workshop are shared.
We said before that every time a process is started, it actually sends a request to the operating system, opens up a memory space, copies the code and data into it, and then the CPU fetches it from the memory space and executes the code. Then the opening of the thread is much simpler than that of the process, and a ''sub-thread'' can be opened almost at the same time as the request is made ( why are the quotation marks here? Because there is no master and child between threads, and between threads is equal status ). No need to re-apply for memory space.
Therefore, a process is only used to group resources together (a process is just a resource unit, or a collection of resources), and a thread is the execution unit on the CPU.
The concept of multithreading (that is, multiple control threads) is that there are multiple control threads in a process, and multiple control threads share the address space of the process, which is equivalent to multiple pipelines in a workshop, all sharing the resources of a workshop .
1 Compare the gap between processes and threads:
1) Threads in a unified process share resources in the process, and thread resources in different processes must be isolated.
2) The overhead of creating a thread is much smaller than creating a process
3) There is no primary and secondary between threads
4) The main thread and the main process are waiting for the difference between the child thread and the child process:
The main thread waits for the end of the sub-threads to ensure that all sub-threads end and then recycle the resources of the entire process. The main process waits for the end of the child process to recycle the resources of the child process. The resources are shared between threads, and the processes are isolated from each other.
Let's look at an example of a thread:
from threading import Thread def task(name): print('%s is running'%name) if __name__ == '__main__': t=Thread(target=task,args=('feng',)) t.start() print( ' main ' .center( 50 , ' * ' )) # The result of the operation is: # feng is running ************************Main ******************** ****
We can see that the thread generation is very fast.
Four deadlocks and recursive locks:
from threading import Thread,Lock,RLock import time # mutexA=Lock() # mutexB=Lock() mutexA=mutexB=RLock() class MyThread(Thread): def run(self): self.f1() self.f2() def f1(self): mutexA.acquire() print ( ' %s got the A lock ' %self.name) # when you get the A lock, add a mutexB.acquire() to the A lock print ( ' %s got the B lock ' % self.name) mutexB.release() mutexA.release() def f2(self): mutexB.acquire() print ( ' %s got the B lock ' % self.name) time.sleep(0.1) mutexA.acquire() print ( ' %s got A lock ' % self.name) mutexA.release() mutexB.release() if __name__ == '__main__': for i in range(10): t=MyThread() t.start() # t1=MyThread() # t1.start() # # t2=MyThread() # t2.start() # # t3=MyThread() # t3.start() print('主')
signal: