First, the process synchronization - lock
Do not share data between processes, but share the same set of file systems, so access to the same file, or print with a terminal, there is no problem. The share is caused by competition, the results of the competition is to bring confusion, how to control, that is, the lock processing.
from multiprocessing import Process,Lock
import os,time
def work(lock):
lock.acquire()
print('%s is running' %os.getpid())
time.sleep(2)
print('%s is done' %os.getpid())
lock.release()
if __name__ == '__main__':
lock=Lock()
for i in range(3):
p=Process(target=work,args=(lock,))
p.start()
Locking ensures that when multiple processes modifying the same piece of data, at the same time only one task can be modified, is about parallel to serial modification, are slowing down, but avoid the competition, ensure data security.
Although the data can be shared to achieve inter-process communication documents, but: 1. Low efficiency (based on a shared data file, and the file is the data on the hard disk) 2 needs its own lock handle.
Second, the queue
mutiprocessing problem based IPC module provides a message communication mechanism (queues, pipes), high efficiency (multiple processes to share a data memory), improving the locking.
ipc mechanism: Process Communications
Pipeline: pipe based on a shared memory space
Queue: pipe + Lock
Queues and pipes are the data stored in the memory, the queue is based on the (pipeline + lock) to achieve, let us free from the lock complex problems.
Methods:
1. q.put方法用以插入数据到队列中，put方法还有两个可选参数：blocked和timeout。如果blocked为True（默认值），并且timeout为正值，该方法会阻塞timeout指定的时间，直到该队列有剩余的空间。如果超时，会抛出Queue.Full异常。如果blocked为False，但该Queue已满，会立即抛出Queue.Full异常。
2. q.get方法可以从队列读取并且删除一个元素。同样，get方法有两个可选参数：blocked和timeout。如果blocked为True（默认值），并且timeout为正值，那么在等待时间内没有取到任何元素，会抛出Queue.Empty异常。如果blocked为False，有两种情况存在，如果Queue有一个值可用，则立即返回该值，否则，如果队列为空，则立即抛出Queue.Empty异常.
3. q.get_nowait():同q.get(False)
4. q.put_nowait():同q.put(False)
5. q.empty():调用此方法时q为空则返回True，该结果不可靠，比如在返回True的过程中，如果队列中又加入了项目。
6. q.full()：调用此方法时q已满则返回True，该结果不可靠，比如在返回True的过程中，如果队列中的项目被取走。
7. q.qsize():返回队列中目前项目的正确数量，结果也不可靠，理由同8. q.empty()和q.full()一样Normally:
from multiprocessing import Process,Queue
q = Queue()
q.put('123')
q.put([456])
q.put(000)
print(q.get())
print(q.get())
print(q.get())
print(q.get())
123
[456]
0Queue full value continue to exist, can block:
from multiprocessing import Process,Queue
q = Queue(4)
q.put('123')
q.put([456])
q.put(789)
q.put(000)
q.put('aaa')
print(q.get())
print(q.get())
print(q.get())
print(q.get()) q.put(block=True)
from multiprocessing import Process,Queue
q = Queue(3)
q.put('123',block=True,timeout=2)
q.put('123',block=True,timeout=2)
q.put('123',block=True,timeout=2)
q.put('456',block=True,timeout=3)
print(q.get())
print(q.get())
print(q.get())
print(q.get())
raise Full
queue.Full q.get(block=True)
from multiprocessing import Process,Queue
q = Queue()
q.put('333')
q.get()
q.get(block=True,timeout=4)
raise Empty
queue.Empty q.put(block=False)
from multiprocessing import Process,Queue
q = Queue(2)
q.put('123')
q.put('123')
q.put('123',block=False,)
raise Full
queue.Full q.get_nowait():
from multiprocessing import Process,Queue
q = Queue(1)
q.put('123')
q.put_nowait("666")
raise Full
queue.FullThird, the producer-consumer model
Producer and consumer use patterns in concurrent programming can solve the vast majority of concurrency problems. The overall pattern to increase the speed of data processing programs by working ability to balance production and consumption thread thread.
Producer-consumer model is the strong coupling problem is solved by a producer and consumer of container. Producers and consumers do not directly communicate with each other, and to communicate by blocking queue, so producers after completion of processing of production data without waiting for the consumer, direct throw blocking queue, consumers do not find a producer to data, but taken directly from blocking the queue, the queue is equivalent to a blocking buffer, balance the producers and consumers of processing power.
Producer consumer model 1:
from multiprocessing import Process,Queue
def producer(q,name,food):
for i in range(1,10):
print(f'{name}生产了{food}{i}')
res = f'{food}{i}'
q.put(res)
q.put(None)
def consumer(q,name):
while True:
res = q.get(timeout=5)
if res is None:
break
print(f'{name}吃了{res}')
if __name__ == '__main__':
q = Queue()
p1 = Process(target=producer,args=(q,'狗不理','包子'))
c1 = Process(target=consumer,args=(q,'狗'))
p1.start()
c1.start()
狗不理生产了包子1
狗不理生产了包子2
狗不理生产了包子3
狗不理生产了包子4
狗不理生产了包子5
狗不理生产了包子6
狗不理生产了包子7
狗不理生产了包子8
狗不理生产了包子9
狗吃了包子1
狗吃了包子2
狗吃了包子3
狗吃了包子4
狗吃了包子5
狗吃了包子6
狗吃了包子7
狗吃了包子8
狗吃了包子9Producer consumer model 2:
from multiprocessing import Process, Queue
import time,random
def producer(q,name,food):
for i in range(1,4):
print(f'{name}生产了{food}{i}')
time.sleep(random.randint(1,3))
res = f'{food}{i}'
q.put(res)
def consumer(q,name):
while True:
res = q.get(timeout= 5)
if res is None:
break
time.sleep(random.randint(1,3))
print(f'{name}吃了{res}')
if __name__ == '__main__':
q = Queue()
p1 = Process(target=producer,args=(q,'狗不理','包子'))
p2 = Process(target=producer, args=(q, '猫不理', '包子'))
p3 = Process(target=producer,args=(q,'猪不理','包子'))
c1 = Process(target=consumer,args=(q,'猪'))
c2 = Process(target=consumer,args=(q,'狗'))
p1.start()
p2.start()
p3.start()
c1.start()
c2.start()
p1.join()
p2.join()
p3.join()
q.put(None)
q.put(None)
狗不理生产了包子1
猪不理生产了包子1
猫不理生产了包子1
猫不理生产了包子2
猪不理生产了包子2
狗不理生产了包子2
猫不理生产了包子3
猪吃了包子1
猪不理生产了包子3
猪吃了包子1
狗吃了包子1
狗不理生产了包子3
狗吃了包子2
猪吃了包子2
狗吃了包子3
猪吃了包子3
狗吃了包子2
猪吃了包子3Producer Consumer Model 3:
from multiprocessing import Process, Queue,JoinableQueue
import time,random
def producer(q,name,food):
for i in range(1,4):
print(f'{name}生产了{food}{i}')
time.sleep(random.randint(1,3))
res = f'{food}{i}'
q.put(res)
def consumer(q,name):
while True:
res = q.get()
time.sleep(random.randint(1,3))
print(f'{name}吃了{res}')
q.task_done()
if __name__ == '__main__':
q = JoinableQueue()
p1 = Process(target=producer,args=(q,'狗不理','包子'))
p2 = Process(target=producer, args=(q, '猫不理', '包子'))
p3 = Process(target=producer,args=(q,'猪不理','包子'))
c1 = Process(target=consumer,args=(q,'猪'))
c2 = Process(target=consumer,args=(q,'狗'))
p1.start()
p2.start()
p3.start()
c1.daemon = True
c2.daemon = True
c1.start()
c2.start()
p1.join()
p2.join()
p3.join()
q.join()
狗不理生产了包子1
猫不理生产了包子1
猪不理生产了包子1
猫不理生产了包子2
狗不理生产了包子2
猫不理生产了包子3
猪不理生产了包子2
狗吃了包子1
猪吃了包子1
狗吃了包子2
狗不理生产了包子3
猪不理生产了包子3
狗吃了包子3
猪吃了包子1
猪吃了包子2
狗吃了包子2
狗吃了包子3
猪吃了包子3There .task_done () method within joinableQueue, .put () method can be placed in a task within the queue is completed, may be thought of as a queue counter: put +1 task_done -1, the counter is not 0 when the join () method blocked by wait counter is 0 after.
Fourth, thread
In traditional operating systems, each process has an address space, but by default there is a thread of control
Thread name suggests, is an assembly line process work, a pipeline must belong to a workshop, a workshop of the working process is a process
Workshop leader to integrate resources together, is a resource unit, and a workshop with at least one pipeline
Assembly line work needs power supply, power supply is equivalent to cpu
So, just the process used to concentrate resources together (process just a resource unit, or set of resources), and on the implementation of the unit thread is the cpu.
Multithreading (i.e. multiple threads of control) concept is that multiple threads of control within a process, controlling the plurality of threads shared address space of the process, a plurality of lines corresponding to a workshop, all share a workshop resources.