python of concurrent programming - communication between processes

Communication between processes

1. Mutex

   Do not share data between processes, but share the same set of file system, or with a printing terminal is not a problem. But sharing has led to competition, if unchecked will cause confusion. as follows:

   from multiprocessing import Process
   import time
   import random
   def task1():
       print('task1开始打印')
       time.sleep(random.randint(1,3))
       print('task1打印完成')
   def task2():
       print('task2开始打印')
       time.sleep(random.randint(1,3))
       print('task2打印完成')
   def task3():
       print('task3开始打印')
       time.sleep(random.randint(1,3))
       print('task3打印完成')
   if __name__ == '__main__':
       p1 = Process(target=task1,)
       p2 = Process(target=task2,)
       p3 = Process(target=task3,)
       p1.start()
       p2.start()
       p3.start()
   # task1开始打印
   # task2开始打印
   # task3开始打印
   # task1打印完成
   # task3打印完成
   # task2打印完成

   Locking can become complicated by the use of the serial, sacrificing efficiency, avoid competition

   from multiprocessing import Process
   from multiprocessing import Lock
   import time
   import random
   def task1(lock):
       lock.acquire()
       print('task1开始打印')
       time.sleep(random.randint(1,3))
       print('task1打印完成')
       lock.release()
   def task2(lock):
       lock.acquire()
       print('task2开始打印')
       time.sleep(random.randint(1,3))
       print('task2打印完成')
       lock.release()
   def task3(lock):
       lock.acquire()
       print('task3开始打印')
       time.sleep(random.randint(1,3))
       print('task3打印完成')
       lock.release()
   if __name__ == '__main__':
       lock = Lock()
       p1 = Process(target=task1,args=(lock,))
       p2 = Process(target=task2,args=(lock,))
       p3 = Process(target=task3,args=(lock,))
       p1.start()
       p2.start()
       p3.start()
   # task1开始打印
   # task1打印完成
   # task2开始打印
   # task2打印完成
   # task3开始打印
   # task3打印完成

   Use multiple processes to grab votes:

   from multiprocessing import Process
   from multiprocessing import Lock
   import time
   import random
   import json
   def search():
       time.sleep(random.random())  # 模拟读取数据网络延迟
       with open('db.json',encoding='utf-8') as f:
        dic = json.load(f)
    print(f'剩余票数{dic["count"]}')
   def get():
       with open('db.json',encoding='utf-8') as f:
           dic = json.load(f)
       time.sleep(random.random())  # 模拟读取数据网络延迟
       if dic['count'] > 0:
           dic['count'] -= 1
           time.sleep(random.random())  # 模拟写数据网络延迟
           with open('db.json',encoding='utf-8',mode='w') as f:
               json.dump(dic,f)
           print(f'{os.getpid()}用户购买成功')
       else:
           print('票没了')
   def task(lock):
       search()
       lock.acquire()
       get()
       lock.release()
   if __name__ == '__main__':
       lock = Lock()
       for i in range(5):
        p = Process(target=task,args=(lock,))
        p.start()
   # 剩余票数1
   # 剩余票数1
   # 14004用户购买成功
   # 剩余票数0
   # 剩余票数0
   # 票没了
   # 剩余票数0
   # 票没了
   # 票没了
   # 票没了

   Although it can share data files to achieve inter-process communication, but the question is:

   1. Low efficiency (sharing data residing files, and data files are on the hard disk)
   2. It needs its own lock handle, easy to form a deadlock, recursive lock (Note that a process can only use a lock)

2. queue

   multiprocessing module supports two forms: the queue and the pipeline, these two methods are transferring data.

   Queue is in memory of a container, the biggest feature is FIFO, fully supports the FIFO principle

   1. Create a class queue (and the pipe bottom is in the manner of the cable)

      Queue ([maxsize]): create a shared process queue, the queue Queue is a multi-process safe, you can use Queue to pass data between multiple processes

   2. Parameter Description

      maxsize is the maximum number of entries allowed in the queue, no size limit is omitted

   3. Methods Introduction

      • The main method

        # q.put方法用以插入数据到队列中，put方法还有两个可选参数：blocked和timeout。如果blocked为True（默认值），并且timeout为正值，该方法会阻塞timeout指定的时间，直到该队列有剩余的空间。如果超时，会抛出Queue.Full异常。如果blocked为False，但该Queue已满，会立即抛出Queue.Full异常。
        # q.get方法可以从队列读取并且删除一个元素。同样，get方法有两个可选参数：blocked和timeout。如果blocked为True（默认值），并且timeout为正值，那么在等待时间内没有取到任何元素，会抛出Queue.Empty异常。如果blocked为False，有两种情况存在，如果Queue有一个值可用，则立即返回该值，否则，如果队列为空，则立即抛出Queue.Empty异常.  
        # q.get_nowait():同q.get(False)
        # q.put_nowait():同q.put(False) 
        # q.empty():调用此方法时q为空则返回True，该结果不可靠，比如在返回True的过程中，如果队列中又加入了项目。
        # q.full()：调用此方法时q已满则返回True，该结果不可靠，比如在返回True的过程中，如果队列中的项目被取走。
        # q.qsize():返回队列中目前项目的正确数量，结果也不可靠，理由同q.empty()和q.full()一样

      • Other methods

        # q.cancel_join_thread():不会在进程退出时自动连接后台线程。可以防止join_thread()方法阻塞
        # q.close():关闭队列，防止队列中加入更多数据。调用此方法，后台线程将继续写入那些已经入队列但尚未写入的数据，但将在此方法完成时马上关闭。如果q被垃圾收集，将调用此方法。关闭队列不会在队列使用者中产生任何类型的数据结束信号或异常。例如，如果某个使用者正在被阻塞在get()操作上，关闭生产者中的队列不会导致get()方法返回错误。
        # q.join_thread()：连接队列的后台线程。此方法用于在调用q.close()方法之后，等待所有队列项被消耗。默认情况下，此方法由不是q的原始创建者的所有进程调用。调用q.cancel_join_thread方法可以禁止这种行为

3. Examples of communication between processes

   Communication between using the queue process: simple, easy, do-it-yourself locked. Queue comes obstruction of sustainable access to data.

   # 用队列来存储网络请求，模拟网购抢购
   import os
   from multiprocessing import Queue
   from multiprocessing import Process
   def task(q):
       try:
           q.put(f'{os.gerpid()}',block=False)  # 模拟存放请求
       except Exception:
           return
   if __name__ == '__name__':
       q = Queue(10)
       for i in range(100):
           p = Process(target=task,args=(q,))
           p.start()
       for i in range(1,11):
           print(f'排名第{i}的用户：{q.get()}')    # 获取前十个请求

4. Producer and consumer model

   Producer: Production Data Process

   Consumers: the data produced by the producer for further processing process

   The need for a container as a buffer for productivity and consumption balance between producers and consumers

   Producer consumer model used for concurrency

   from multiprocessing import Process
   from multiprocessing import Queue
   import time
   import random
   def producer(name,q):
       for i in range(1,6):
           time.sleep(random.randint(1,3))
           ret = f'{i}号包子'
           q.put(ret)
           print(f'\033[0;32m 生产者{name}:生产了{ret}\033[0m')
   def consumer(name,q):
       while 1:
           try:
               time.sleep(random.randint(1,3))
               ret = q.get(timeout=5)
               print(f'消费者{name}:吃了{ret}')
           except Exception:
               return
   if __name__ == '__main__':
       q = Queue()
       p1 = Process(target=producer,args=('铁憨憨',q))
       p2 = Process(target=consumer,args=('皮皮寒',q))
       p1.start()
       p2.start()
   # 生产者铁憨憨:生产了1号包子
   #消费者皮皮寒:吃了1号包子
   # 生产者铁憨憨:生产了2号包子
   #消费者皮皮寒:吃了2号包子
   # 生产者铁憨憨:生产了3号包子
   # 生产者铁憨憨:生产了4号包子
   #消费者皮皮寒:吃了3号包子
   # 生产者铁憨憨:生产了5号包子
   #消费者皮皮寒:吃了4号包子
   #消费者皮皮寒:吃了5号包子

5. to sum up

   Communication between processes:

   1. Based on the form + file locks: inefficiency, trouble.
   2. Recommended forms: based queues.
   3. Based Pipeline: Pipeline own lock, the underlying corruption of data loss can occur.

   Multiple processes to seize a resource: serial, orderly, and data security.

   Multiple processes to achieve concurrent effects: producer consumer model.