python multiprocessing multi-process shared variables to achieve and locked

This article describes the python multiprocessing and multi-process shared variable locking implementation, the paper sample code described in great detail, has a certain reference value of learning for all of us to learn or work, you need friends to below as the small series together to learn from it
python multi-process and multi-threading is that we will focus on understanding the part, because a lot of work, if not before and interdependencies, then in fact the order is not very important, the use of words execution of the order would surely cause unnecessary waiting, despite cpu and memory waste, which we want to see.

To solve this problem, the way we multi-threaded or multi-process may be employed (repeat after us multithreading), and is the essential difference between the two. On memory, the process is known to have a separate memory unit in the implementation process, and multiple threads is shared memory, which is a big difference between multi-process and multi-threaded.

Value synchronization using the variables in different processes

In the multi-process, since the memory between processes are isolated from each other, so the method can not be directly read in multiple processes using shared variables, this time you can use multiprocessing library Value shared variables in their isolation process.

Here is an example of a multi-process:

Suppose there is a counter used to record the total number of cycles through the program, every call after a count function of counter will increase by 20, 10 open cycle processes are used in the main function call count, under less than ideal state in 10 process shared counter value to the end of the program should be 200.

from multiprocessing import Process, Value
import time
 
def count(v):
  for i in range(20):
    time.sleep(0.01)
    v.value += 1
 
def main():
  value = Value('i',0)
  processes = [Process(target=count, args=(value,)) for i in range(10)]
 
  for p in processes:
    p.start()
  for p in processes:
    p.join()
 
  print(value.value)
 
if __name__ == '__main__':
 
  for i in range(10):
    main()

To run this example, you get what result?

188
180
168
186
183
179
186
181
166
186

I run this program ten times in the main program, and the end result is between 160-180, in short, no time to 200. This is what causes it?

I believe many people have realized the problem, and that is because the Value in multiprocessing library is fine-grained, Value has a ctypes type of the object has a value attribute to characterize the actual object in memory. Value can guarantee that only one single thread or process in a read or write value value. So look no problems.

However, when the first process to load value value, the program can not stop the process of loading a second old value. Both processes will copy the value to its own private memory and then processed, and written back to the shared values ​​inside.

So then what will happen?

The final share value just received an increase in value once, rather than twice.

Lock locked in the use of different process variables shared
problems above can actually use a very simple way to solve, we just need to call the multiprocessing library Lock (lock) you can ensure that only one process accesses the shared variable. The modified code as follows:

from multiprocessing import Process, Value, Lock
from time import sleep
 
def count(x,lock):
  for i in range(20):
    sleep(0.01)
    with lock:
      x.value += 1
 
 
def main():
  counter = Value('i',0)
  lock = Lock()
  processes = [Process(target=count,args=(counter,lock)) for i in range(10)]
  for p in processes:
    p.start()
  for p in processes:
    p.join()
 
  print(counter.value)
 
if __name__ == '__main__':
  for i in range(10):
    main()

As a result, the output 200 will be a constant.

Some supplements

1. Call get_lock () function is
   in fact the Value package already includes the concept of the lock, if you call get_lock () function can automatically lock to shared variables. In fact, this is more recommended way, because it does not need to call two packages at the same time. amend as below:

from multiprocessing import Process, Value
from time import sleep
 
def count(x):
  for i in range(20):
    global counter # 声明全局变量
    sleep(0.01)
    with counter.get_lock(): # 直接调用get_lock()函数获取锁
      x.value += 1
 
def main():
  processes = [Process(target=count, args=(counter,)) for i in range(10)]
  for p in processes:
    p.start()
  for p in processes:
    p.join()
 
  print(counter.value)
 
if __name__ == '__main__':
  counter = Value('i', 0) # 需要把全局变量移到主程序
  main()

The above procedure is more clear, and the final result is 200.

2. Use multiprocessing.RawValue
   entire multiprocessing bag just called Value and Lock can also be unified multiprocessing.RawValue replaced.
   Content on more than how many, and finally to recommend a good reputation in the number of public institutions [programmers], there are a lot of old-timers learning

Skills, learning experience, interview skills, workplace experience and other share, the more carefully prepared the zero-based introductory information, information on actual projects,

The method has timed programmer Python explain everyday technology, to share some of the learning and the need to pay attention to small details