[Python] parallel processes

Four stages of design concurrent programs (PCAM design methodology):

1. Division (Partitioning): broken down into smaller tasks, pioneering concurrency

2. Communication (Communication): determining the data exchange between the various tasks, monitor the rationality of division;

3. Combination (Agglomeration): according to local tasks, combined into a larger mission;

4. Mapping (Mapping): Each task is assigned to the processor, to improve the performance of the algorithm.

---

Example:

import multiprocessing

def make_data(queue, num, work_nums):
    for i in range(num):
        queue.put(i)
    for i in range(work_nums):
        queue.put(None)

def handle_data(queue, share_value, lock):
    while True:
        data = queue.get()
        if data is None:
            break
        lock.acquire()
        share_value.value = share_value.value + data
        lock.release()

if __name__ == "__main__":
    queue = multiprocessing.Queue()  # 进程间通信所用
    share_value = multiprocessing.Value("i", 0)  # 进程间共享所用
    lock = multiprocessing.Lock()  # 进程间共享内存时，采用锁同步机制
    num = 10000  #
    work_nums = 5  # work进程个数
    sub_process = []  # 处理数据进程集合

    master_process = multiprocessing.Process(target=make_data, args=(queue, num, work_nums, ))  # 生成数据进程
    for i in range(work_nums):
        sub_process1 = multiprocessing.Process(target=handle_data, args=(queue, share_value, lock,))
        sub_process.append(sub_process1)

    master_process.start()
    for p in sub_process:
        p.start()
        #p.setDaemon(True)

    master_process.join()
    for p in sub_process:
        p.join()

    # 结果对比
    result = 0
    for i in range(num):
        result = result + i
    print("result should be " + str(result))
    print("fact is " + str(share_value.value))

Output:

result should be 49995000

fact is 49995000

---

Process([group [, target [, name [, args [, kwargs]]]]])

　　group: thread group, has not yet achieved a library reference in a presentation must be None; 
　　target: the method to execute; 
　　name: process name; 
　　args / kwargs: method parameters to be passed.

Examples of methods:

　　is_alive (): Returns the process is running.

　　join ([timeout]): obstruction of the process of the current context, this method is called until the process is terminated or reaches the specified timeout (optional parameter).

　　start (): the process is ready, waiting for the CPU scheduling

　　run (): start () method calls the run, if not established when the incoming target instance of the process, start the implementation of the default run () method.

　　terminate (): Whether the task is completed, immediately stop the work process

Attributes:

　　daemon: setDeamon functions and threads as daemon

　　ExitCode (process runtime None, if -N, representing the end signal N)

　　name: the name of the process.

　　pid: process ID.

       IS / setDaemon (BOOL) : Gets / sets a background thread (the default foreground thread (False)). (Set prior to start)

　　　　If a background thread, the main thread execution process, the background thread is also carried out after the main thread is finished, the background thread whether successful or not, the main thread and the background thread are stopped
       　　if a foreground thread, the main thread execution process, the foreground thread also carried out after the main thread is finished, waiting foreground thread is also performed after the completion of the program stops

     

   join a timeout parameter:   

•  When setting up a daemon thread [setDaemon (True)], the meaning is the main thread to child thread timeout waiting time will kill the child thread, and finally exit the program. So, if there are 10 sub-threads, all waiting time is cumulative and each timeout. Simply put, is to give each child a timeout thread of time, let him go to execution, time is up, no matter the task has not been completed, directly kill.
• When not set a daemon thread, the main thread will wait for the timeout accumulated and this period of time, the time comes, the end of the main thread, but did not kill the child thread, the child thread can still continue until the child thread the end of all, the program drop out.
   

 

---

Thread to use:

import threading
import time
#方法一：将要执行的方法作为参数传给Thread的构造方法
def action(arg):
    time.sleep(1)
    print 'the arg is:%s\r' %arg

for i in xrange(4):
    t =threading.Thread(target=action,args=(i,))
    t.start()

print 'main thread end!'

#方法二：从Thread继承，并重写run()
class MyThread(threading.Thread):
    def __init__(self,arg):
        super(MyThread, self).__init__()#注意：一定要显式的调用父类的初始化函数。
        self.arg=arg
    def run(self):#定义每个线程要运行的函数
        time.sleep(1)
        print 'the arg is:%s\r' % self.arg

for i in xrange(4):
    t =MyThread(i)
    t.start()

print 'main thread end!'

 

ref : 

       link 

       link2

       python - threading multithreaded summary

       Detailed threading modules: basic concepts, join the main thread blocking and setDaemon Guardian