Quickly learn Python programming of concurrent engineering implementation (under)

About
program ape a thoughtful, lifelong learning practitioners, is currently in a start-up team any team lead, technology stack involves Android, Python, Java, and Go, this is the main technology stack our team.
GitHub: https://github.com/hylinux1024
micro-channel public number: Lifetime developer (angrycode)

0x00 using the process concurrency

The article describes the use of threads. However, Pythondue to the Global Interpreter Lock(Global Interpreter Lock GIL) exist, each thread in the implementation need to get to this GIL, at the same time only one thread gets executed explain the lock, Pythonthe thread does not execute concurrently in the true sense, multithreading the efficiency is not necessarily more efficient than single-threaded.
If you want to take full advantage of modern multi-core CPUconcurrency, it is necessary to use multipleprocessingmodule of.

0x01 multipleprocessing

And the use of threads threadingmodule similar to the multipleprocessingmodule offers many advanced API. The most common is Poolthe object, and use its interface is very easy to write code that can be executed concurrently.

from multiprocessing import Pool

def f(x):
    return x * x

if __name__ == '__main__':
    with Pool(5) as p:
        # map方法的作用是将f()方法并发地映射到列表中的每个元素
        print(p.map(f, [1, 2, 3]))

# 执行结果
# [1, 4, 9]

About Poolhereinafter also referred to, where we first look Process.

Process

To create a process can use Processthe class, using the start()method to start the process.

from multiprocessing import Process
import os

def echo(text):
    # 父进程ID
    print("Process Parent ID : ", os.getppid())
    # 进程ID
    print("Process PID : ", os.getpid())
    print('echo : ', text)

if __name__ == '__main__':
    p = Process(target=echo, args=('hello process',))
    p.start()
    p.join()
    
# 执行结果
# Process Parent ID :  27382
# Process PID :  27383
# echo :  hello process

Process pool

As mentioned at the beginning of the multiprocessingmodule provides a Poolclass can easily achieve simple multi-process scenarios.
It mainly has the following interfaces

apply(func[, args[, kwds]])
Execution func(args,kwds)method, the method returns before the end of the clog.
apply_async(func[, args[, kwds[, callback[, error_callback]]]])
Asynchronous execution func(args,kwds), will immediately return a resulttarget, if you specify callbackparameters, the results will be returned through the callback method, you can also specify a callback method execution errorerror_callback()
map(func, iterable[, chunksize])
Similarly built-in functions map()may be performed concurrently func, a synchronization method
map_async(func, iterable[, chunksize[, callback[, error_callback]]])
Asynchronous versionmap
close()
Close process pool. When all worker processes in the pool were completed, the process will exit.
terminate()
Termination process pool
join()
Waiting for the process of executing the work, you must first call close()orterminate()

from multiprocessing import Pool

def f(x):
    return x * x

if __name__ == '__main__':
    with Pool(5) as p:
        # map方法的作用是将f()方法并发地映射到列表中的每个元素
        a = p.map(f, [1, 2, 3])
        print(a)
        # 异步执行map
        b = p.map_async(f, [3, 5, 7, 11])
        # b 是一个result对象，代表方法的执行结果
        print(b)
        # 为了拿到结果，使用join方法等待池中工作进程退出
        p.close()
        # 调用join方法前，需先执行close或terminate方法
        p.join()
        # 获取执行结果
        print(b.get())

# 执行结果
# [1, 4, 9]
# <multiprocessing.pool.MapResult object at 0x10631b710>
# [9, 25, 49, 121]

map_async()And apply_async()the execution will return an class multiprocessing.pool.AsyncResultobject, it's get()possible to get the results, ready()you can determine AsyncResultwhether the result is ready.

Transmission of data between processes

multiprocessingModule provides data sharing between the two modes for the process: Queue ( Queue) and pipe ( Pipe)

QueueIs thread-safe, the process is safe. Use Queuecan achieve data sharing between processes, for example, the following demoneutron process putan object in the main process will be able getto this object.
Any serializable object can Queuebe transmitted.

from multiprocessing import Process, Queue

def f(q):
    q.put([42, None, 'hello'])

if __name__ == '__main__':
    # 使用Queue进行数据通信
    q = Queue()
    p = Process(target=f, args=(q,))
    p.start()
    # 主进程取得子进程中的数据
    print(q.get())  # prints "[42, None, 'hello']"
    p.join()

# 执行结果
# [42, None, 'hello']

Pipe()Returns a pair of pipes connected by Connectionthe object. These two objects will be appreciated as the ends of the pipe, which by send()and recv()send and receive data.

from multiprocessing import Process, Pipe

def write(conn):
    # 子进程中发送一个对象
    conn.send([42, None, 'hello'])
    conn.close()

def read(conn):
    # 在读的进程中通过recv接收对象
    data = conn.recv()
    print(data)

if __name__ == '__main__':
    # Pipe()方法返回一对连接对象
    w_conn, r_conn = Pipe()

    wp = Process(target=write, args=(w_conn,))
    rp = Process(target=read, args=(r_conn,))

    wp.start()
    rp.start()

# 执行结果
# [42, None, 'hello']

It should be noted that the two processes can not simultaneously be a connection object sendor recvoperation.

Synchronize

We know that synchronization is achieved by locking mechanism between threads, processes, too.

from multiprocessing import Process, Lock
import time

def print_with_lock(l, i):
    l.acquire()
    try:
        time.sleep(1)
        print('hello world', i)
    finally:
        l.release()

def print_without_lock(i):
    time.sleep(1)
    print('hello world', i)

if __name__ == '__main__':
    lock = Lock()

    # 先执行有锁的
    for num in range(5):
        Process(target=print_with_lock, args=(lock, num)).start()
    # 再执行无锁的
    # for num in range(5):
    #     Process(target=print_without_lock, args=(num,)).start()

Lock code will turn print per second

hello world 0
hello world 1
hello world 2
hello world 3
hello world 4

If you do not lock the code, the results on my computer like this

hello worldhello world  0
1
hello world 2
hello world 3
hello world 4

In addition Lock, also include RLock, Condition, Semaphoreand Eventsynchronization primitives such as inter-process. Its usage is very similar to the inter-thread synchronization primitives. APIUse the links referenced in the document refer to the end of the text.
Data sharing between processes in the project should be prioritized using queues or pipes.

0x02 summary

In this paper, multiprocessingthe module common APIsimple introduction. It tells the story Processand Poolcommon usage, also introduced inter-process data mode: queues and pipes. Finally, a simple understanding of the inter-process synchronization primitives.
By Part learn contrast, the contents of this article should be easier to grasp.