Python3 Quick Start (nine) - Python3 concurrent programming

Python3 Quick Start (nine) - Python3 concurrent programming

A, Python threading module

1. Introduction thread

A standard thread by the thread ID, the current instruction pointer (PC), register set and stack components. A thread is a physical process, is the basic unit of independent scheduling and dispatch system, the thread itself does not have the system resources with all other threads share the process resources within the process. A process has at least one thread, and as a program entry, i.e. the main thread, another thread called work threads.
     Multithreading refers to achieve concurrent execution of multiple threads technology from the software or hardware. Support multi-threading capabilities of the computer due to hardware support and is capable of executing multiple threads at the same time, thereby enhancing the overall processing performance.

2, thread state

There are threads ready, blocking, running three basic states. Ready state means that all the conditions have threads running, waiting for execution in the CPU; running state refers thread owns the CPU is running; blocking state is not executable threads waiting on an event logic.
Interconversion of the three states as shown below:

2, threading threading module

Python3 by _threadproviding support for threads and threading two modules.
_threadIt provides a low-level, original thread lock and simple, compared to the threading module limited functionality, compatibility support programs that have been abandoned thread module.
In addition to the threading module comprises _threada method of all methods modules, also provided:
threading.currentThread (): returns the current thread variable.
threading.enumerate (): Returns a list of running threads. Refers to the thread starts running, before the end, it does not include a thread before starting and after termination.
threading.activeCount (): Returns the number of threads that are running, and len (threading.enumerate ()) have the same result.
Thread class provides as follows:
RUN (): method of thread activity to FIG.
start (): start thread activity.
join ([time]): Wait until the thread suspended. Block the calling thread until the thread's join () method is called suspension - normal exit or throw an unhandled exception - or the optional timeout occurs.
isAlive (): Returns the thread is active.
getName (): Returns the thread name.
setName (): Set the thread name.

3, multiprocessing module

multiprocessing module is cross-platform version of the multi-process module, provides a Process class represents a process object. When you create a child process, only you need to pass a parameter to perform functions and function to create a Process instance.
Process（self,group=None,target=None,name=None,args=(),kwargs=()）
group parameter is not used, the value is always None.
target object that represents the call, the child process to execute the task.
name can be named as a child process.
args junction specified target transfer function of position parameters, is in the form of a tuple, there must be a comma, such as: args = ( 'monicx',)
kwargs target node specified keyword parameter transfer function, a dictionary, such as kwargs = { ' name ':' monicx ',' Age ':} 18 is
process as follows:
start (): start process, and calls the run () method of the child process.
run (): the process started running into the method invocation target specified function, the child class must implement the run method in the run.
terminate (): forced to terminate the process, will not carry out any cleaning operation if a process creates a child process, the child process will become a zombie process; if the process also holds a lock, the lock will not release process, which led to a deadlock.
is_alive (): determine whether the process is "alive" state.
join (timeout): Let the primary process child process to wait for a process to finish before continuing execution of the main process. timeout is optional timeout, more than one time will not wait for the main course.

4, the global interpreter lock GIL

Python does not support multi-threading in the true sense. Python module provides multi-threaded, multi-threaded but if you want to increase the speed of the code does not recommend the use of multi-threading module. Python has a global lock Global Interpreter Lock (GIL), a global lock ensures that multiple threads at any time, only one will be executed. Thread execution speed is very fast, you will think threads are executed in parallel, but actually take turns to perform. After GIL treatment will increase the cost of thread execution.
Global Lock GIL (Global interpreter lock) is not a Python features, but a concept at the time of implementation of the Python parser (CPython) introduced. Python has CPython, PyPy, Psyco, such as different Python execution environment, which JPython no GIL. CPython is the default under most environmental Python execution environment, GIL is not a Python features, Python can not rely on the GIL.
GIL limits the same time, only one thread is running, you can not take advantage of multi-core CPU. GIL is essentially a mutex, all will run concurrently become a serial, in order to control the sharing of data within the same time can only be modified by a task, thus ensuring data security. Python in a process, not only is the level of the main thread or threads from other threads main Chengkai Qi, as well as the interpreter turned garbage collection explanation. In the process, all the data is shared, as a kind of code data will be shared by all threads, multiple threads first visit to the interpreter code that is executed to get the permissions, and then to the target code interpreter code to perform, the interpreter code is shared by all threads, so the garbage collector thread may also have access to the interpreter code execution away, so in order to ensure data security need to lock handle, namely GIL.
Because of the GIL, the same process in the same time only one thread is executed. Multi-core CPU can be done in parallel computing, and therefore can enhance the multi-core computing performance, but in the event of CPU I / O blocking, still need to wait, so multi-core CPU to I / O-intensive task is not obvious. According mission is computationally intensive or I / O intensive, different scenarios using different methods, for compute-intensive tasks, multi-process dominated for I / O intensive, multithreaded dominant.
Compute-intensive tasks - multi-process solution:

# -*- coding:utf-8 -*-
from multiprocessing import Process
import os
import time

def work():
    result = 0
    for x in range(100000000):
        result *= x

if __name__ == "__main__":
    processes = []
    print("CPU: ", os.cpu_count())
    start = time.time()
    for i in range(4):
        p = Process(target=work)
        processes.append(p)
        p.start()
    for p in processes:
        p.join()
    end = time.time()
    print("计算密集型任务，多进程耗时 %s" % (end - start))

# output:
# CPU:  4
# 计算密集型任务，多进程耗时 9.485123872756958

Compute-intensive tasks - multithreaded scenarios:

# -*- coding:utf-8 -*-
from threading import Thread
import os, time

def work():
    res = 0
    for x in range(100000000):
        res *= x

if __name__ == "__main__":
    threads = []
    print("CPU: ",os.cpu_count())
    start = time.time()
    for i in range(4):
        thread = Thread(target=work)  # 多进程
        threads.append(thread)
        thread.start()
    for thread in threads:
        thread.join()
    end = time.time()
    print("计算密集型任务，多线程耗时 %s" % (end - start))

# output:
# CPU:  4
# 计算密集型任务，多线程耗时 18.434288501739502

IO-intensive tasks - multi-process solution:

# -*- coding:utf-8 -*-
from multiprocessing import Process
import os, time

def work():
    time.sleep(2)
    print("hello,Python----------------------------------------------------", file=open("tmp.txt", "w"))

if __name__ == "__main__":
    processes = []
    print("CPU: ", os.cpu_count())
    start = time.time()
    for i in range(400):
        p = Process(target=work)  # 多进程
        processes.append(p)
        p.start()
    for p in processes:
        p.join()
    stop = time.time()
    print("I/0密集型任务，多进程耗时 %s" % (stop - start))

# output:
# CPU:  4
# I/0密集型任务，多进程耗时 2.8894519805908203

IO-intensive tasks - multithreaded scenarios:

# -*- coding:utf-8 -*-
from threading import Thread
import os, time

def work():
    time.sleep(2)
    print("hello,Python----------------------------------------------------", file=open("tmp.txt", "w"))

if __name__ == "__main__":
    threads = []
    print("CPU: ", os.cpu_count())
    start = time.time()

    for x in range(400):
        thread = Thread(target=work)
        threads.append(thread)
        thread.start()
    for thread in threads:
        thread.join()
    end = time.time()
    print("IO密集型任务，多线程耗时 %s" % (end - start))

# output:
# CPU:  4
# IO密集型任务，多线程耗时 2.044438362121582

Second, create a thread

1, threading.Thread instantiated

threading.Thread constructor as follows:

def __init__(self, group=None, target=None, name=None,
             args=(), kwargs=None, *, daemon=None):

Creating threading.Thread instance, calling its start () method.

# -*- coding:utf-8 -*-
import time
import threading

def work_task(counter):
    print("%s %s" % (threading.current_thread().name, time.ctime(time.time())))
    n = counter;
    while n > 0:
        time.sleep(1)
        n -= 1

if __name__ == "__main__":
    print("main thread start：", time.strftime("%Y-%m-%d %H:%M:%S"))

    threads = []
    for x in range(10):
        thread = threading.Thread(target=work_task, args=(x, ))
        threads.append(thread)

    for thread in threads:
        thread.start()

    for thread in threads:
        thread.join()

    print("main thread end：", time.strftime("%Y-%m-%d %H:%M:%S"))

# output:
# main thread start： 2019-07-03 21:49:58
# Thread-1 Wed Jul  3 21:49:58 2019
# Thread-2 Wed Jul  3 21:49:58 2019
# Thread-3 Wed Jul  3 21:49:58 2019
# Thread-4 Wed Jul  3 21:49:58 2019
# Thread-5 Wed Jul  3 21:49:58 2019
# Thread-6 Wed Jul  3 21:49:58 2019
# Thread-7 Wed Jul  3 21:49:58 2019
# Thread-8 Wed Jul  3 21:49:58 2019
# Thread-9 Wed Jul  3 21:49:58 2019
# Thread-10 Wed Jul  3 21:49:58 2019
# main thread end： 2019-07-03 21:50:07

2, threading.Thread child thread

By creating a new subclass directly from threading.Thread class inheritance, in subclasses override the run () and init () method, instantiated call start () method to start a new thread, internal thread start function calls run () method.

# -*- coding:utf-8 -*-
import threading
import time

class WorkThread(threading.Thread):
    def __init__(self, thread_id, name):
        threading.Thread.__init__(self)
        self.thread_id = thread_id
        self.name = name

    def run(self):
        print("start thread: ", self.name)
        work(self.name, self.thread_id)
        print("end thread: ", self.name)

def work(thread_name, thread_id):
    print("%s %s %s" % (thread_name, thread_id, time.ctime(time.time())))
    i = 0;
    while i < 2:
        i += 1
        time.sleep(1)

if __name__ == '__main__':
    thread1 = WorkThread(1, "Thread1")
    thread2 = WorkThread(2, "Thread2")

    thread1.start()
    thread2.start()
    thread1.join()
    thread2.join()

    print("exit main thread")

# output:
# start thread:  Thread1
# Thread1 1 Tue Jul  2 20:39:42 2019
# start thread:  Thread2
# Thread2 2 Tue Jul  2 20:39:42 2019
# end thread:  end thread: Thread1
#  Thread2
# exit main thread

If the need to pass the function from the outside, may be passed as an argument an instance attribute child thread, the run call in an example method.
target = self.target
self.args = args

# -*- coding:utf-8 -*-
import threading
import time

class WorkThread(threading.Thread):
    def __init__(self, target, args):
        threading.Thread.__init__(self)
        self.target = target
        self.args = args

    def run(self):
        print("start thread: ", self.name)
        self.target(*self.args)
        print("end thread: ", self.name)

def work_task(counter):
    time.sleep(1)
    print("%s %s" % (threading.currentThread().name, time.ctime(time.time())))
    i = counter;
    while i > 0:
        i -= 1

if __name__ == '__main__':
    print("main thread start：", time.strftime("%Y-%m-%d %H:%M:%S"))

    threads = []
    for x in range(10):
        thread = threading.Thread(target=work_task, args=(x,))
        threads.append(thread)

    for thread in threads:
        thread.start()

    for thread in threads:
        thread.join()

    print("main thread end：", time.strftime("%Y-%m-%d %H:%M:%S"))

# output:
# main thread start： 2019-07-03 22:02:32
# Thread-1 Wed Jul  3 22:02:33 2019Thread-5 Wed Jul  3 22:02:33 2019
# Thread-2 Wed Jul  3 22:02:33 2019
# Thread-3 Wed Jul  3 22:02:33 2019
# Thread-4 Wed Jul  3 22:02:33 2019
#
# Thread-7 Wed Jul  3 22:02:33 2019Thread-6 Wed Jul  3 22:02:33 2019
# Thread-10 Wed Jul  3 22:02:33 2019
# Thread-8 Wed Jul  3 22:02:33 2019
#
# Thread-9 Wed Jul  3 22:02:33 2019
# main thread end： 2019-07-03 22:02:33

3, start and run

import threading
import time

def work_task(counter):
    n = counter
    while n > 0:
        n -= 1
        print("thread name： %s， id: %s" % (threading.currentThread().name, threading.currentThread().ident))

if __name__ == "__main__":
    print("main thread start")
    thread1 = threading.Thread(target=work_task, args=(5,))
    thread2 = threading.Thread(target=work_task, args=(5,))
    thread1.start()
    thread2.start()

    print("main thread end")

# output:
# main thread start
# thread name： Thread-1， id: 139926959064832thread name： Thread-2， id: 139926880384768main thread end
#
#
# thread name： Thread-1， id: 139926959064832
# thread name： Thread-2， id: 139926880384768thread name： Thread-1， id: 139926959064832
#
# thread name： Thread-1， id: 139926959064832
# thread name： Thread-2， id: 139926880384768thread name： Thread-1， id: 139926959064832
#
# thread name： Thread-2， id: 139926880384768
# thread name： Thread-2， id: 139926880384768

Using the start () method to start two new sub-thread alternately runs, each child process ID is different, start the thread name is name set when the defined thread object = "xxxx" value, if no name parameter value, Thread-x will print the name assigned by the system.

import threading
import time

def work_task(counter):
    n = counter
    while n > 0:
        n -= 1
        print("thread name： %s， id: %s" % (threading.currentThread().name, threading.currentThread().ident))

if __name__ == "__main__":
    print("main thread start")
    thread1 = threading.Thread(target=work_task, args=(5,))
    thread2 = threading.Thread(target=work_task, args=(5,))
    thread1.run()
    thread2.run()

    print("main thread end")

# output:
# main thread start
# thread name： MainThread， id: 140683421988672
# thread name： MainThread， id: 140683421988672
# thread name： MainThread， id: 140683421988672
# thread name： MainThread， id: 140683421988672
# thread name： MainThread， id: 140683421988672
# thread name： MainThread， id: 140683421988672
# thread name： MainThread， id: 140683421988672
# thread name： MainThread， id: 140683421988672
# thread name： MainThread， id: 140683421988672
# thread name： MainThread， id: 140683421988672
# main thread end

With the run () method to start a thread, the thread name is printed MainThread, that is the main thread. Both threads () method to start with a run, but to run thread1.run (), after running only run sequentially thread2.run (), two threads work in the main thread, not start a new thread, therefore, run () method is just an ordinary function call.

4, join method

When a process is started, it will generate a default main thread because the thread is the smallest unit of program execution flow when setting multiple threads, the main thread creates multiple child threads. In Python, the main thread After the implementation of its mandate, it will exit the default, then the child thread will continue to perform their task until their task is completed.

import threading
import time

def work_task(counter):
    n = counter
    while n > 0:
        n -= 1
        print("thread name： %s， id: %s" % (threading.currentThread().name, threading.currentThread().ident))

if __name__ == "__main__":
    print("main thread start")
    threads = []
    for x in range(5):
        thread = threading.Thread(target=work_task, args=(5,))
        threads.append(thread)
    for thread in threads:
        thread.start()

    print("main thread end")

# output:
# main thread start
# thread name： Thread-1， id: 140306042726144thread name： Thread-2， id: 140306034333440
# thread name： Thread-2， id: 140306034333440
# thread name： Thread-2， id: 140306034333440
# thread name： Thread-2， id: 140306034333440
# thread name： Thread-2， id: 140306034333440
# thread name： Thread-3， id: 140306025940736
#
# thread name： Thread-3， id: 140306025940736
# thread name： Thread-3， id: 140306025940736
# thread name： Thread-1， id: 140306042726144thread name： Thread-3， id: 140306025940736
# thread name： Thread-3， id: 140306025940736
#
# thread name： Thread-1， id: 140306042726144
# thread name： Thread-1， id: 140306042726144
# thread name： Thread-1， id: 140306042726144
# thread name： Thread-4， id: 140306034333440
# thread name： Thread-4， id: 140306034333440
# thread name： Thread-5， id: 140306042726144thread name： Thread-4， id: 140306034333440
# main thread endthread name： Thread-4， id: 140306034333440
# thread name： Thread-4， id: 140306034333440
#
# thread name： Thread-5， id: 140306042726144
#
# thread name： Thread-5， id: 140306042726144
# thread name： Thread-5， id: 140306042726144
# thread name： Thread-5， id: 140306042726144

Task when setDaemon (True) method, set up when the child thread as a daemon thread, the main thread once the execution is completed, the entire thread is terminated all executed, there may be a child thread has not fully implemented the end, it was forced to stop. Set setDaemon must be set before the promoter thread.

import threading
import time

def work_task(counter):
    n = counter
    time.sleep(1)
    while n > 0:
        n -= 1
        print("thread name： %s， id: %s" % (threading.currentThread().name, threading.currentThread().ident))

if __name__ == "__main__":
    print("main thread start")
    threads = []
    for x in range(5):
        thread = threading.Thread(target=work_task, args=(5,))
        threads.append(thread)
    for thread in threads:
        thread.setDaemon(True)
        thread.start()

    print("main thread end")

# output:
# main thread start
# main thread end

After the method used to join the main thread waits for executing the sub-line and returns the result, and then the main thread to perform the rest of the content, not the child thread completed execution, the main thread has been waiting state.

import threading
import time

def work_task(counter):
    n = counter
    time.sleep(1)
    while n > 0:
        n -= 1
        print("thread name： %s" % threading.currentThread().name)

if __name__ == "__main__":
    print("main thread start")
    threads = []
    for x in range(5):
        thread = threading.Thread(target=work_task, args=(5,))
        threads.append(thread)
    for thread in threads:
        thread.setDaemon(True)
        thread.start()
        thread.join()

    print("main thread end")

# output:
# main thread start
# thread name： Thread-1
# thread name： Thread-1
# thread name： Thread-1
# thread name： Thread-1
# thread name： Thread-1
# thread name： Thread-2
# thread name： Thread-2
# thread name： Thread-2
# thread name： Thread-2
# thread name： Thread-2
# thread name： Thread-3
# thread name： Thread-3
# thread name： Thread-3
# thread name： Thread-3
# thread name： Thread-3
# thread name： Thread-4
# thread name： Thread-4
# thread name： Thread-4
# thread name： Thread-4
# thread name： Thread-4
# thread name： Thread-5
# thread name： Thread-5
# thread name： Thread-5
# thread name： Thread-5
# thread name： Thread-5
# main thread end

join a timeout parameter, when set daemon thread, Cheng Duizi main thread waits timeout time, give each child thread a timeout time, let the child thread execution, time is up, no matter the task has not been completed, directly kill. If more than one child threads, all waiting time for each child thread timeout and accumulation.

import threading
import time

def work_task(counter):
    print("thread name： %s work task start" % threading.currentThread().name)
    n = counter
    time.sleep(4)
    while n > 0:
        n -= 1
    else:
        print("thread name： %s work task end" % threading.currentThread().name)

if __name__ == "__main__":
    print("main thread start")
    threads = []
    for x in range(5):
        thread = threading.Thread(target=work_task, args=(5,))
        threads.append(thread)

    for x in range(5):
        threads[x].setDaemon(True)
        threads[x].start()
        threads[x].join(1)

    print("main thread end")

# output:
# main thread start
# thread name： Thread-1 work task start
# thread name： Thread-2 work task start
# thread name： Thread-3 work task start
# thread name： Thread-4 work task start
# thread name： Thread-5 work task start
# thread name： Thread-1 work task end
# main thread end

When not set a daemon thread, the main thread will wait for a period of time and accumulate timeout, the time is up, the end of the main thread, but did not kill the child thread, the child thread can still continue until the end of all the child thread, the program exits .

import threading
import time

def work_task(counter):
    print("thread name： %s work task start" % threading.currentThread().name)
    n = counter
    time.sleep(4)
    while n > 0:
        n -= 1
    else:
        print("thread name： %s work task end" % threading.currentThread().name)

if __name__ == "__main__":
    print("main thread start")
    threads = []
    for x in range(5):
        thread = threading.Thread(target=work_task, args=(5,))
        threads.append(thread)

    for x in range(5):
        threads[x].start()
        threads[x].join(1)

    print("main thread end")

# output:
# main thread start
# thread name： Thread-1 work task start
# thread name： Thread-2 work task start
# thread name： Thread-3 work task start
# thread name： Thread-4 work task start
# thread name： Thread-5 work task start
# thread name： Thread-1 work task end
# main thread end
# thread name： Thread-2 work task end
# thread name： Thread-3 work task end
# thread name： Thread-4 work task end
# thread name： Thread-5 work task end

Third, thread synchronization

If multiple threads of a common data modification, unpredictable results may occur, in order to ensure the accuracy of the data, the need for multiple threads to synchronize.

1, mutex

Lock threading.Thread class lock and lock can be realized simply Rlock thread synchronization, and Rlock Lock Lock Lock both acquire method and release method, allows only one thread data operation needs to be placed between the acquire and release operation method .

# -*- coding:utf-8 -*-
import threading
import time

class WorkThread(threading.Thread):
    def __init__(self, thread_id, name):
        threading.Thread.__init__(self)
        self.thread_id = thread_id
        self.name = name

    def run(self):
        thread_locker.acquire()
        print("start thread: ", self.name)
        work(self.name, self.thread_id)
        print("end thread: ", self.name)
        thread_locker.release()

def work(thread_name, thread_id):
    print("%s %s %s" % (thread_name, thread_id, time.ctime(time.time())))
    i = 0;
    while i < 2:
        i += 1
        time.sleep(1)

thread_locker = threading.Lock()
threads = []

if __name__ == '__main__':
    thread1 = WorkThread(1, "Thread1")
    thread2 = WorkThread(2, "Thread2")

    thread1.start()
    thread2.start()

    threads.append(thread1)
    threads.append(thread2)

    for t in threads:
        t.join()

    print("exit main thread")

# output:
# start thread:  Thread1
# Thread1 1 Tue Jul  2 20:48:05 2019
# end thread:  Thread1
# start thread:  Thread2
# Thread2 2 Tue Jul  2 20:48:07 2019
# end thread:  Thread2
# exit main thread

2, the semaphore

Mutex while allowing only one thread access to shared data, while the semaphore while allowing a certain number of threads access shared data, such as bank counter has five windows, while five people are allowed to conduct business, the latter can only wait in front of someone after finishing business can enter the counter.

# -*- coding:utf-8 -*-
import threading
import time

semaphore = threading.BoundedSemaphore(5)
threads = []

def do_work(name):
    semaphore.acquire()
    time.sleep(2)
    print(f"{time.strftime('%Y-%m-%d %H:%M:%S')} {threading.currentThread().name} is carrying on business")
    semaphore.release()

if __name__ == '__main__':
    print("main thread start：", time.strftime("%Y-%m-%d %H:%M:%S"))

    for i in range(10):
        t = threading.Thread(target=do_work, args=(i,))
        threads.append(t)

    for thread in threads:
        thread.start()

    for thread in threads:
        thread.join()

    print("main thread end：", time.strftime("%Y-%m-%d %H:%M:%S"))

# output:
# main thread start： 2019-07-03 22:31:06
# 2019-07-03 22:31:08 Thread-1 is carrying on business
# 2019-07-03 22:31:08 Thread-3 is carrying on business
# 2019-07-03 22:31:08 Thread-2 is carrying on business
# 2019-07-03 22:31:08 Thread-4 is carrying on business
# 2019-07-03 22:31:08 Thread-5 is carrying on business
# 2019-07-03 22:31:10 Thread-6 is carrying on business
# 2019-07-03 22:31:10 Thread-7 is carrying on business
# 2019-07-03 22:31:10 Thread-9 is carrying on business
# 2019-07-03 22:31:10 Thread-8 is carrying on business
# 2019-07-03 22:31:10 Thread-10 is carrying on business
# main thread end： 2019-07-03 22:31:10

3, condition variables

Condition variable allows a thread after notification A stop, waiting for the other thread B, thread B satisfies a condition (notify) A thread continues to run. Thread first obtain a lock condition variable, if conditions are not met, the thread wait (wait) and release the lock condition variable; if the conditions are met thread of execution, may also notify other states to wait threads. Other threads in the wait state after receiving the notice will be re-judged conditions.

import threading
import time

class ThreadA(threading.Thread):
    def __init__(self, con, name):
        super(ThreadA, self).__init__()
        self.cond = con
        self.name = name

    def run(self):
        self.cond.acquire()
        print(self.name + ": What can I do for you?")
        self.cond.notify()
        self.cond.wait()
        print(self.name + ": Five yuan.")
        self.cond.notify()
        self.cond.wait()
        print(self.name + ": You are welcome.")
        self.cond.release()

class ThreadB(threading.Thread):
    def __init__(self, con, name):
        super(ThreadB, self).__init__()
        self.cond = con
        self.name = name

    def run(self):
        self.cond.acquire()
        time.sleep(1)
        print(self.name + ": A hot dog, please！")
        self.cond.notify()
        self.cond.wait()
        print(self.name + ": Thanks.")
        self.cond.notify()
        self.cond.release()

if __name__ == "__main__":
    cond = threading.Condition()
    thread1 = ThreadA(cond, "ThreadA")
    thread2 = ThreadB(cond, "ThreadB")
    thread1.start()
    thread2.start()

# output:
# ThreadA: What can I do for you?
# ThreadB: A hot dog, please！
# ThreadA: Five yuan.
# ThreadB: Thanks.
# ThreadA: You are welcome.

4 events

Events for inter-thread communication. A thread sends a signal, the other one or more threads wait, wait method calls the event object, the thread will block until the other thread set, only to be awakened.

import threading
import time

class ThreadA(threading.Thread):
    def __init__(self, _event, name):
        super(ThreadA, self).__init__()
        self.event = _event
        self.name = name

    def run(self):
        print(self.name + ": What can I do for you?")
        self.event.set()
        time.sleep(0.5)
        self.event.wait()
        print(self.name + ": Five yuan.")
        self.event.set()
        time.sleep(0.5)
        self.event.wait()
        self.event.clear()
        print(self.name + ": You are welcome!")

class ThreadB(threading.Thread):
    def __init__(self, _event, name):
        super(ThreadB, self).__init__()
        self.event = _event
        self.name = name

    def run(self):
        self.event.wait()
        self.event.clear()
        print(self.name + ": A hot dog, please!")
        self.event.set()
        time.sleep(0.5)
        self.event.wait()
        print(self.name + ": Thanks!")
        self.event.set()

if __name__ == "__main__":
    event = threading.Event()
    thread1 = ThreadA(event, "ThreadA")
    thread2 = ThreadB(event, "ThreadB")
    thread1.start()
    thread2.start()

# output:
# ThreadA: What can I do for you?
# ThreadB: A hot dog, please!
# ThreadA: Five yuan.
# ThreadB: Thanks!
# ThreadA: You are welcome!

5, thread priority queue

The Python Queue module provides synchronization queue based thread-safe, comprising a FIFO (first in first out) queue Queue, the LIFO (last in, first out) queue LifoQueue, priority queue PriorityQueue.
Queue, LifoQueue, PriorityQueue have achieved locking primitives can be used directly in multiple threads, queues can be used to achieve synchronization between threads.
Queue module common method:
Queue.qsize () returns the size of the queue
Queue.empty () if the queue is empty, returns True, otherwise return False
Queue.full () if the queue is full, return True, otherwise return False
Queue.get ([Block [, timeout]]) Acquisitions queue, timeout wait time
Queue.get_nowait () rather Queue.get (False)
Queue.put (Item) write queue, timeout wait time
Queue.put_nowait (item) rather Queue.put (Item, False)
Queue.task_done () after completion of a job, Queue.task_done () function to send a signal to the task has been completed queue
Queue.join () blocks until the queue is empty, then perform other operations

# -*- coding:utf-8 -*-
import threading
import time
import queue

exitFlag = 0

class WorkThread(threading.Thread):
    def __init__(self, id, name, q):
        threading.Thread.__init__(self)
        self.thread_id = id
        self.name = name
        self.queue = q

    def run(self):
        work(self.name, self.queue)

def work(thread_name, q):
    while not exitFlag:
        thread_locker.acquire()
        if not work_queue.empty():
            data = q.get()
            print("%s processing %s" % (thread_name, data))
            thread_locker.release()
        else:
            thread_locker.release()
        time.sleep(1)

thread_locker = threading.Lock()
thread_list = ["Thread1", "Thread2", "Thread3"]
work_queue = queue.Queue(10)
messages = ["one", "two", "three", "four", "five"]
threads = []
thread_id = 1

if __name__ == '__main__':
    # 创建新线程
    for name in thread_list:
        thread = WorkThread(thread_id, name, work_queue)
        thread.start()
        threads.append(thread)
        thread_id += 1

    # 填充队列
    thread_locker.acquire()
    for word in messages:
        work_queue.put(word)
    thread_locker.release()

    # 等待队列清空
    while not work_queue.empty():
        pass

    # 通知线程是时候退出
    exitFlag = 1

    # 等待所有线程完成
    for t in threads:
        t.join()
    print("exit main thread")

# output:
# Thread1 processing one
# Thread3 processing two
# Thread2 processing three
# Thread1 processing four
# Thread3 processing five
# exit main thread

6, thread deadlock

Deadlock is a phenomenon of two or more processes or threads in the implementation process, a result of competition for resources caused by waiting for each other. When multiple resources shared among threads, if possession were part of the resources and at the same time waiting for each other's resources, it will result in a deadlock. A database, for example, need to thread the thread operation result B is operated, the results need to thread B A thread operates, when A, B are not operated during the threads outcome, the case A, B in the thread will always waiting for the end of the other state.

import time
import threading

class Account:
    def __init__(self, _id, balance, lock):
        self.id = _id
        self.balance = balance
        self.lock = lock

    def withdraw(self, amount):
        self.balance -= amount

    def deposit(self, amount):
        self.balance += amount

def transfer(_from, to, amount):
    if _from.lock.acquire():
        _from.withdraw(amount)
        time.sleep(1)
        print('wait for lock...')
        if to.lock.acquire():
            to.deposit(amount)
            to.lock.release()
        _from.lock.release()
    print('finish...')

if __name__ == "__main__":
    a = Account('a', 1000, threading.Lock())
    b = Account('b', 1000, threading.Lock())
    threading.Thread(target=transfer, args=(a, b, 100)).start()
    threading.Thread(target=transfer, args=(b, a, 200)).start()

One solution to the deadlock problem is to assign a unique id for each program in the lock and only allows the use of multiple rules in ascending lock.

Fourth, the thread pool

1, the thread pool Profile

Thread pool is to create a large number of idle threads at system startup, as long as the program will be submitted to a task function to the thread pool, thread pool will be free to start a thread to execute it. When the End Task function is executed, the thread does not die, but returned again to the thread pool becomes idle, waiting for the next task function.
Using the thread pool number of concurrent threads in the system can be effectively controlled. When the system contains a large number of concurrent threads, it will lead to a sharp decline in system performance, and even lead to the collapse of the Python interpreter, while the maximum number of threads in the thread pool parameters to control the number of concurrent threads the system does not exceed this number.
concurrent.futures module Executor is an abstract base class thread pool, Executor provides two sub-categories, namely ThreadPoolExecutor and ProcessPoolExecutor, which ThreadPoolExecutor used to create a thread pool, ProcessPoolExecutor process used to create the pool.
Exectuor provides a common interface follows:
submit(fn, *args, **kwargs): function fn will be submitted to the thread pool. on behalf of args parameter passed to the function fn, it is a tuple type, kwargs representatives in the form of keyword arguments passed to fn function parameters, is a dictionary type. submit method returns a Future object
map(func, *iterables, timeout=None, chunksize=1): map function will start multiple threads to execute asynchronously map process immediately iterables.
shutdown (wait = True): close the thread pool.
Future provides the following method:
the Cancel (): Cancel Future behalf of threaded tasks. If the task is being performed, irrevocable, it would return False; otherwise, the program will cancel the task and return True.
cancelled (): Returns the Future represents the thread whether the task was successfully canceled.
running (): If the Future represents the thread is executing the task, can not be canceled, it returns True.
done (): If Funture representatives thread task is completed successfully canceled or executed, returns True.
result (timeout = None): Gets Future-threaded tasks on behalf of the final returns the result. If the representative of the Future threaded task has not been completed, result method will block the current thread, where the timeout parameter specifies how many seconds the most obstructive.
exception (timeout = None): Gets Future tasks on behalf of the thread was thrown. If the task is completed without exception, then the method returns None.
add_done_callback (fn): Future is threaded tasks on behalf of a registered "callback function", when the threaded tasks successfully completed, the program will automatically trigger fn function.
After using a thread pool runs out, you should call the thread pool shutdown () method, shutdown sequence will disable the thread pool starts. After calling shutdown () method thread pool is no longer receiving new tasks, but will perform all the tasks that are submitted complete. When all tasks are executed to complete the thread pool, all threads in the thread pool of death.

2, the thread pool

ThreadPoolExecutor (max_works), if not explicitly specified max_works, the default thread pool creates a CPU的数目*5number of threads.

# -*- coding:utf-8 -*-
from concurrent.futures import ThreadPoolExecutor
import threading
import time
import os
import string

class WorkThread(threading.Thread):
    def __init__(self):
        threading.Thread.__init__(self)

    def run(self):
        print('Process[%s]:%s start and run task' % (os.getpid(), threading.currentThread().getName()))
        time.sleep(2)
        return "Process[{}]:{} end".format(os.getpid(), threading.currentThread().getName())

def work_task(thread_name):
    print('Process[%s]:%s start and run task' % (os.getpid(), threading.currentThread().getName()))
    time.sleep(5)
    return "Process[{}]:{} end".format(os.getpid(), threading.currentThread().getName())

def get_call_back(future):
    print(future.result())

if __name__ == '__main__':
    print('main thread start')

    # create thread pool
    thread_pool = ThreadPoolExecutor(5)
    futures = []
    for i in range(5):
        thread = WorkThread()
        future = thread_pool.submit(thread.run)
        futures.append(future)

    for i in range(5):
        future = thread_pool.submit(work_task, i)
        futures.append(future)

    for future in futures:
        future.add_done_callback(get_call_back)

    # thread_pool.map(work_task, (2, 3, 4))
    thread_pool.shutdown()

# output:
# main thread start
# Process[718]:ThreadPoolExecutor-0_0 start and run task
# Process[718]:ThreadPoolExecutor-0_1 start and run task
# Process[718]:ThreadPoolExecutor-0_2 start and run task
# Process[718]:ThreadPoolExecutor-0_3 start and run task
# Process[718]:ThreadPoolExecutor-0_4 start and run task
# Process[718]:ThreadPoolExecutor-0_3 end
# Process[718]:ThreadPoolExecutor-0_3 start and run task
# Process[718]:ThreadPoolExecutor-0_1 end
# Process[718]:ThreadPoolExecutor-0_1 start and run task
# Process[718]:ThreadPoolExecutor-0_2 end
# Process[718]:ThreadPoolExecutor-0_2 start and run task
# Process[718]:ThreadPoolExecutor-0_0 end
# Process[718]:ThreadPoolExecutor-0_0 start and run task
# Process[718]:ThreadPoolExecutor-0_4 end
# Process[718]:ThreadPoolExecutor-0_4 start and run task
# Process[718]:ThreadPoolExecutor-0_2 end
# Process[718]:ThreadPoolExecutor-0_3 end
# Process[718]:ThreadPoolExecutor-0_1 end
# Process[718]:ThreadPoolExecutor-0_4 end
# Process[718]:ThreadPoolExecutor-0_0 end

3, process pool

ProcessPoolExecutor (max_works), if not explicitly specified max_works, the default process pool will create CPU的数目*5a number of processes.
Process pool synchronization scheme:

from concurrent.futures import ProcessPoolExecutor
import os
import time
import random

def work_task(n):
    print('Process[%s] is running' % os.getpid())
    time.sleep(random.randint(1,3))
    return n**2

if __name__ == '__main__':
    start = time.time()
    pool = ProcessPoolExecutor()
    for i in range(5):
        obj = pool.submit(work_task, i).result()
    pool.shutdown()
    print('='*30)
    print("time: ", time.time() - start)

# output;
# Process[7372] is running
# Process[7373] is running
# Process[7374] is running
# Process[7375] is running
# Process[7372] is running
# ==============================
# time:  10.023026466369629

Process pool asynchronous scheme:

from concurrent.futures import ProcessPoolExecutor
import os
import time
import random

def work_task(n):
    print('Process[%s] is running' % os.getpid())
    time.sleep(random.randint(1, 3))
    return n**2

if __name__ == '__main__':
    start = time.time()
    pool = ProcessPoolExecutor()
    objs = []
    for i in range(5):
        obj = pool.submit(work_task, i)
        objs.append(obj)
    pool.shutdown()
    print('='*30)
    print([obj.result() for obj in objs])
    print("time: ", time.time() - start)

# output;
# Process[8268] is running
# Process[8269] is running
# Process[8270] is running
# Process[8271] is running
# Process[8270] is running
# ==============================
# [0, 1, 4, 9, 16]
# time:  2.0124566555023193

Fifth, the producer-consumer model

import threading
from queue import Queue
from urllib.request import urlopen

ips = ["www.baidu.com",
       "www.taobao.com",
       "www.huawei.com",
       "www.alibaba.com",
       "www.meituan.com",
       "www.xiaomi.com"]
ports = [80, 443]

class Producer(threading.Thread):
    def __init__(self, _queue):
        super(Producer, self).__init__()
        self.queue = _queue

    def run(self):
        urls = ["http://%s:%s" % (ip, port) for ip in ips for port in ports]
        for url in urls:
            self.queue.put(url)

class Consumer(threading.Thread):
    def __init__(self, _queue):
        super(Consumer, self).__init__()
        self.queue = _queue

    def run(self):
        try:
            url = self.queue.get()
            urlopen(url)
        except Exception as e:
            print("%s is unknown url" % url)
        else:
            print("%s is ok" % url)

if __name__ == "__main__":
    # 实例化一个队列
    queue = Queue()

    for i in range(2):
        producer = Producer(queue)
        producer.start()

    for i in range(30):
        consumer = Consumer(queue)
        consumer.start()

# output:
# http://www.taobao.com:443 is unknown url
# http://www.huawei.com:443 is unknown url
# http://www.huawei.com:443 is unknown urlhttp://www.taobao.com:443 is unknown url
#
# http://www.baidu.com:80 is ok
# http://www.baidu.com:443 is unknown url
# http://www.xiaomi.com:443 is unknown url
# http://www.baidu.com:80 is ok
# http://www.baidu.com:443 is unknown url
# http://www.xiaomi.com:443 is unknown url
# http://www.alibaba.com:443 is unknown url
# http://www.alibaba.com:443 is unknown url
# http://www.meituan.com:443 is unknown urlhttp://www.meituan.com:443 is unknown url
#
# http://www.huawei.com:80 is ok
# http://www.huawei.com:80 is ok
# http://www.xiaomi.com:80 is ok
# http://www.xiaomi.com:80 is ok
# http://www.taobao.com:80 is ok
# http://www.meituan.com:80 is ok
# http://www.meituan.com:80 is ok
# http://www.taobao.com:80 is ok
# http://www.alibaba.com:80 is ok
# http://www.alibaba.com:80 is ok