python: concurrent programming (seven)

foreword

This article will discuss with you python's multi-threaded concurrent programming (Part 1) , use the built-in basic library threadingto achieve concurrency, and first use this module simply through the official. Lay a good foundation first, so that you can have a basic usage and cognition, and we will use it in detail in the follow-up articles.

This article is the seventh article of python concurrent programming. The address of the previous article is as follows:

Python: Concurrent Programming (6)_Lion King's Blog-CSDN Blog

The address of the next article is as follows:

python: concurrent programming (eight)_Lion King's Blog-CSDN Blog

1. Quick start 

Official documentation: threading --- Thread-based parallelism — Python 3.11.4 documentation

1. Thread local data

threadingThe module provides the function of thread-local data, which allows each thread to have its own independent data space, and the data between different threads does not interfere with each other.

A thread-local data object can be created using threading.local()a function. Each thread gets and sets its own independent data by accessing this object. Each thread's operation on this object is for its own data space and will not affect other threads.

Here is a simple example:

import threading

# 创建线程本地数据对象
local_data = threading.local()

def set_data(value):
    # 设置线程本地数据
    local_data.value = value

def get_data():
    # 获取线程本地数据
    return local_data.value

def worker():
    # 在各个线程中设置和获取线程本地数据
    set_data(threading.current_thread().name)
    print(get_data())

# 创建多个线程并启动
threads = []
for i in range(5):
    t = threading.Thread(target=worker)
    threads.append(t)
    t.start()

# 等待所有线程执行完成
for t in threads:
    t.join()

When each thread executes workerthe function, it will set the name of the current thread as data into the thread local data object, and then obtain and print it out. Since each thread has its own independent data space, the data obtained by each thread is set by itself without interfering with each other.

2. Thread object

from threading import Thread

# 创建一个新的线程，并指定目标函数为print，参数为1
t = Thread(target=print, args=[1])

# 启动新线程，开始执行目标函数
t.start()

start() can be run()replaced by a method. However, it should be noted that calling run()a method does not start a new thread to execute the target function, but directly calls the target function in the current thread. So, to start a new thread and execute the target function, start()a method should be used, not run()a method.

In Python, threadingmodules provide Threadclasses to create and manipulate thread objects. Thread objects are used to execute concurrent code blocks, so that multiple code blocks can be executed at the same time.

Here is a simple example:

import threading

# 线程执行的目标函数
def worker():
    print("Thread executing")

# 创建线程对象
t = threading.Thread(target=worker)

# 启动线程
t.start()

# 等待线程执行完成
t.join()

print("Thread finished")

The above code creates a thread object tand specifies the target function workeras the content to be executed by the thread. The thread is then start()started by calling a method, and the thread starts executing the code block in the target function. In the example, the target function simply prints a message.

The use join()method allows the main thread to wait for the child thread to complete execution. This ensures that the main thread continues execution after the child thread completes. Finally, the main thread prints a message to indicate the end of thread execution.

Through Threadthe class, you can create multiple thread objects to perform concurrent tasks as needed. Each thread object has its own execution context, including the state of the thread, code pointer, etc. You can use the methods of the thread object to control the behavior of the thread, such as starting, pausing, terminating, and so on.

It should be noted that the thread object does not directly call the target function, but start()starts the thread through a method, and the Python interpreter calls the target function at an appropriate time. This ensures correct startup and execution of threads.

In addition, the thread object also provides some other methods and properties, such as is_alive()used to check whether the thread is active, namethe property is used to obtain the name of the thread, and so on. Through these methods and attributes, the behavior of threads can be more flexibly controlled and managed.

3. Lock object

In threadinga module, lock objects are used to control access to shared resources among multiple threads. By locking objects, it can be ensured that only one thread can access shared resources at any time, thereby avoiding problems of race conditions and data inconsistencies.

In Python, you can use Lockclasses to create lock objects. Here is a simple example:

import threading

# 创建锁对象
lock = threading.Lock()

# 共享资源
shared_resource = 0

# 线程函数
def worker():
    global shared_resource
    # 获取锁
    lock.acquire()
    try:
        # 访问共享资源
        shared_resource += 1
    finally:
        # 释放锁
        lock.release()

# 创建多个线程并启动
threads = []
for _ in range(5):
    t = threading.Thread(target=worker)
    t.start()
    threads.append(t)

# 等待所有线程执行完成
for t in threads:
    t.join()

print("Shared resource:", shared_resource)

In the example, we create a lock object lockto control shared_resourceaccess to this shared variable. During the execution of each thread, acquire()the lock is acquired by calling the method, so that only one thread can successfully acquire the lock, and other threads will be blocked. Then try...finallyaccess the shared resource in blocks, and at the end use release()a method to release the lock so other threads can continue accessing.

By using lock objects, we ensure safe access to shared resources and avoid data inconsistency problems caused by multiple threads modifying shared resources at the same time. Only the thread that has acquired the lock can access the shared resource, and other threads will be blocked during the waiting period. This mechanism ensures synchronization and mutual exclusion between threads.

It should be noted that in order to avoid deadlocks, we should follow certain lock usage rules, such as avoid calling blocking operations while holding locks, and avoid nested use of locks. Reasonable use of lock objects can ensure data consistency and concurrency safety among threads.

4. Recursive lock object

In threadingthe module, in addition to ordinary lock objects Lock, recursive lock objects are also provided RLock, also known as reentrant locks.

Recursive locks are similar to ordinary locks and are used to control access to shared resources by multiple threads. But unlike ordinary locks, recursive locks allow the same thread to acquire the same lock multiple times without deadlock.

A recursive lock maintains a lock layer when the same thread acquires the lock multiple times. Each time a thread successfully acquires a lock, the number of layers increases by 1; each time a lock is released, the number of layers decreases by 1. Only when the lock level is 0, other threads can acquire the lock.

Here is a simple example:

import threading

# 创建递归锁对象
lock = threading.RLock()

# 共享资源
shared_resource = 0

# 递归函数
def recursive_function(count):
    global shared_resource

    # 获取递归锁
    lock.acquire()
    try:
        if count > 0:
            shared_resource += 1
            recursive_function(count - 1)
    finally:
        # 释放递归锁
        lock.release()

# 创建并启动线程
t = threading.Thread(target=recursive_function, args=(5,))
t.start()
t.join()

print("Shared resource:", shared_resource)

In the example, we create a recursive lock object lockand then define a recursive function recursive_function. In a recursive function, acquire()the recursive lock is acquired by calling a method, and calls itself multiple times during the recursion. A value that a recursive function increments each time it is called shared_resourceuntil the end of the recursion.

Due to the use of recursive locks, the same thread can acquire the lock multiple times without deadlock. At each recursive call, the thread successfully acquires the lock and releases the lock after the recursion ends. This ensures thread-safe access to shared resources.

It should be noted that the use of recursive locks needs to be cautious to ensure that the locks are released accordingly after each lock is acquired, so as to avoid deadlocks caused by unbalanced layers. Recursive locks are mainly used to deal with complex nested lock scenarios, ensuring that threads can correctly acquire and release locks in multi-layer nesting.

5. Condition object

In threadingmodules, the condition object ( Condition) is a high-level facility for inter-thread communication and synchronization. A condition object provides a mechanism that allows one or more threads to wait for a certain condition to occur, and when the condition is met, the thread can continue to execute.

The condition object is implemented based on the lock object ( Lock) and used in conjunction with the lock object. It provides methods such as wait(), notify()and notify_all(), for waiting and notification between threads.

The following is an example of basic usage of the condition object:

import threading

# 创建条件对象和关联的锁对象
condition = threading.Condition()

# 共享资源
shared_resource = []

# 生产者函数
def producer():
    global shared_resource
    with condition:
        # 检查条件是否满足，如果满足则等待
        while shared_resource:
            condition.wait()

        # 生产一个新的元素
        new_item = len(shared_resource) + 1
        shared_resource.append(new_item)

        # 通知等待的消费者线程
        condition.notify()

# 消费者函数
def consumer():
    global shared_resource
    with condition:
        # 检查条件是否满足，如果不满足则等待
        while not shared_resource:
            condition.wait()

        # 消费最后一个元素
        consumed_item = shared_resource.pop()

        # 通知等待的生产者线程
        condition.notify()

        # 打印消费的元素
        print("Consumed item:", consumed_item)

# 创建并启动生产者和消费者线程
producer_thread = threading.Thread(target=producer)
consumer_thread = threading.Thread(target=consumer)
producer_thread.start()
consumer_thread.start()
producer_thread.join()
consumer_thread.join()

The above code implements a simple producer-consumer model, using threading.Conditionobjects for synchronization and communication between threads.

The code condition = threading.Condition()creates a condition object and associated lock object.

Both the producer function producer()and the consumer function consumer()use with condition:the statement to acquire the lock on the condition object and execute the thread's operation within the block.

In the producer function, first check shared_resourcewhether the shared resource is empty, if it is empty, the call condition.wait()enters a waiting state until a consumer thread notifies the producer thread to continue execution. Then, produce a new element, add it to the shared resource, and finally call condition.notify()the notification waiting consumer thread.

In the consumer function, first check whether the shared resource is empty. If it is empty, the call condition.wait()enters a waiting state until a producer thread notifies the consumer thread to continue execution. Then, take the last element from the shared resource, call condition.notify()the producer thread that notifies the waiting, and print the consumed element.

Finally, create and start the producer and consumer threads, start()start the threads by calling the method, and use join()the method to wait for the thread execution to complete.

In this way, the producer and consumer threads can synchronize and communicate through the condition object and the associated lock object to ensure that the thread enters the waiting state when the shared resource does not meet a specific condition, and wakes up the waiting thread to continue execution when the condition is met.