python: concurrent programming (eighteen)

foreword

This article will discuss with you the practical application of python concurrent programming, and will share some cases that I actually use, or some typical cases. The case used in this article is the case I actually used (Part 2) , which is rewritten into concurrent programming based on the previously inefficient code. Let's take a look at the transformation process, so that we will have a clear understanding of the high efficiency of concurrent programming and learn some knowledge during the transformation process.

This article is the eighteenth article of python concurrent programming. The address of the previous article is as follows:

python: concurrent programming (seventeen)_Lion King's Blog-CSDN Blog

The address of the next article is as follows:

Python: Concurrent Programming (19)_Lion King's Blog-CSDN Blog

1. Implementation plan: Is it possible that the image_gray function can also be optimized

Through the first two chapters, we optimized the assert_run function, and optimized some time through the process. Of course, the most time-consuming part of the image_gray function has not been optimized. This chapter will discuss the optimization of this part together.

1. Can image_gray be optimized concurrently?

Let's first look at the two for loops used by this function, and first use other for loops to simulate the time it uses, that is, the following simple function:

import time


st=time.time()

for i in range(1000):
    for j in range(2000):
        print(i*j)

et=time.time()
print(et-st)

The above function takes 16.6 seconds without concurrency.

2. Multi-threaded transformation of 2 for loops

(1) The single thread takes about 39 seconds

import concurrent.futures
import time

def process(i, j):
    result = i * j
    return result

st=time.time()
with concurrent.futures.ThreadPoolExecutor(max_workers=1) as executor:
    futures = []
    for i in range(1000):
        for j in range(2000):
            future = executor.submit(process, i, j)
            futures.append(future)

    # 获取结果
    for future in concurrent.futures.as_completed(futures):
        result = future.result()
        print(result)
et=time.time()
print(et-st)

(2) 10 threads take about 60 seconds

3. Multi-process transformation of 2 for loops

(1) A single process takes about 502 seconds

import concurrent.futures
import time

def process(i, j):
    result = i * j
    return result

if __name__ == "__main__":
    st=time.time()
    with concurrent.futures.ProcessPoolExecutor(max_workers=1) as executor:
        futures = []
        for i in range(1000):
            for j in range(2000):
                future = executor.submit(process, i, j)
                futures.append(future)

        # 获取结果
        for future in concurrent.futures.as_completed(futures):
            result = future.result()
            print(result)
    et=time.time()
    print(et-st)

 (2) 10 processes take about 520 seconds

2. A little thought from two for loops

1. Why do the two for loops only take 16 seconds without adding threads. It takes 60 seconds to add a thread?

If threads are added in two nested for loops and in each iteration a new thread is started to perform computation tasks, this can lead to increased execution time for the following reasons:

(1) Thread creation and destruction overhead: Creating and destroying threads will bring a certain amount of overhead. If new threads are created on each iteration, the overhead of thread creation and destruction will become a performance bottleneck, increasing overall execution time.

(2) Thread switching overhead: Switching between threads also requires a certain amount of overhead. If there are too many thread switches, such as switching threads on every iteration, then the overhead of thread switching can accumulate and lead to increased execution time.

(4) GIL (Global Interpreter Lock) restriction: If you are using the CPython interpreter, and each thread is performing CPU-intensive tasks (such as calculations), then due to the existence of the GIL, there can only be one thread at a time Execute Python bytecode and other threads will be blocked. This means that multiple threads cannot truly perform computing tasks in parallel, but will bring additional thread switching overhead, thereby increasing the overall execution time.

To sum up, if the purpose of adding threads is to speed up computing tasks, but in CPU-intensive scenarios, the execution time may increase due to the overhead of thread creation and destruction, thread switching overhead, and GIL limitations. In such cases, consider using other concurrency models, such as multiprocessing or asynchronous programming, for better utilization of computing resources and improved performance.

2. Why do the two for loops only take 16 seconds without adding a process. Adding a process takes 500 seconds?

If processes are added in two nested for loops, and a new process is started in each iteration to perform computational tasks, this can lead to increased execution time for the following reasons:

(1) Process creation and destruction overhead: Creating and destroying processes will bring a large overhead. If new processes are created in each iteration, the process creation and destruction overhead will become a performance bottleneck, thereby increasing the overall execution time.

(2) Inter-process communication overhead: In multi-process programming, communication and synchronization operations are required between processes. This includes data transfers, shared memory, etc., and these operations introduce certain overhead. Involving inter-process communication in each iteration adds additional overhead resulting in increased execution time.

(3) Context switching overhead: In multi-process programming, since the operating system needs to perform context switching between processes, this will also introduce certain overhead. A process switch is involved in each iteration, which increases the number of context switches, thereby increasing execution time.

(4) Global Interpreter Lock (GIL) restriction: If you are using the CPython interpreter, and each process is performing CPU-intensive tasks (such as calculations), then due to the existence of the GIL, there can only be one process at a time Execute Python bytecode and other processes will be blocked. This means that multiple processes cannot truly perform computing tasks in parallel, but will bring additional process switching and communication overhead, thereby increasing the overall execution time.

To sum up, if the purpose of adding a process is to speed up computing tasks, but in CPU-intensive scenarios, due to process creation and destruction overhead, inter-process communication overhead, context switching overhead, and GIL limitations, it may cause execution time Increase. In such cases, consider using other concurrency models, such as multithreading or asynchronous programming, for better utilization of computing resources and improved performance.

3. Two for loops, 10 processes take 500 seconds, 10 threads take 60 seconds, why are processes so much slower than threads?

Performance differences between processes and threads can be affected by several factors:

(1) Resource consumption: The creation and destruction of processes is usually more expensive than threads, because independent memory space and system resources are required between processes. Creating and destroying processes requires more overhead, including memory allocation, file descriptor copying and cleanup, etc. Therefore, creating and destroying multiple processes in each iteration may result in high resource consumption and thus increase execution time.

(2) Context switching overhead: Context switching between processes is more expensive than context switching between threads. Context switching is the process by which the operating system switches from one execution unit (process or thread) to another. In the case of multiple processes, due to the independent memory space and resources between processes, context switching requires more overhead, including saving and restoring the state of the process, etc. Therefore, switching between multiple processes in each iteration may result in high context switching overhead and thus increase execution time.

(3) Global Interpreter Lock (GIL): If you are using the CPython interpreter and each thread is performing CPU-intensive tasks (such as calculations), then due to the existence of the GIL, only one thread can execute at the same time Python bytecode, other threads will be blocked. However, in the case of multithreading, the overhead of thread switching is usually much smaller than that of process switching. Therefore, when it comes to CPU-intensive tasks, threads can make better use of computing resources, while processes may not be able to fully utilize multi-core processors due to GIL constraints, resulting in lower performance.

In summary, the reason why processes are slower than threads may be the overhead of creating and destroying processes, the overhead of context switching between processes, and the impact of the Global Interpreter Lock (GIL). In certain scenarios, threads may be better suited for CPU-intensive tasks, while processes are better suited for I/O-intensive tasks or parallel processing. However, for different applications and environments, performance differences may vary, so choosing an appropriate concurrency model needs to consider specific requirements and conditions.

3. Use other methods to optimize the image_gray function

1. Original function optimization

The code of the original function is used to calculate the average gray value of a picture. It opens the picture, traverses each pixel and calculates its gray value, and finally finds the average gray value and returns it.

The logic in the code is correct, but in order to improve the calculation efficiency, some optimization techniques can be considered, such as reducing the number of unnecessary loops, using vectorized calculations, etc. Let's modify it as follows:

def image_gray(img):
    # 打开图片
    img = Image.open(img)

    # 将图像转换为灰度图
    img_gray = img.convert("L")

    # 获取图像的像素数据
    pixels = list(img_gray.getdata())

    # 计算平均灰度值
    total_gray = sum(pixels)
    avg_gray = total_gray / len(pixels)

    return avg_gray

In the optimized code, the following improvements are mainly made:

(1) Use the image convertmethod to convert the image to a grayscale image, avoiding the mode of judging pixels in the loop.

(2) Use getdatathe method to get the pixel data of the image, and return a list containing all pixel values.

(3) Use sumthe function to calculate the sum of pixel values ​​without explicitly performing loop accumulation.

(4) Directly use the length of the list as the total number of pixels, avoiding counting in the loop.

These optimizations can improve computational efficiency and performance. Note that the specific optimization method also depends on the image size and processing requirements, and can be adjusted and improved according to the actual situation.

2. Optimization effect

In the case of a single thread, it only took 4 seconds to process 40 images. In the case of 10 processes, it only takes less than 1 second, so far, the optimization of this project is over.

import os
from PIL import Image
import time
import concurrent.futures

def calculate_runtime(func):
    def wrapper(*args, **kwargs):
        start_time = time.time()
        result = func(*args, **kwargs)
        end_time = time.time()
        runtime = end_time - start_time
        print(f"程序运行时间：{runtime}秒")
        return result
    return wrapper

# @calculate_runtime
# def image_gray(img):
#     # 打开图片
#     img = Image.open(img)
#
#     # 计算平均灰度值
#     gray_sum = 0
#     count = 0
#     for x in range(img.width):
#         for y in range(img.height):
#             if img.mode == "RGB":
#                 r, g, b = img.getpixel((x, y))
#                 gray_sum += (r + g + b) / 3
#             elif img.mode == "L":
#                 gray_value = img.getpixel((x, y))
#                 gray_sum += gray_value
#             count += 1
#
#     avg_gray = gray_sum / count
#     return avg_gray

@calculate_runtime
def image_gray(img):
    # 打开图片
    img = Image.open(img)

    # 将图像转换为灰度图
    img_gray = img.convert("L")

    # 获取图像的像素数据
    pixels = list(img_gray.getdata())

    # 计算平均灰度值
    total_gray = sum(pixels)
    avg_gray = total_gray / len(pixels)

    # print(avg_gray)

    return avg_gray



@calculate_runtime
def find_image(folder_path):
    # 定义一个列表存储图片路径
    images = []
    # 遍历文件夹下的所有文件
    for root, dirs, files in os.walk(folder_path):
        for file in files:
            file_path = os.path.join(root, file)
            # 处理每个文件,将其添加到列表中
            images.append(file_path)
    return images[0:40]

def process_image(img):
    gray = image_gray(img)
    # 灰度值小于50，将认为是黑图
    if gray < 50:
        print(img, "：", gray)

@calculate_runtime
def assert_run(folder_path):
    images = find_image(folder_path)

    with concurrent.futures.ProcessPoolExecutor(max_workers=1) as executor:
        # 提交任务并发执行
        futures = [executor.submit(process_image, img) for img in images]

        # 等待所有任务完成
        concurrent.futures.wait(futures)


if __name__ == "__main__":
    folder_path = r'C:\Users\yeqinfang\Desktop\临时文件\文件存储'
    assert_run(folder_path)