1. Brief introduction

Python 标准库It is very large and provides a wide range of components. The official reference link is https://docs.python.org/zh-cn/3.9/library/index.html . This library consists of several built-in modules (written in C) that Python programmers must rely on for system-level functionality such as file I/O, in addition to a large number of modules written in Python, 提供了日常编程中许多问题的标准解决方案.

We usually develop algorithms on Python, because Python programming is convenient and easy to quickly verify algorithms. After verifying that the algorithm is correct, if there is a higher requirement for operating efficiency, the computationally intensive modules will be executed using multi-threads/multi-processes to achieve the effect of accelerating code operating efficiency. So, here comes the Python Standard Library - Concurrent Execution. https://docs.python.org/zh-cn/3.9/library/concurrency.html

insert image description here

2. Program example

The following programming examples are related actual combat codes carried out by individuals based on the Python standard library - concurrent execution .

2.1 threading programming example

import threading
import queue

g_result = queue.Queue()


# calculate sum from start to end
def do_threading(start, end, thread_id):

    sum_value = 0

    for i in range(start, end):
        sum_value += i

    g_result.put((thread_id, sum_value))


thread_num = 16
thread_pool = []

for task_id in range(thread_num):
    thread_task = threading.Thread(target=do_threading,
                                   args=(task_id * 100, (task_id + 1) * 100, task_id))

    thread_pool.append(thread_task)

for thread_task in thread_pool:
    thread_task.start()

for thread_task in thread_pool:
    thread_task.join()

for task_id in range(thread_num):
    result = g_result.get()
    print("threading no.", result[0], ", result is", result[1])

2.2 Multiprocessing programming example

import multiprocessing


# calculate sum from start to end
def do_process(start, end):

    sum_value = 0

    for i in range(start, end):
        sum_value += i

    return sum_value


cpu_num = multiprocessing.cpu_count()
pool = multiprocessing.Pool(processes=cpu_num)
results = []
task_num = 16

for task_id in range(task_num):
    results.append(pool.apply_async(do_process, (task_id * 100, (task_id + 1) * 100)))

pool.close()
pool.join()

for task_id in range(task_num):
    print("process no.", task_id, ", result is", results[task_id].get())

2.3 concurrent.futures programming example

import time
import numpy as np
from concurrent.futures import ThreadPoolExecutor


def read_files_thread(files_seq, threads_num=4):
    thread_pool = ThreadPoolExecutor(max_workers=threads_num)

    def read_file(file_path):
        return np.fromfile(file_path, dtype=np.float32)

    files_val_seq = [i for i in thread_pool.map(read_file, files_seq)]
    thread_pool.shutdown()
    return files_val_seq


def read_files(files_seq):

    def read_file(file_path):
        return np.fromfile(file_path, dtype=np.float32)

    files_val_seq = [read_file(i) for i in files_seq]
    return files_val_seq


file_sequence = ['1.bin', '2.bin', '3.bin', '4.bin', '5.bin', '6.bin']


start_time = time.time()

method_1 = read_files_thread(file_sequence, 8)

print("method_1 time consume ", time.time() - start_time)

start_time = time.time()

method_2 = read_files(file_sequence)

print("method_2 time consume ", time.time() - start_time)

Python Standard Library - Concurrent Execution