[100 days proficient in python] Day25: Python programming method and detailed explanation of concurrent programming

Python is a high-level programming language with concise and easy-to-read syntax and a rich standard library. It is widely used in many fields, including web development, data science, artificial intelligence, network programming, etc. The programming style of Python can be summarized as follows:

Script programming: Python is suitable as a scripting language, which can quickly write and run simple scripts to handle various tasks, such as file operations and data processing.

Object-oriented programming: Python supports object-oriented programming, which can create classes and objects, and realize code modularization and reuse through inheritance, encapsulation, and polymorphism.

Functional programming: Python supports functional programming. You can pass functions as parameters, return functions, and use higher-order functions to facilitate the compounding and abstraction of functions.

Asynchronous programming: Python provides asynciomodules, supports asynchronous programming, allows multiple I/O tasks to be processed in a single thread, and improves the concurrent performance of the program.

Concurrent programming: Python provides threadingand multiprocessingmodules that allow concurrent programming through multi-threading and multi-processing, making full use of multi-core CPUs.

Other features: Python also has features such as list comprehension, generator expressions, and decorators, making the code more concise and efficient.

In general, Python's programming method is simple and flexible, suitable for rapid development, suitable for projects of all sizes, and has become the language of choice for many developers and scientists.

2 sequential programming

Sequential programming refers to executing the program sequentially in the order in which the code is written, and executing the code line by line from top to bottom. This is the simplest and most basic way of programming, suitable for simple tasks and small-scale programs.

Example:

# 顺序编程示例
def add(a, b):
    return a + b

def multiply(a, b):
    return a * b

x = 5
y = 10

result1 = add(x, y)
result2 = multiply(x, y)

print("Result of addition:", result1)
print("Result of multiplication:", result2)

3 Object-oriented programming

Object-oriented programming is a programming paradigm that represents real-world things and relationships by creating classes and objects. In object-oriented programming, data and functionality are organized into objects, with classes defining properties and methods.

Example:

# 面向对象编程示例
class Rectangle:
    def __init__(self, width, height):
        self.width = width
        self.height = height

    def area(self):
        return self.width * self.height

    def perimeter(self):
        return 2 * (self.width + self.height)

rect = Rectangle(5, 10)
print("Area:", rect.area())
print("Perimeter:", rect.perimeter())

4 Functional programming

Functional programming is a programming paradigm that views computation as the application of mathematical functions, emphasizing the purity and immutability of functions. In functional programming, functions can be passed as arguments to other functions, and can also return other functions.

Example:

# 函数式编程示例
def add(a, b):
    return a + b

def multiply(a, b):
    return a * b

def apply_operation(operation, a, b):
    return operation(a, b)

x = 5
y = 10

result1 = apply_operation(add, x, y)
result2 = apply_operation(multiply, x, y)

print("Result of addition:", result1)
print("Result of multiplication:", result2)

5 Concurrent programming

Concurrent programming refers to the programming method of executing multiple tasks or processing multiple operations at the same time, which can improve the execution efficiency and responsiveness of the program. Python provides a variety of concurrent programming methods, including multi-threading, multi-processing and asynchronous programming.

5.1 Multithreaded programming

Multithreading is a method of concurrent programming that allows a program to execute multiple threads simultaneously, with each thread processing an independent task. Python threadingmodules provide multithreading support.

Example:

import threading
import time

def print_numbers():
    for i in range(1, 6):
        print(i)
        time.sleep(1)

def print_letters():
    for letter in 'ABCDE':
        print(letter)
        time.sleep(1)

if __name__ == "__main__":
    # 创建两个线程
    thread1 = threading.Thread(target=print_numbers)
    thread2 = threading.Thread(target=print_letters)

    # 启动线程
    thread1.start()
    thread2.start()

    # 等待两个线程执行完成
    thread1.join()
    thread2.join()

    print("All threads have finished.")

In the above example, we defined two functions print_numbersand print_letterswhich are used to print numbers and letters respectively. Then, we created two threads thread1and thread2and used these two functions as the execution targets of the threads. Threads are started by calling startmethods, and they execute concurrently. Finally, we use jointhe method to wait for both threads to finish executing, and then output "All threads have finished.".

It should be noted that due to Python's global interpreter lock (Global Interpreter Lock, GIL), multithreading cannot exert real parallel execution capabilities, and is suitable for use in IO-intensive tasks, such as network requests, file reading and writing, etc. For CPU-intensive tasks, it is recommended to use multi-process programming, that is, to use multiprocessingmodules to achieve.

`threading`Common usage of modules

threadingModules are the standard library for multithreaded programming in Python, which provides classes and functions for creating and managing threads.

1 Create a thread:

import threading

def my_function():
    # 任务代码

my_thread = threading.Thread(target=my_function)

2 Start the thread:

my_thread.start()

3 Wait for the thread to finish executing:

my_thread.join()

4 Get the current number of active threads:

threading.active_count()

5 Get the current thread object:

threading.current_thread()

6 Set the thread name:

my_thread = threading.Thread(target=my_function, name="MyThread")

7 Get the thread name:

my_thread.getName()

8 Set up a daemon thread (automatically exits when the main thread exits):

my_thread.daemon = True

9 thread synchronization - use Lock:

lock = threading.Lock()

def my_function():
    lock.acquire()
    # 临界区代码
    lock.release()

10 thread synchronization - use Semaphore (semaphore):

semaphore = threading.Semaphore(2)  # 限制同时执行的线程数为2

def my_function():
    semaphore.acquire()
    # 临界区代码
    semaphore.release()

11 thread synchronization - use Condition:

condition = threading.Condition()

def producer():
    with condition:
        # 生产者代码
        condition.notify()  # 通知消费者

def consumer():
    with condition:
        condition.wait()  # 等待生产者通知
        # 消费者代码

12 Inter-thread communication - using Queue:

import queue

my_queue = queue.Queue()

def producer():
    my_queue.put("hello")

def consumer():
    data = my_queue.get()
    print(data)

Note: Due to the Global Interpreter Lock (GIL) in Python, multithreading does not allow true parallel execution. For CPU-intensive tasks, multi-process programming ( multiprocessingmodules) is recommended. For IO-intensive tasks, multithreading is a good choice, which can switch to other threads while IO is waiting to improve the responsiveness of the program.

5.2 Multi-process programming

Multiprocessing is a method of concurrent programming that allows a program to execute multiple processes at the same time, and each process runs in an independent memory space. Python multiprocessingmodules provide multiprocessing support.

`multiprocessing` Common usage of modules

multiprocessingIt is a module in the Python standard library for multi-process concurrent programming. It provides threadinga module-like interface, but can execute tasks in parallel in multiple processes, taking full advantage of multi-core CPUs. Here are multiprocessingsome common uses of modules:

Create a process: Use multiprocessing.Processthe class to create a new process.

Start a process: Call start()the method of the process object to start a new process and start executing tasks.

Inter-process communication: Use classes such as multiprocessing.Queue, , multiprocessing.Pipeetc. to implement communication between multiple processes.

Process pool: Use multiprocessing.Poolthe class to create a process pool, which can facilitate process reuse and task scheduling.

Here is a simple example that demonstrates how to use multiprocessingmodules to implement multi-process concurrent programming:

import multiprocessing
import time

def worker(name):
    print(f"Worker {name} started")
    time.sleep(2)
    print(f"Worker {name} finished")

if __name__ == "__main__":
    # 创建两个新进程
    p1 = multiprocessing.Process(target=worker, args=("A",))
    p2 = multiprocessing.Process(target=worker, args=("B",))

    # 启动进程
    p1.start()
    p2.start()

    # 等待两个进程结束
    p1.join()
    p2.join()

    print("All processes finished")

output:

In the example above, we defined a workerfunction that simulates the tasks performed by each process. Then use multiprocessing.Processthe class to create two new processes and start them separately. Finally, use join()the method to wait for both processes to finish, and print "All processes finished" when all processes are finished.

It should be noted that when using multiprocessingmodules, ensure that the code of the main program is placed in if __name__ == "__main__":a statement block to avoid repeated execution of the code of the main program in the subprocess. At the same time, multiprocessingwhen the module is used under the Windows system, it is necessary to if __name__ == "__main__":create a process in the statement block to prevent infinite recursion of the process.

In the above sample code, we did not touch on the usage of inter-process communication and process pool. Below we will improve the sample code, including the use of inter-process communication and process pool:

import multiprocessing
import time

def worker(name, queue):
    print(f"Worker {name} started")
    time.sleep(2)
    queue.put(f"Result from worker {name}")

if __name__ == "__main__":
    multiprocessing.set_start_method('spawn')
    multiprocessing.freeze_support()

    # 创建进程间通信的队列
    manager = multiprocessing.Manager()
    queue = manager.Queue()

    # 创建进程池，池中有2个进程
    pool = multiprocessing.Pool(processes=2)

    # 启动进程池中的进程，每个进程执行worker函数，并使用 apply 方法改为阻塞方式
    pool.apply(worker, args=("A", queue))
    pool.apply(worker, args=("B", queue))

    # 关闭进程池，不再接受新的任务
    pool.close()

    # 等待所有进程完成
    pool.join()

    # 从队列中获取进程的结果
    results = []
    while not queue.empty():
        results.append(queue.get())

    print("All processes finished")
    print("Results:", results)

output:

         In the sample code above, we multiprocessing.Queueimplemented inter-process communication using After each child process completes the task, it will put the results into the queue, and the main process will get these results from the queue. At the same time, we multiprocessing.Poolcreated a process pool containing two processes, and apply_asyncsubmitted tasks to the process pool through methods.

        It should be noted that the size of the process pool can be adjusted according to the number of CPU cores of the system and the complexity of tasks to make full use of system resources.

        The above sample code shows the use of inter-process communication and process pool, so that multiple processes can execute tasks concurrently and communicate when needed. This can improve the efficiency and concurrent processing capabilities of the program.

Notice:

      On Windows, in order for to work properly, you need to add the and code multiprocessingbefore creating the process , which can ensure the normal execution of inter-process communication.multiprocessing.set_start_method('spawn')multiprocessing.freeze_support()

        In addition, please note that some IDEs (such as IDLE, etc.) are not friendly enough for multi-process output support, and sometimes cannot display the output of sub-processes. You can try running the code on the command line to see if it works.

        If there is still no output from the command line, there may be other problems, such as those related to the operating system or Python environment. You can try running the code in a different environment, or check for other exceptions.

5.3 Asynchronous programming

Asynchronous programming is a method of concurrent programming that allows programs to perform time-consuming operations without blocking the execution of other tasks, thereby improving program performance and responsiveness. In Python, asynchronous programming is usually asyncioimplemented using modules.

The key to asynchronous programming is to use the " async"and" await"keyword to define asynchronous functions and perform asynchronous operations. An asynchronous function can be defined by the " async def"keyword, and inside an asynchronous function, the " await"keyword can be used to wait for the completion of the asynchronous operation.

import asyncio

async def foo():
    print("Start foo")
    await asyncio.sleep(2)
    print("End foo")

async def bar():
    print("Start bar")
    await asyncio.sleep(1)
    print("End bar")

# 创建一个事件循环
loop = asyncio.get_event_loop()

# 执行异步任务
loop.run_until_complete(asyncio.gather(foo(), bar()))

# 关闭事件循环
loop.close()

        In the above example, we defined two asynchronous functions fooand bar, respectively simulating time-consuming tasks. In the main program, we use asyncio.gatherfunction to run these two async tasks at the same time, and use loop.run_until_completeto run event loop until all async tasks complete.

        Asynchronous programming is especially useful when dealing with IO-intensive tasks, such as network requests, file reading and writing, and so on. By using asynchronous programming, we can continue to perform other tasks while waiting for IO operations, thereby improving the efficiency of the program.

        It should be noted that in asynchronous programming, asynchronous functions cannot be called within ordinary synchronous functions, but only within other asynchronous functions. In addition, make sure that all asynchronous operations are non-blocking, otherwise the entire program may be blocked.

        Overall, asynchronous programming is a powerful way of concurrent programming that can significantly improve program performance and responsiveness. But it also requires careful design and consideration to ensure proper handling of asynchronous operations and avoid potential concurrency issues.

Common usage of asyncio module

asynciois an asynchronous programming module provided in the Python standard library for writing asynchronous code and managing the event loop. It provides a set of tools for defining asynchronous functions and handling asynchronous tasks. Here are asynciosome common uses of modules:

Defining asynchronous functions: Use async defkeywords to define asynchronous functions, which can contain awaitkeywords to wait for the completion of an asynchronous operation.

Create an event loop: Use asyncio.get_event_loop()to get an event loop object.

Run event loop: Use loop.run_until_complete()to run the event loop until the specified asynchronous task completes.

Execute asynchronous tasks concurrently: Use asyncio.gather()functions to run multiple asynchronous tasks concurrently and wait for them all to complete.

Asynchronous IO operations: Use asynciothe provided asynchronous IO functions, such as asyncio.open()and asyncio.write(), to perform non-blocking IO operations.

Here is a simple example that demonstrates how to use asynciomodules to execute asynchronous tasks concurrently:

import asyncio

async def foo():
    print("Start foo")
    await asyncio.sleep(2)
    print("End foo")

async def bar():
    print("Start bar")
    await asyncio.sleep(1)
    print("End bar")

async def main():
    # 并发执行foo和bar函数
    await asyncio.gather(foo(), bar())

# 创建一个事件循环
loop = asyncio.get_event_loop()

# 运行事件循环，直到main函数完成
loop.run_until_complete(main())

# 关闭事件循环
loop.close()

In the above example, we defined two asynchronous functions fooand barthen mainused asyncio.gather()function in function to run these two asynchronous functions at the same time. run_until_complete()Finally, run the function through the method of the event loop mainuntil all asynchronous tasks are completed.

It should be noted that when using asyncioasynchronous programming, avoid calling asynchronous functions in synchronous code, otherwise it may cause blocking. Try to use the keyword in the asynchronous environment awaitto wait for the completion of the asynchronous operation to ensure the efficiency and responsiveness of the program.