Master Python multithreading and multiprocessing in one article

Python multithreading and multiprocessing

1. Introduction

Concurrency is an important capability in computer programming today, especially when faced with tasks that require heavy computation or I/O operations. Python provides a variety of concurrency processing methods. This article will discuss two of them in depth: multi-threading and multi-processing, analyze their usage scenarios, advantages, and disadvantages, and combine code examples for in-depth interpretation.

Two, multithreading

Threads in Python are threadingimplemented using modules. Threads are different tasks running within the same process.

2.1 Basic use of threads

Creating and starting threads in Python is simple. Here is a simple example:

import threading
import time

def print_numbers():
    for i in range(10):
        time.sleep(1)
        print(i)

def print_letters():
    for letter in 'abcdefghij':
        time.sleep(1.5)
        print(letter)

thread1 = threading.Thread(target=print_numbers)
thread2 = threading.Thread(target=print_letters)

thread1.start()
thread2.start()

In this example, the print_numbersand print_lettersfunctions are executed in their own threads without interfering with each other.

2.2 Thread synchronization

Since threads share memory, data between threads can be accessed by each other. However, problems arise when multiple threads modify data at the same time. In order to solve this problem, we need to use thread synchronization tools, such as lock (Lock) and condition (Condition), etc.

import threading

class BankAccount:
    def __init__(self):
        self.balance = 100  # 共享数据
        self.lock = threading.Lock()

    def deposit(self, amount):
        with self.lock:  # 使用锁进行线程同步
            balance = self.balance
            balance += amount
            self.balance = balance

    def withdraw(self, amount):
        with self.lock:  # 使用锁进行线程同步
            balance = self.balance
            balance -= amount
            self.balance = balance

account = BankAccount()

Special Note : Although Python threads are limited by the Global Interpreter Lock (GIL), for IO-intensive tasks (such as network IO or disk IO), using multithreading can significantly improve the execution efficiency of the program.

Three, multi-process

Processes in Python are multiprocessingimplemented through modules. A process is an execution entity in the operating system. Each process has its own memory space and does not affect each other.

3.1 Basic use of processes

Creating and starting a process in Python is also very simple:

from multiprocessing import Process
import os

def greet(name):
    print(f'Hello {name}, I am process {os.getpid()}')

if __name__ == '__main__':
    process = Process(target=greet, args=('Bob',))
    process.start()
    process.join()

3.2 Communication between processes

Since processes do not share memory, inter-process communication (IPC) needs to use specific mechanisms, such as pipes (Pipe), queues (Queue), and so on.

from multiprocessing import Process, Queue

def worker(q):
    q.put('Hello from

 process')

if __name__ == '__main__':
    q = Queue()
    process = Process(target=worker, args=(q,))
    process.start()
    process.join()

    print(q.get())  # Prints: Hello from process

Special note : Python's multi-process is a good choice for computing-intensive tasks, because each process has its own Python interpreter and memory space, which can be calculated in parallel.

One More Thing

Let's take a deeper look at concurrent.futuresmodules, a more advanced tool for handling both multithreading and multiprocessing in Python. concurrent.futuresmold

The block provides a high-level interface, which puts asynchronously executed tasks into the pool of threads or processes, and then obtains the execution results through the future object. This module makes working with threads and processes easier.

Below is an example:

from concurrent.futures import ThreadPoolExecutor, as_completed

def worker(x):
    return x * x

with ThreadPoolExecutor(max_workers=4) as executor:
    futures = {executor.submit(worker, x) for x in range(10)}
    for future in as_completed(futures):
        print(future.result())

This code creates a thread pool and submits 10 tasks to the thread pool. Then, get the result of each task through the future object. The functions here as_completedprovide a way to handle completed futures.

This way you can easily switch threads and processes, just ThreadPoolExecutorchange to ProcessPoolExecutor.

Whether you are dealing with IO-intensive tasks or computationally-intensive tasks, Python's multithreading and multiprocessing provide a good solution. Understanding their operating mechanism and applicable scenarios can help you better design and optimize your programs.

If it is helpful, please pay more attention to the personal WeChat public account: [Python full perspective] TeahLead_KrisChang, 10+ years of experience in the Internet and artificial intelligence industry, 10+ years of experience in technology and business team management, Tongji Software Engineering Bachelor, Fudan Engineering Management Master, Aliyun certified cloud service senior architect, head of AI product business with hundreds of millions of revenue.