The principle of python multithreading and thread pool

Multithreading: do multiple things at the same time

Daemon thread: If the child thread is set as a daemon thread in the program, the child thread will automatically exit when the main thread ends. The setting method is thread.setDaemon(True), which should be set before thread.start(). The default is False, that is, when the main thread ends, the child threads are still executing.

thread.join(): Before the child thread finishes running, the parent thread of the child thread (usually the main thread) will always exist, that is, it will be blocked

The principle of
thread pool The basic principle of thread pool: We put tasks into the queue, and then open N threads, each thread goes to the queue to take a task, after the execution is finished, tell the system that I have executed it, and then go to the queue to take it The next task, until all tasks in the queue are empty, exit the thread.

The above example generates a thread pool with 3 threads, and each thread infinitely loops to block tasks that read the Queue queue. All tasks will only be processed by these 3 pre-generated threads.

The specific work is described as follows:

Create an instance of Queue.Queue(), and then fill it with data or tasks.
Generate a daemon thread pool and set the thread to a daemon daemon thread.
Each thread infinitely loops to block reading items from the queue queue, and processes
each time it completes a job. , Use the queue.task_done() function to send a signal to the queue where the task has been completed. The
main thread sets queue.join() to block until the task queue has been emptied, unblocked, and execute downward
. There are several points to note in this mode:

Setting the thread of the thread pool to the daemon process means that when the main thread exits, the daemon thread will automatically exit. If the default
daemon=False is used, the non-daemon thread will block the exit of the main thread, so even queue tasks Completed The
thread pool still blocks the infinite loop waiting for tasks, so that the main thread will not exit.

When the main thread uses queue.join(), it means that the main thread will block until the queue is emptied. How does the main thread know that the queue is emptied? That is, after each thread queue.get() and after processing the task, send the queue.task_done() signal, the queue data will be reduced by 1, until the queue data is empty, queue.join() unblocks, down carried out.

This mode is mainly based on the task of the queue and exits after completing the task. Since the thread pool is a daemon, all threads of the main exit thread pool will exit. Unlike our usual thread.join() blocking, which may lead to a queue, this kind of thread blocks the main thread before it completes. See which join() needs to be used:

If you want to finish the queue with a certain number of tasks, use queue.join(), such as crawling a specified number of web pages.
If you want the thread to finish the task, use thread.join()

import time
import queue
import threading

queue = queue.Queue()


# Define tasks that need to be performed by the thread pool
def do_job():
    while True:
        i = queue.get()
        time.sleep(1)
        print(i, threading.current_thread())
        queue.task_done()


if __name__ == '__main__':
    # Create a thread pool including 3 threads
    time.sleep(3)
    for i in range(10):
        queue.put(i)

    for i in range(3):
        t = threading.Thread(target=do_job)
        t.daemon = True # Set the thread daemon main thread exits, the daemon thread will also be launched, even if it is running
        t.start()

    # Simulate the creation of the thread pool and stuff 10 tasks into the queue after 3 seconds
    queue.join()