C++ skills series (9) - How to implement thread pool [detailed explanation]

Table of Contents of Series Articles

C++ High Performance Optimized Programming Series
In-depth Understanding of Software Architecture Design Series
Advanced C++ Concurrent Thread Programming
C++ Skills Series

Looking forward to your attention! ! !

现在的一切都是为将来的梦想编织翅膀，让梦想在现实中展翅高飞。
Now everything is for the future of dream weaving wings, let the dream fly in reality.

1. To achieve an efficient thread pool, you can consider the following points

• Control the number of threads: The size of the thread pool should be set according to system resource conditions and task volume. Too few threads will cause tasks to be blocked, and too many threads will consume too many system resources. This can be controlled using fixed-size thread pools, cacheable thread pools, or timer thread pools.

• Task queue management: The thread pool should have a task queue to store tasks waiting to be executed. Tasks can be managed using bounded or unbounded queues. A bounded queue can control the number of tasks, while an unbounded queue can accept any number of tasks, but may cause memory overflow.

• Thread scheduling strategy: The thread pool should have a suitable thread scheduling strategy, such as first-in-first-out, priority, etc. You can use a predefined implementation of a thread pool, or a custom implementation.

• Thread error handling: The thread pool should have an error handling mechanism to capture exceptions that may occur during thread execution to avoid causing the entire thread pool to collapse. Exceptions can be handled using try-catch statements or other exception handling mechanisms.

• Monitoring and tuning: The thread pool should have a monitoring and tuning mechanism to monitor the status and performance of the thread pool in real time and make corresponding adjustments. Monitoring and tuning can be done using monitoring tools, performance analysis tools, etc.

By properly setting the size of the thread pool, task queue management, thread scheduling strategy, error handling mechanism and monitoring and tuning, an efficient thread pool can be achieved and the program's concurrency performance and resource utilization can be improved.

2. To implement the thread pool, you can follow the following steps:

(1) Determine the basic parameters of the thread pool: including thread pool size, task queue size, rejection policy, etc. These parameters can be set according to actual needs.

(2) Create a task queue: used to store tasks to be executed. Queue data structures can be used, such as ArrayBlockingQueue, LinkedBlockingQueue, etc.

(3) Create a thread pool class: Define a thread pool class, including methods such as thread pool initialization, task submission, task execution, and shutdown. Thread pools can be implemented using the ThreadPoolExecutor class.

(4) Initialize thread pool: In the thread pool class, provide an initialization method that creates a fixed number of threads based on the thread pool size and puts them into the idle thread pool.

(5) Submit tasks: The thread pool class provides a method to submit tasks to add tasks to be executed to the task queue. Task submission can be achieved by calling the execute method of the thread pool class.

(6) Execute tasks: The thread pool will automatically obtain tasks from the task queue and assign them to idle threads for execution. After the task execution is completed, the thread will return to the thread pool and wait for the next task.

(7) Close the thread pool: The thread pool class provides a shutdown method to stop the running of the thread pool. When the thread pool is shut down, it waits for all tasks to complete and then terminates all threads.

(7) Error handling and monitoring: Error handling logic can be added to the thread pool to capture exceptions during task execution to avoid thread pool collapse. At the same time, a monitoring mechanism can be added to monitor the status and performance of the thread pool in real time.

According to the above steps, you can customize a thread pool class to implement the function of the thread pool. In actual use, set the parameters of the thread pool and tune the performance of the thread pool according to specific needs.

3. Simple C++ thread pool code example

The ThreadPool class in this example implements a simple thread pool, including thread creation, task submission, execution, and thread pool closing. In the main function, 10 tasks are submitted using the thread pool, and each task outputs its own number and the thread ID that executes it. After all tasks are performed, the program waits for 2 seconds before exiting.

Please note that this sample code is just a funny demonstration and may not have some important details of thread safety and practical application. Please do not use it in actual production environments. When actually using the thread pool, you need to consider more issues such as thread synchronization, task splitting, and exception handling.

#include <iostream>
#include <thread>
#include <vector>
#include <queue>
#include <functional>

class ThreadPool {
    
    
public:
    ThreadPool(int numThreads) : stop(false) {
    
    
        for (int i = 0; i < numThreads; ++i) {
    
    
            threads.emplace_back(std::thread([this](){
    
    
                while (true) {
    
    
                    std::function<void()> task;
                    {
    
    
                        std::unique_lock<std::mutex> lock(queueMutex);
                        condition.wait(lock, [this]{
    
     return stop || !tasks.empty(); });
                        if (stop && tasks.empty()) {
    
    
                            return;
                        }
                        task = std::move(tasks.front());
                        tasks.pop();
                    }
                    task();
                }
            }));
        }
    }

    ~ThreadPool() {
    
    
        {
    
    
            std::unique_lock<std::mutex> lock(queueMutex);
            stop = true;
        }
        condition.notify_all();
        for (auto& thread : threads) {
    
    
            thread.join();
        }
    }

    template <typename FuncType>
    void submit(FuncType f) {
    
    
        {
    
    
            std::unique_lock<std::mutex> lock(queueMutex);
            tasks.emplace([f]() {
    
     f(); });
        }
        condition.notify_one();
    }

private:
    std::vector<std::thread> threads;
    std::queue<std::function<void()>> tasks;
    std::mutex queueMutex;
    std::condition_variable condition;
    bool stop;
};

int main() {
    
    
    ThreadPool pool(4);
    for (int i = 0; i < 10; ++i) {
    
    
        pool.submit([i]() {
    
    
            std::cout << "Task " << i << " executed by thread " << std::this_thread::get_id() << std::endl;
        });
    }
    std::this_thread::sleep_for(std::chrono::seconds(2)); // 等待所有任务执行完
    return 0;
}

• Use std::thread::hardware_concurrency() to determine the number of threads in the thread pool, which is usually equal to the number of processor cores to fully utilize system resources.

• Use std::function as the task type, which can accept any callable object, and use Lambda expressions to encapsulate specific tasks.

• Use std::queue as the task queue, and implement thread synchronization and mutual exclusion through std::mutex and std::condition_variable.

• The thread's suspension and wake-up are controlled through wait() and notify_one() of the condition variable std::condition_variable.

4. Source code library written based on boost - thread pool

4.1 Source code library address written based on boost

Official website address: http://threadpool.sourceforge.net

Header file directory:

4.2 First-in-first-out, last-in-first-out, and priority code examples of boost thread pool

#include <./boost/threadpool.hpp>
using namespace std;
using namespace boost::threadpool;
// Helpers
boost::mutex m_io_monitor;

void print(string text)
{
    
    
	boost::mutex::scoped_lock lock(m_io_monitor);
	cout << text;
}

template<typename T>
string to_string(T const & value)
{
    
    
	ostringstream ost;
	ost << value;
	ost.flush();
	return ost.str();
}

// An example task functions
void task_1()
{
    
    
	Sleep(3000);
	print("  task_1()\n");
}

void task_2()
{
    
    
	Sleep(3000);
	print("  task_2()\n");
	//Sleep(10000);
}

void task_3()
{
    
    
	print("  task_3()\n");
}

int task_4()
{
    
    
	print("  task_4()\n");
	return 4;
}

void task_with_parameter(int value)
{
    
    
	Sleep(3000);
	print("  task_with_parameter(" + to_string(value) + ")\n");
}

int loops = 0;
bool looped_task()
{
    
    
	print("  looped_task()\n");
	return ++loops < 5; 
}


int task_int()
{
    
    
	print("  task_int()\n");
	return 23;
}


void fifo_pool_test()
{
    
    
	pool tp;
	
	tp.schedule(&task_1);
	tp.schedule(boost::bind(task_with_parameter, 4));

	if(!tp.empty())
	{
    
    
		tp.clear();  // remove all tasks -> no output in this test
	}

	size_t active_threads   = tp.active();
	size_t pending_threads  = tp.pending();
	size_t total_threads    = tp.size();

	size_t dummy = active_threads + pending_threads + total_threads;
	dummy++;

	tp.size_controller().resize(5);
	tp.wait();
}

void lifo_pool_test()
{
    
    
	lifo_pool tp;
	tp.size_controller().resize(0);
	schedule(tp, &task_1);
	tp.size_controller().resize(10);
	tp.wait();
}

void prio_pool_test()
{
    
    
	prio_pool tp(2);
	schedule(tp, prio_task_func(1, &task_1));
	schedule(tp, prio_task_func(10,&task_2));
	tp.schedule(prio_task_func(3000,boost::bind(task_with_parameter, 4)));
	tp.schedule(prio_task_func(3500,boost::bind(task_with_parameter, 5)));
	tp.schedule(prio_task_func(3600,boost::bind(task_with_parameter, 6)));
	tp.schedule(prio_task_func(3900,boost::bind(task_with_parameter, 9)));
	tp.schedule(prio_task_func(5000,boost::bind(task_with_parameter, 10)));
	tp.schedule(prio_task_func(8000,boost::bind(task_with_parameter, 11)));
	tp.schedule(prio_task_func(3000,boost::bind(task_with_parameter, 4)));
	tp.schedule(prio_task_func(3500,boost::bind(task_with_parameter, 5)));
	tp.schedule(prio_task_func(3600,boost::bind(task_with_parameter, 6)));
	tp.schedule(prio_task_func(3900,boost::bind(task_with_parameter, 9)));
	tp.schedule(prio_task_func(5000,boost::bind(task_with_parameter, 10)));
	tp.schedule(prio_task_func(8000,boost::bind(task_with_parameter, 11)));
	tp.schedule(prio_task_func(3000,boost::bind(task_with_parameter, 4)));
	tp.schedule(prio_task_func(3500,boost::bind(task_with_parameter, 5)));
	tp.schedule(prio_task_func(3600,boost::bind(task_with_parameter, 6)));
	tp.schedule(prio_task_func(3900,boost::bind(task_with_parameter, 9)));
	tp.schedule(prio_task_func(5000,boost::bind(task_with_parameter, 10)));
	tp.schedule(prio_task_func(8000,boost::bind(task_with_parameter, 11)));
	tp.schedule(prio_task_func(3000,boost::bind(task_with_parameter, 4)));
}

void future_test()
{
    
    
	fifo_pool tp(5);
	future<int> fut = schedule(tp, &task_4);
	int res = fut();
}


int main (int , char * const []) 
{
    
    
	//fifo_pool_test();
	//lifo_pool_test();
	prio_pool_test();
	//future_test();
	system("pause");
	return 0;
}

5. Take a look at how other people write thread pools - you need to understand the essence

Address: https://github.com/xyygudu/ThreadPool

6. Thread pool application scenarios and practices

6.1 Server application

Thread pools have a wide range of application scenarios in server applications. Servers often need to handle a large number of client requests. When a client request arrives, the server can use a thread from the thread pool to handle the request, enabling efficient task scheduling and resource utilization.

• Request processing
  Allocate client requests to threads in the thread pool for processing, which can effectively achieve load balancing. The server can dynamically adjust the number of threads in the thread pool based on the load of each thread. This helps maintain server performance and responsiveness during peaks and valleys.

• Establishing connections
  The thread pool is used to establish new connections. When a new client connection arrives, a thread in the thread pool can perform handshake and initialization operations. In this way, when there are many client connection requests, the thread pool can quickly handle new connections and avoid creating a large number of short-lived threads.

• Data Reading/Writing
  The thread pool can be used to handle data reading/writing operations with the client. When a read/write operation blocks, other threads in the thread pool can still continue to process subsequent requests.

• Asynchronous operations
  Thread pools can be used to implement asynchronous operations. For example, the server may need to write the results of the client's operations to a log or database. One thread in the thread pool can perform these operations without affecting other threads that are processing the request.

• Advantages
  Servers using thread pools have the following advantages:
  (1) Improve response speed. Threads in the thread pool can start executing new tasks immediately without waiting for the operating system to create new threads.
  (2) Improve resource utilization. By reusing threads, thread pools can reduce the overhead of creating and destroying threads and save resources.
  (3) Control the number of concurrencies. The thread pool can limit the number of threads running at the same time to avoid excessive thread competition leading to system performance degradation.
  (4) Provide scalability. The thread pool can dynamically adjust the number of threads according to the system load to adapt to different operating environments.
  In summary, using thread pools in server applications helps improve performance, reduce resource consumption, and provide good scalability.

6.2 Data processing and computationally intensive tasks

Thread pools demonstrate superior performance and ease of use in data processing and computationally intensive tasks. Large-scale data processing and computationally intensive tasks can often be split into multiple smaller subtasks, which can be computed independently and executed concurrently.

• Data processing tasks
  Data processing tasks involve operations such as cleaning, classification, and retrieval of large amounts of data. Assigning these operations to threads in a thread pool can speed up the data processing process. For example, when performing a full-text search on a large-scale data set, the thread pool can divide the data set into multiple subsets and let each thread search on one subset. In this way, the data processing process can be executed in parallel, greatly shortening the task completion time.

• Computing-intensive tasks
  Computing-intensive tasks require a large number of arithmetic operations or logical operations, such as image processing, video encoding and decoding, and machine learning. These tasks are characterized by heavy computation and long execution time, and usually require high-performance computing resources. Using the thread pool can make full use of the computing power of multi-core processors and improve the efficiency of task execution.

• Data Parallelism and Task Parallelism
  In data processing and computing-intensive tasks, the thread pool can adopt data parallelism and task parallelism strategies.
  (1) Data parallelism: Split the data set into multiple subsets, and each thread operates on a subset. Data parallelism is suitable for processing different subsets of tasks independently.
  (2) Task parallelism: Split the task into multiple subtasks, and each thread executes one subtask. Task parallelism is suitable for scenarios where there are dependencies between subtasks.
  Based on the task characteristics and data size, you can choose an appropriate parallel strategy and adjust the number of threads in the thread pool to optimize performance.

• Advantages
  Using thread pools in data processing and computing-intensive tasks has the following advantages:
  (1) Increase execution speed. The thread pool can make full use of multi-core processors for concurrent calculations and shorten task completion time.
  (2) Reduce resource consumption. By reusing threads, thread pools reduce the overhead of creating and destroying threads.
  (3) Flexible scheduling. The thread pool can dynamically adjust the number of threads based on the type of task and data size, providing scalability.
  (4) Simplify the programming model. The thread pool encapsulates thread management and task scheduling, reducing programming difficulty and complexity.
  Therefore, using thread pools in data processing and computing-intensive tasks can improve task execution efficiency and simplify the programming model of parallel computing.

6.3 Graphical interface and event-driven program

Thread pools play an important role in graphical interfaces and event-driven programs. In order to maintain the smoothness of the user interface (UI), time-consuming operations often need to be executed in worker threads in the thread pool to avoid blocking the UI thread.

• Background tasks
  In many graphical interface applications, some time-consuming tasks need to be performed in the background, such as file operations, network requests, large-scale calculations, etc. These tasks can be put into the thread pool for execution to avoid blocking the UI thread. After the task is completed, the results can be passed to the UI thread for display through a callback function or other means.

• Asynchronous event handling
  Event-driven programs need to respond to events from external or internal sources. These events may have undetermined delays. In order to avoid blocking the UI thread, event processing tasks can be submitted to the thread pool. This way, when handling multiple events, the UI thread can remain responsive between any events.

• Scheduled tasks
  Some graphical interface applications need to perform tasks at specific times, such as animations, timers, etc. Assigning these tasks to threads in the thread pool for processing can ensure that the timer tasks are triggered at precise times and avoid blocking of the UI thread.

• Advantages
  Using thread pools in graphical interfaces and event-driven programs has the following advantages:
  (1) Keep the UI smooth. Worker threads in the thread pool can execute time-consuming tasks concurrently to avoid blocking the UI thread.
  (2) Optimize resource utilization. The thread pool manages worker threads and reduces the overhead of creating and destroying threads.
  (3) Asynchronous event processing. The thread pool provides a simple and efficient way to handle events from internal or external sources, improving the responsiveness of the program.
  (3) Adaptive scheduling. The thread pool can dynamically adjust the number of threads based on task load to adapt to changes when the program is running.
  Solve time-consuming tasks and event processing problems in graphical interfaces and event-driven programs through thread pools, which helps avoid UI thread blocking and improve program responsiveness. At the same time, the thread pool optimizes resource utilization and adapts to load changes when the program is running.

7. Advanced applications and practical cases of C++ thread pool

7.1 Task allocation strategy based on load balancing

Load balancing is critical to the performance and stability of the thread pool when handling multiple concurrent tasks. The following strategies help achieve load balancing based task distribution:

• Dynamic task scheduling
  Dynamic task scheduling means monitoring the workload of individual threads in the thread pool in real time so that the workload is taken into account when assigning tasks. When a new task enters the thread pool, it is assigned to the thread with the lowest current workload. Task execution times may not be consistent, so choosing the least loaded thread to run a new task can help avoid processing bottlenecks.
  To implement dynamic task scheduling, the following methods can be used:
  (1) Polling scheduling: assign each new task to a thread in the thread pool in turn. This approach is simple and effective, but may lead to uneven distribution of tasks in some cases.
  (2) Minimum load first: Calculate the thread load according to the current number of tasks of the thread or the size of the assigned tasks, and assign new tasks to the thread with the lowest load.

• Thread load monitoring
  By monitoring each thread in the thread pool in real time, we can understand their load status so that they can be assigned tasks according to actual needs. The following indicators can be used to represent thread load:
  (1) Current number of tasks
  (2) Number of tasks waiting to be processed
  (3) Number of completed tasks
  (4) CPU usage of threads
  Combining these thread load information with task scheduling, you can Allow the thread pool to better distribute tasks and adapt to load changes.

• Solving for optimal allocation
  In order to achieve optimal load balancing, various methods can be used to find the best task allocation scheme. Two possible methods are introduced here:
  (1) Greedy algorithm: Make the local situation optimal by always assigning tasks to the thread with the lowest current load. The advantage of this method is that it is simple and easy to implement, but it may not find the global optimal solution.
  (2) Simulated annealing algorithm: For more complex load balancing problems, the simulated annealing algorithm can be used to solve the global optimal solution. Although it is possible to find task allocations that are close to the global optimum, it is computationally expensive in some cases.
  Considering the difficulty of implementation and operating effects, under normal circumstances, simple methods such as round-robin scheduling and minimum load priority can effectively achieve load balancing. In scenarios with very complex load conditions, you can consider using optimization algorithms such as simulated annealing to find better solutions.

7.2 Thread pool performance optimization skills

To improve thread pool performance, you need to pay attention to the following aspects:

• Moderate concurrency
  An appropriate concurrency level can not only make full use of system resources, but also ensure that threads run efficiently with a limited number of cores. A concurrency level that is too low will lead to a waste of resources, while a concurrency level that is too high may lead to increased thread competition, thus affecting performance. The concurrency level in the thread pool can be set based on the following empirical values:
  (1) CPU-bound tasks: Set the concurrency level to the number of processor cores to ensure full utilization of CPU resources in highly computing-intensive scenarios.
  (2) I/O bound tasks: When processing I/O-intensive tasks, set the concurrency level to slightly higher than the number of processor cores. This allows other threads to continue executing while waiting for the I/O operation to complete, thus Improve overall performance.

• Reduce lock contention
  Avoiding unnecessary lock contention is very important to improve thread pool performance. The following methods can help reduce the impact of lock competition:
  (1) Lock-free data structure: Using lock-free data structures can achieve better performance in a multi-threaded environment.
  (2) Fine-grained lock: Limiting the scope of the lock to the resources or operations that need to be protected can reduce the possibility of conflicts.
  (3) Read-write lock: Such as std::shared_mutex in C++. In the scenario of more reading and less writing, the performance of read-write lock is better than that of ordinary mutex lock (such as std::mutex).

• Writing efficient code
  Writing efficient thread task code is critical to the overall performance of the thread pool. The following principles help improve task code efficiency:
  (1) Avoid repeated calculations and inefficient operations: Avoid repeated calculations and inefficient operations as much as possible to improve the efficiency of computing-intensive tasks.
  (2) Make full use of C++ containers and algorithms: Properly use the containers and algorithms provided in the C++ standard library to achieve high-performance and concise code.
  (3) Master the concurrent programming features of C++: Make full use of the concurrency and multi-thread support tools in C++11/14/17/20, such as std::thread, std::async, std::future, std::atomic etc. to avoid inefficient and redundant concurrent structures.
  Following these principles and taking action can significantly improve the performance and stability of the thread pool, ensuring good accuracy and efficiency in handling complex multi-tasking scenarios.

8. Actual case analysis and excellent practices

The following will analyze the application of thread pools in various scenarios through several practical cases, and discuss how to improve task processing efficiency by combining excellent practices.

8.1 Case 1: Concurrent network service

When dealing with concurrent network services, the thread pool can be used to handle requests from clients, such as establishing connections, reading and writing data, and processing tasks. By assigning these tasks to thread processing in a thread pool, the server can achieve better performance, responsiveness, and scalability.
(1) Use a thread pool to handle network tasks such as connections, reading and writing, and reduce the pressure on single-threaded servers.
(2) Allocate an appropriate number of threads to process tasks based on actual business needs to achieve high performance and low latency.
(3) Properly adopt load balancing strategies to allocate tasks to ensure that the workload of each thread is close to balance.

8.2 Case 2: Parallel Computing and Data Processing

When dealing with parallel computing and data processing tasks, these tasks can be divided into multiple subtasks and assigned to different threads for processing. The thread pool can quickly implement efficient parallel computing and improve processing speed.
(1) Split large-scale parallel computing tasks into multiple subtasks, and assign the subtasks to threads in the thread pool.
(2) Define different parallel strategies based on different characteristics and sizes of tasks and data scale, such as data parallelism and task parallelism.
(3) When processing complex numerical calculations, make full use of the computing power of multi-core processors and optimize the concurrency level.

8.3 Case 3: High-performance Web server

High-performance web servers need to handle thousands of concurrent requests. To deal with such high-stress scenarios, thread pools are ideal to distribute the tasks of processing and responding to incoming requests to different threads.
(1) Processing requests: Allocate read/write requests for each client connection to threads in the thread pool for processing.
(2) Queued tasks: In order to avoid long-awaited response requests from blocking other tasks, priority queues or other scheduling strategies can be used to arrange the processing order of tasks.
(3) Resource separation: Assign processing tasks of different resources to different types of thread pools to achieve the goals of resource isolation and performance optimization.

By combining these actual cases with excellent practices, the thread pool can achieve excellent performance in various scenarios, thereby improving the efficiency and stability of our task processing.