Table of Contents of Series Articles
1.项目介绍
2.相关技术补充
3.日志系统框架
4.代码设计
5. Functional testing
6.Performance testing
Article directory
Table of contents
Table of Contents of Series Articles
2. Relevant technical supplements
Technical implementation of log system
Preface
Why you need a logging system
- To ensure the stability and security of products in the production environment, developers are not allowed to attach debuggers to troubleshoot problems. You can use the log system to print some logs to help developers solve problems.
- There are bugs in products that are launched on the client side that cannot be reproduced and solved. You can use the log system to print logs and upload them to the server to help developers analyze them.
- For some high-frequency operations (timers, heartbeat packets), the behavior we want may not be triggered with a small number of debugging times. By pausing with breakpoints, we have to repeat the operation dozens of times or more, leading to troubleshooting problems. The efficiency is very low. You can check the problem by printing logs.
- In distributed, multi-threaded/multi-process code, it is difficult to locate bugs. You can use the log system to print logs to help locate bugs.
- Help people who come into contact with code for the first time understand the running process of the code
1. Project introduction
This project mainly implements a log system, which mainly supports the following functions:
- Supports multi-level log messages
- Supports synchronous and asynchronous logging
- Supports reliable writing of logs to console, files, and rolling files
- Supports concurrent log writing by multi-threaded programs
- Supports expansion of different log landing destinations
2. Development environment
- centos7
- vscode/vim
- g++/gdb
- Makefile
3. Core technology
- Class hierarchy design (application of inheritance and polymorphism)
- C++11 features (multithreading, auto, smart pointers, rvalue references, etc.)
- double buffer
- Production and consumption model
- Multithreading
- Design patterns (singleton, factory, proxy, template, etc.)
4. Introduction to log system
Technical implementation of log system
- Use printf, std::cout and other output functions to print log information to the console
- For large-scale commercial projects, in order to facilitate troubleshooting, logs are generally output to files or database systems to facilitate log query and analysis. They are mainly divided into synchronous logs and asynchronous logs.
Sync log
Synchronous log means that when outputting logs, you must wait for the execution of the log output statement to be completed before executing subsequent business logic statements. Log output statements and business logic statements run in a thread. Each time the print log API is called, there is a corresponding system call write to write the log file.
In high concurrency scenarios, as the number of logs continues to increase, the synchronous log system is prone to bottlenecks:
On the one hand, a large number of logs are trapped in the same number of write system calls, which has a certain system overhead.
On the one hand, the process of printing logs is accompanied by a large amount of synchronous disk IO, which affects program performance.
Asynchronous log
Asynchronous logging means that when performing log output, the log output statement and the business logic statement do not run in the same thread, but a dedicated thread is used for log output operations. The business thread only needs to put the log in a memory buffer and can execute subsequent business logic without waiting (as the producer of the log), while the landing operation of the log is left to the log thread alone (as the consumer of the log). It is a typical producer-consumer model.
The advantage of this is that even if the log is not output, it will not affect the main business of the program and can improve the performance of the program:
The main thread calls the log printing interface to become a non-blocking operation
Synchronized disk IO is separated from the main thread and handed over to a separate thread for completion.
Summarize
This section mainly introduces the overview of the logging system. Please see subsequent articles for details.