Spring Boot Series: Detailed explanation of log dynamic configuration

1. Introduction

Spring Boot version: 2.3.4.RELEASE

I don’t know if you have ever needed some DEBUG logs when there are problems online, but INFO is currently being used.

If you want to enable DEBUG, you need to repackage and release the version, but in some scenarios, the problem may not recur after restarting, which is really painful.

Today we will talk about the dynamic adjustment of the log configuration under Spring Boot, so that your log level can be adjusted as you want.

Spring Boot log

What is actually used inside Spring Boot is Commons Logging, and the configuration loading mechanism based on Spring Boot provides us with Java Util Logging, Log4j2, and Logback logging methods.

Logback is its default log framework. If there is no special need, it is really not recommended to replace it. (Don't talk about performance)

Log format

Don't underestimate the format, it is an important thing in actual application.

I don't know if your company has a unified log basic component. Of course, there is no or there will probably be a unified log configuration file.

Think about if your log format is not uniform, if each project has its own style, how can you ask your operation and maintenance partner to help you split the log? Do you call the police? That's really regular writing to death, completely relying on love to generate electricity. (For example, if the Loghub we use is not uniform, it will be killed by operation and maintenance)

Let’s take a look at our log format configuration, here only PATTERN

-|%d{yyyy-MM-dd HH:mm:ss.SSS}|%-5level|%X{tid}|%thread|%logger{36}.%M:%L-%msg%n

First explain each position:

• %d{yyyy-MM-dd HH:mm:ss.SSS}: time
• %-5level: log level
• %X{tid}: Our custom distributed tracking ID
• %thread: thread
• %logger{36}.%M:%L: the full name of the class (36 represents the longest character). Information line number
• %msg%n: output information to wrap

I don’t know if you can understand why there is a -| in front of it? In fact, it is for the convenience of distinguishing the newline log when regular segmentation, such as exception stack information.

Several knowledge points

Let's talk about a few other small points used by Spring Boot, let's get to the topic

• Here, Logback does not have a FATAL level and is classified as ERROR
• You can configure debug=true in application.properties to enable debug mode, you can also configure trace=true to enable trace mode
• You can use logging.level.= in application.properties to configure various log levels, such as org.hibernate level is set to ERROR logging.level.org.hibernate=error
• The concept of log group, if such a configuration is annoying, you can set a group for its overall configuration. Such as: logging.group.tomcat=org.apache.catalina, org.apache.coyote configuration level is TRACE logging.level.tomcat=TRACE
• If you use the Logback configuration file, the official recommendation is to use a name like logback-spring.xml. If you use logback.xml, Spring's log initialization should be a problem.
• If you control the log configuration, but do not want to use logback.xml as the name of the Logback configuration, you can specify a custom name through the logging.config property in the application.properties configuration file: logging.config=classpath:xxx-log.xml

Two, dynamically modify the log level

Let's talk about how the Spring Boot application in the running state changes the dynamic log level

Spring Boot Actuator

Actuator must know Spring Boot's name, monitoring, auditing, measurement information and so on. The graceful shutdown, health check, and log information we use are all achieved through this stuff.

After we use Actuator, we can use its Logger REST interface to manipulate our logs, there are three as follows

• GET http://127.0.0.1:6080/actuator/loggers returns all log level information of the current application (what you want or what)
• GET http://127.0.0.1:6080/actuator/loggers/{name} returns the log level of {name}
• POST http://127.0.0.1:6080/actuator/loggers/{name} configuration parameters {"configuredLevel":"INFO","effectiveLevel":"INFO"} modify log level

Use Actuator mechanism to dynamically modify the level

1), rely on the necessary configuration Spring Boot Actuator (if you inherit the parent, you don’t need a version)

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
    <version>2.3.4.RELEASE</version>
</dependency>

2), configuration file

#注意，这里我只开启了loggers,health两个Actuator
management.endpoints.web.exposure.include=loggers,health
management.endpoint.loggers.enabled=true

3), create a Controller

@RestController
@RequestMapping("/log")
public class TestLogController {

    private Logger log = LoggerFactory.getLogger(TestLogController.class);

    @GetMapping
    public String log() {
        log.trace("This is a TRACE level message");
        log.debug("This is a DEBUG level message");
        log.info("This is an INFO level message");
        log.warn("This is a WARN level message");
        log.error("This is an ERROR level message");
        return "See the log for details";
    }
}

4), start the application

At this point, after you start the application, if you use IDEA, you can see the following Mapping, there are three REST interfaces about logger

5), test

At this point, I execute GET http://127.0.0.1:6080/log first, and the console prints the following information

2020-10-15 23:14:51.204  INFO 52628 --- [nio-6080-exec-7] c.q.chapter1.failure.TestLogController   : This is an INFO level message
2020-10-15 23:14:51.220  WARN 52628 --- [nio-6080-exec-7] c.q.chapter1.failure.TestLogController   : This is a WARN level message
2020-10-15 23:14:51.221 ERROR 52628 --- [nio-6080-exec-7] c.q.chapter1.failure.TestLogController   : This is an ERROR level message

At this time, we execute GET http://127.0.0.1:6080/actuator/loggers/ROOT and return {"configuredLevel":"INFO","effectiveLevel":"INFO"}, which means that the log level of our application is INFO at this time. ROOT represents the root node.

6), modify the level

At this time, we don’t want to look at the INFO information anymore. If we want to change the entire application log level to WARN, let’s execute POST http://127.0.0.1:6080/actuator/loggers/ROOT The parameter is {"configuredLevel":"TRACE", "EffectiveLevel":"TRACE"}

At this point, we will execute GET http://127.0.0.1:6080/log again, and the console will print the following information

2020-10-15 23:24:11.481 TRACE 53552 --- [nio-6080-exec-3] c.q.chapter1.failure.TestLogController   : This is a TRACE level message
2020-10-15 23:24:11.481 DEBUG 53552 --- [nio-6080-exec-3] c.q.chapter1.failure.TestLogController   : This is a DEBUG level message
2020-10-15 23:24:11.481  INFO 53552 --- [nio-6080-exec-3] c.q.chapter1.failure.TestLogController   : This is an INFO level message
2020-10-15 23:24:11.481  WARN 53552 --- [nio-6080-exec-3] c.q.chapter1.failure.TestLogController   : This is a WARN level message
2020-10-15 23:24:11.481 ERROR 53552 --- [nio-6080-exec-3] c.q.chapter1.failure.TestLogController   : This is an ERROR level message

Other methods

• Configuration file scanning: It is the feature modification level of Logback automatic scanning. With logback-spring.xml enabled, you can dynamically modify the internal log level of logback-spring.xml. If you are interested, you can try it yourself.
• arthas dynamic modification
• Combined with remote configuration center, such as Apollo to achieve dynamic level modification

Three, realization principle

Here we mainly use Spring Boot Actuator Log, so we will also talk about its principle.

Endpoint loading

First, we find our LoggersEndpoint from the dependent spring-boot-actuator (all Actuators are of this number), as shown in the figure:

Friends who are familiar with Spring Boot's loading mechanism know that behind each actuator Endpoint, there must be a xxxEndpointAutoConfiguration to load the Endpoint for us.

These loading mechanisms are all stored in spring-boot-actuator-autoconfigure, where we can find the LoggersEndpointAutoConfiguration to load the configuration class of LoggersEndpoint.

Take a look at its core:

@Bean
	@ConditionalOnBean(LoggingSystem.class)
	@Conditional(OnEnabledLoggingSystemCondition.class)
	@ConditionalOnMissingBean
	public LoggersEndpoint loggersEndpoint(LoggingSystem loggingSystem,
			ObjectProvider<LoggerGroups> springBootLoggerGroups) {
		return new LoggersEndpoint(loggingSystem, springBootLoggerGroups.getIfAvailable(LoggerGroups::new));
	}

You can see two important parameters:

• LoggingSystem: an abstract top-level class
• springBootLoggerGroups: stores the current log group data

To summarize:

1. After we rely on the spring-boot-starter-actuator package, it relies on spring-boot-actuator-autoconfigure

2. When starting to scan to META-INF/spring.factories under spring-boot-actuator-autoconfigure, LoggersEndpointAutoConfiguration will be loaded to

3. LoggersEndpoin is declared in LoggersEndpointAutoConfiguration and assigned LoggingSystem and springBootLoggerGroups as its parameters

4. After the project is started, we will access log data through the LoggersEndpoint interface

LoggingSystem

Inheritance of LoggingSystem

From the above, we can understand that LoggingSystem is the core of log operation management, so let's take a look at his Diagrams first

Through the inheritance relationship, we can see at a glance that the LogbackLoggingSystem is our master. Although we know the master, how exactly is our LoggingSystem loaded?

Loading of LoggingSystem

The main participants are as follows:

• LoggingApplicationListener
• ApplicationStartingEvent
• LogbackLoggingSystem
• LoggingSystem

Not much to say, first picture

1. The application uses SpringApplication.run(Chapter1Application.class, args); start

2. Send startup events ApplicationStartingEvent, ApplicationEnvironmentPreparedEvent, etc.

3. LoggingApplicationListener receives events for event distribution

4. LoggingApplicationListener receives the event ApplicationStartingEvent

5. LoggingApplicationListener calls the internal onApplicationStartingEvent(ApplicationStartingEvent event) method, and initializes the LoggingSystem using LoggingSystem.get(classloader).

6. Then call LogbackLoggingSystem.beforeInitialize() (because we are using logback here)

7, LoggingApplicationListener receives the event ApplicationEnvironmentPreparedEvent

8. LoggingApplicationListener calls the internal onApplicationEnvironmentPreparedEvent(ApplicationEnvironmentPreparedEvent event) method

9. onApplicationEnvironmentPreparedEvent calls the initialize(ConfigurableEnvironment environment, ClassLoader classLoader) method, which will initialize the logging system according to the environment variable configuration

LoggersEndpoint

Finally, let’s take a look at LoggersEndpoint. I won’t copy all of them here for the convenience of interpretation. Where to choose?

First look at our three interfaces for log operations:

• GET /actuator/loggers corresponds to the public Map<String, Object> loggers() method
• GET /actuator/loggers/{name} 对应 public LoggerLevels loggerLevels(@Selector String name)方法
• POST /actuator/loggers/{name} 对应 public void configureLogLevel(@Selector String name, @Nullable LogLevel configuredLevel) 方法

Here is an example of the public Map<String, Object> loggers() method, the others are similar

@ReadOperation
	public LoggerLevels loggerLevels(@Selector String name) {
		Assert.notNull(name, "Name must not be null");
		LoggerGroup group = this.loggerGroups.get(name);
		if (group != null) {
			return new GroupLoggerLevels(group.getConfiguredLevel(), group.getMembers());
		}
		LoggerConfiguration configuration = this.loggingSystem.getLoggerConfiguration(name);
		return (configuration != null) ? new SingleLoggerLevels(configuration) : null;
	}

The code is actually very simple, so I won't do too much interpretation.

Four, talk

In fact, logs are very important in our system applications, and are also important evidence for troubleshooting. According to experience, the log of our system should do a few things:

• Unified log format
• It is best to provide a unified log component, for example, we use the public logback-spring.xml component. If we need to modify certain characteristics of the log, such as adding APM logs, etc., we only need to change one point, and all the log status of our system will change.