table of Contents
1. Background
There are two ways to record logs in Log4j 2: synchronous log and asynchronous log. Asynchronous log can be divided into two ways: using AsyncAppender and using AsyncLogger.
2.Sync log in Log4j2
The so-called synchronous log, that is, when outputting the log, you must wait for the log output statement to be executed before executing the following business logic statement.
Let's use an example to understand the synchronization log in Log4j2, and use this to explore the entire log output process.
The log4j2.xml configuration is as follows:
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="debug" name="MyApp" packages="">
<!--全局Filter-->
<ThresholdFilter level="ALL"/>
<Appenders>
<Console name="Console" target="SYSTEM_OUT">
<PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n" />
</Console>
<RollingFile name="RollingFile" fileName="logs/app.log"
filePattern="logs/app-%d{yyyy-MM-dd HH}.log">
<!--Appender的Filter-->
<ThresholdFilter level="info" onMatch="ACCEPT" onMismatch="DENY"/>
<PatternLayout>
<Pattern>%d %p %c{1.} [%t] %m%n</Pattern>
</PatternLayout>
<Policies>
<SizeBasedTriggeringPolicy size="500MB"/>
</Policies>
</RollingFile>
</Appenders>
<Loggers>
<Logger name="com.meituan.Main" level="trace" additivity="false">
<!--Logger的Filter-->
<ThresholdFilter level="debug"/>
<appender-ref ref="RollingFile"/>
</Logger>
<Root level="debug">
<AppenderRef ref="Console"/>
</Root>
</Loggers>
</Configuration>The java code is as follows:
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
public class Main {
public static void main(String args[]) {
Logger logger = LogManager.getLogger(Main.class);
Person person = new Person("Li", "lei");
logger.info("hello, {}", person);
}
private static class Person {
private String firstName;
private String lastName;
public Person(String firstName, String lastName) {
this.firstName = firstName;
this.lastName = lastName;
}
public String toString() {
return "Person[" + firstName + "," + lastName + "]";
}
}
}Using the above configuration, when we run the program, the following log will be added to logs/app.log.
2017-09-13 19:41:00,889 INFO c.m.Main [main] hello, Person[Li,lei]What happened when logger.info was running? How is the log information output to app.log?
The detailed process of log output in Log4j2 is as follows:
1. First use the global Filter to filter the log events.
The log level in Log4j2 is divided into 8 levels, the priority from high to low is OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE, ALL.
The Level of the global Filter is ALL, which means that all levels of logs are allowed to be output. logger.info() requests to output INFO-level logs, passed.
2. Use Logger's Level to filter log events.
The Logger's Level is TRACE, which means that logs of TRACE level and above are allowed to be output. logger.info() requests to output INFO-level logs, passed.
3. Generate the log output Message.
Use placeholders to output the log. The output statement is in the form of logger.info("increase {} from {} to {}", arg1, arg2, arg3). The content at the placeholder {} will be used in the final output The string padding of arg1, arg2, and arg3.
log4j2 uses Object[] to save parameter information. At this stage, Object[] is converted to String[] to generate a Message containing the output pattern string "increase {} from {} to {}" and the parameter array String[], which is Prepare for subsequent log formatted output.
4. Generate LogEvent.
LogEvent contains loggerName (log output), level (log level), timeMillis (log output time), message (log output content), threadName (thread name) and other information.
In the above program, the attribute values of the generated LogEvent are loggerName=com.meituan.Main, Level=INFO, timeMillis=1505659461759, and message is the Message created in step 3, threadNama=main.
5. Use the Filter configured by Logger to filter log events.
The Level of the Filter configured by the Logger is DEBUG, which means that DEBUG and above logs are allowed to be output. logger.info() requests to output INFO-level logs, passed.
6. Use the Filter configured by the Appender corresponding to the Logger to filter log events.
The INFO level log onMatch=ACCEPT of the Filter configuration of the Appender configuration indicates that the INFO level log is allowed to be output. logger.info() requests to output INFO-level logs, passed.
7. Determine whether the rollover needs to be triggered.
This step is not a necessary step for log output. If the configured Appender is an Appender that does not require rollover, there is no such step.
Because RollingFileAppender is used and a rollover trigger strategy based on file size is configured, at this stage, it is determined whether a rollover needs to be triggered. The judgment method is whether the current file size reaches the specified size, and if it reaches, the rollover operation is triggered. For the rollover of RollingFileAppender in Log4j2 , please refer to the file rolling update mechanism of RollingFile in Log4j2 .
8.PatternLayout formats LogEvent to generate a string that can be output.
The meaning of the Pattern and each parameter configured in the above log4j2.xml file is as follows:
<Pattern>%d %p %c{1.} [%t] %m%n</Pattern>| parameter |
significance |
|---|---|
|
Date format, the default format is 2012-11-02 14:34:02,781 |
|
Log level |
|
%c represents the Logger name, {1.} represents the accuracy. If the Logger name is org.apache.commons.Foo, oacFoo will be output. |
|
The name of the thread that processes the LogEvent |
|
Log content |
|
Line separator. "\n" or "\r\n". |
In this step, PatternLayout will use various Converters to convert the relevant information of LogEvent according to the pattern of Pattern, and finally concatenate it into a log string that can be output.
For example, DatePatternConverter formatted and converted the log output time of LogEvent; LevelPatternConverter formatted and converted the log level information of LogEvent; LoggerPatternConverter formatted and converted the Logger name of LogEvent; MessagePatternConverter formatted and converted the log output content of LogEvent, etc.
After conversion by various Converters, the information of LogEvent is formatted into a string in a specified format.
9. Use OutputStream to output the log to a file.
Serialize the log string into a byte array, and use the byte stream OutoutStream to output the log to a file. If immediateFlush is configured as true, the output log can be observed by opening app.log.
3. Asynchronous log in Log4j2
Use log4j2's synchronization log for log output. The log output statement and the business logic statement of the program will run in the same thread, as in the above example, the thread name displayed in the printed log is main, which is in the same thread as the business logic statement (This sentence is wrong. The thread name in LogEvent is not the thread that outputs LogEvent, but the thread that generates LogEvent. The thread that outputs LogEvent and the thread that generates LogEvent may not be the same thread!) .
When the asynchronous log is used for output, the log output statement and the business logic statement are not run in the same thread, but a dedicated thread is used for log output operations. The main thread that processes the business logic can perform subsequent business without waiting. logic.
There are two ways to implement asynchronous logging in Log4j2: AsyncAppender and AsyncLogger.
Among them, AsyncAppender uses ArrayBlockingQueue to save log events that need to be output asynchronously; AsyncLogger uses the Disruptor framework to achieve high throughput.
3.1 AsyncAppender
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="warn">
<Appenders>
<RollingFile name="MyFile" fileName="logs/app.log">
<PatternLayout>
<Pattern>%d %p %c{1.} [%t] %m%n</Pattern>
</PatternLayout>
<SizeBasedTriggeringPolicy size="500MB"/>
</RollingFile>
<Async name="Async">
<AppenderRef ref="MyFile"/>
</Async>
</Appenders>
<Loggers>
<Root level="error">
<AppenderRef ref="Async"/>
</Root>
</Loggers>
</Configuration>The above is a typical configuration using AsyncAppender. After configuring AsyncAppender, the operation of writing log events to files will be executed in a separate thread.
Common parameters of AsyncAppender
| parameter name |
Types of |
Description |
|---|---|---|
| name | String | The name of Async Appender. |
| AppenderRef | String | Multiple Appender names can be configured for asynchronous calls. |
| blocking | boolean | The default is true. If true, the appender will wait until the queue is free; if false, when the queue is full, the log event will be discarded. (If the error appender is configured, the log events to be discarded will be processed by the error appender) |
| bufferSize | integer | The maximum number of log events that can be stored in the queue. The default is 128. (128 in the source code, 1024 on the official website of Log4j2, the official website information is incorrect) |
For other parameters of AsyncAppender , please refer to Log4j2 's detailed introduction to AsyncAppender .
Each Async Appender maintains an ArrayBlockingQueue internally and creates a thread to output log events. If multiple AppenderRefs are configured, the corresponding Appenders will be used for log output.
3.2 AsyncLogger
The Disruptor framework is used internally in AsyncLogger in Log4j2.
Introduction to Disruptor
Disruptor is a high-performance queue developed by British foreign exchange trading company LMAX. The system based on Disruptor can support 6 million orders per second in a single thread.
At present, many well-known projects including Apache Strom and Log4j2 have applied Disruptor to obtain high performance.
The core data structure inside the Disruptor framework is RingBuffer, which is a lock-free ring queue.
A single thread can process 6 million orders per second. Why is Disruptor so fast?
a.lock-free-CAS is used to achieve thread safety
ArrayBlockingQueue uses locks to achieve concurrency control. When get or put, the current accessing thread will be locked. In the case of a large number of concurrency with multiple producers and multiple consumers, there will be performance losses due to lock competition, thread switching, etc.
Disruptor realizes concurrent access to RingBuffer by multiple producers and consumers through CAS. CAS is equivalent to optimistic lock, and its performance is better than that of Lock.
b. Use cache line padding to solve the problem of false sharing
In the computer architecture, the access speed of the memory is far lower than the running speed of the CPU. Between the memory and the CPU, the Cache is added. The CPU first accesses the data in the Cache, and the CaChe misses the data before accessing the data in the memory.
Pseudo-sharing: Cache is stored in units of cache lines. When multiple threads modify independent variables, if these variables share the same cache line, they will unintentionally affect each other's performance.
For in-depth analysis of false sharing, please refer to the article " False sharing, the performance killer of concurrent programming ".
AsyncLogger
How to log output of Log4j2 asynchronous log, we also start from an example to explore the asynchronous log of Log4j2.
The log4j2.xml configuration is as follows:
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="debug" name="MyApp" packages="">
<Appenders>
<Console name="Console" target="SYSTEM_OUT">
<PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n" />
</Console>
<RollingFile name="RollingFile" fileName="logs/app.log"
filePattern="logs/app-%d{yyyy-MM-dd HH}.log">
<PatternLayout>
<Pattern>%d %p %c{1.} [%t] %m%n</Pattern>
</PatternLayout>
<Policies>
<SizeBasedTriggeringPolicy size="500MB"/>
</Policies>
</RollingFile>
<RollingFile name="RollingFile2" fileName="logs/app2.log"
filePattern="logs/app2-%d{yyyy-MM-dd HH}.log">
<PatternLayout>
<Pattern>%d %p %c{1.} [%t] %m%n</Pattern>
</PatternLayout>
<Policies>
<SizeBasedTriggeringPolicy size="500MB"/>
</Policies>
</RollingFile>
</Appenders>
<Loggers>
<AsyncLogger name="com.meituan.Main" level="trace" additivity="false">
<appender-ref ref="RollingFile"/>
</AsyncLogger>
<AsyncLogger name="RollingFile2" level="trace" additivity="false">
<appender-ref ref="RollingFile2"/>
</AsyncLogger>
<Root level="debug">
<AppenderRef ref="Console"/>
<AppenderRef ref="RollingFile"/>
</Root>
</Loggers>
</Configuration>The java code is as follows:
public class Main {
public static void main(String args[]) {
Logger logger = LogManager.getLogger(Main.class);
Logger logger2 = LogManager.getLogger("RollingFile2");
Person person = new Person("Li", "lei");
logger.info("hello, {}", person);
logger2.info("good bye, {}", person);
}Two AsyncLoggers are configured in the above log4j2.xml, the names are com.meituan.Main and RollingFile2.
In addition, two loggers are used in the main method to output two logs.
In the startup phase of loading log4j2.xml, if it is detected that AsyncRoot or AsyncLogger is configured, a disruptor instance will be started.
In the above program, the main thread is the producer, and the EventProcessor thread is the consumer.
Producer produces news
When running to output statements similar to logger.info and logger.debug, put the generated LogEvent into the RingBuffer.
Consumer consumption news
If there is a LogEvent in the RingBuffer that needs to be processed, the EventProcessor thread retrieves the LogEvent from the RingBuffer and calls the Appender associated with the Logger to output the LogEvent (the specific output process is the same as the synchronization process, and also requires filter filtering, PatternLayout formatting and other steps).
If there is no LogEvent in the RingBuffer to be processed, the EventProcessor thread will be in a waiting blocking state (default strategy).
It should be noted that although multiple AsyncLoggers are configured in log4j2.xml, not each AsyncLogger corresponds to a processing thread, but only one EventProcessor thread for asynchronous log processing.
4. Summary
|
|
Log output method |
|---|---|
| sync | Print logs synchronously. Log output and business logic are in the same thread. After the log output is completed, subsequent business logic operations can be performed |
| Async Appender | Print logs asynchronously, using ArrayBlockingQueue internally, and create a thread for each AsyncAppender to process log output. |
| Async Logger | Print logs asynchronously, using a high-performance concurrency framework Disruptor, and create a thread to process log output. |