[Yuanchuang Conference Preview] 1024 Programmers’ Day (two days before), meet at the Open Source China office, let’s talk about AI! >>>

1 Problem review

1.1 Problem description

In the performance test of the project, as the number of concurrency of related interfaces increases, the response time of the interface becomes longer, the interface throughput no longer increases, and the CPU usage of the application is high.

1.2 Analysis ideas

Who causes high CPU and blocks the increase of interface TPS? What is the call chain distribution of the interface response time? Is there any slow point?

1) The CPU of the application using flame graph analysis is as follows. The log4j2 log accounts for about 40% of the CPU. It is initially suspected to be a problem with log4j2.

2) Analysis of call chain

Checking the call chain through pfinder, we found that the interface took a total of 78ms, and there were no obviously slow calling methods or slow SQL. We first eliminated the code problems of the interface itself.

1.3 Preliminary conclusions

For log4j2 problems, you need to analyze the log-related configuration log4j2.xml in detail.

As you can see above, the nodes under the Loggers node in the log and the nodes are all printed synchronization logs. Synchronized logs mean that the business logic and log output statements of the program are both run in one thread. When there are too many logs, it blocks the operating efficiency of the business to a certain extent. Try changing to asynchronous logging:

Change to asynchronous log configuration: use AsyncLogger

1.4 Regression verification

After changing the synchronous log to an asynchronous log. For the same 10 concurrency, the TP99 of the interface dropped from 51ms to 23ms, the throughput TPS of the interface increased from 493 to 1078, and the application CPU dropped from 82% to 57%.

Perfect end. The problem is solved, but we still need to study the log4j2 log in detail.

1.5 Conclusion

Log4j2's use of asynchronous logging will greatly improve performance and reduce the impact on the application itself.
Fundamentally reduce unnecessary log output.

But how is log4j2 asynchronous log implemented and what is the difference between synchronous log? Why is asynchronous logging more efficient? It inspired me to learn some related knowledge of log4j2. Let me share it with you:

2 log4j2 log

2.1 Advantages of log4j2

Log4j2 is an upgraded version of log4j 1.x. It refers to some excellent designs of logback and fixes some problems, thus bringing some major improvements, mainly including:

Exception handling, in logback, exceptions in Appender will not be perceived by the application, but in log4j2, some exception handling mechanisms are provided.
Performance improvement. Compared with log4j 1 and logback, log4j2 has obvious performance improvement. There will be official test data later.
The automatic reload configuration refers to the design of logback. Of course, it will provide automatic refresh parameter configuration. The most practical thing is that we can dynamically modify the log level in production without restarting the application - which is very sensitive for monitoring. of.
No garbage mechanism. In most cases, log4j2 can use its designed garbage-free mechanism to avoid jvm gc caused by frequent log collection.

2.2 Log4J2 log classification

There are two ways to record logs in Log4j2: synchronous logs and asynchronous logs. Asynchronous logs can be divided into two ways: using AsyncAppender and using AsyncLogger. There are three logging modes using LOG4J2, fully asynchronous logging, mixed mode, and synchronous logging. The performance ranges from high to low. The more threads, the higher the efficiency. It can also avoid the occurrence of log stuck threads.

Performance comparison of synchronous and asynchronous logs:

2.3 Synchronization log

Use the synchronized log of log4j2 for log output. The log output statement and the business logic statement of the program will run in the same thread. That is, when outputting the log, you must wait for the log output statement to be executed before executing the subsequent business logic statement.

2.4 Asynchronous logs

When using asynchronous logs for output, the log output statements and business logic statements do not run in the same thread. Instead, a dedicated thread is used for log output operations. The main thread that processes the business logic can execute subsequent business without waiting. logic.

The biggest feature of log4j2 is asynchronous logging. Its performance improvement mainly benefits from asynchronous logging. Let’s take a look at how to use log4j2’s asynchronous logging.

There are two ways to implement asynchronous logs in Log4j2: AsyncAppender and AsyncLogger.
in:

AsyncAppender uses ArrayBlockingQueue to save log events that need to be output asynchronously;
AsyncLogger uses the Disruptor framework to achieve high throughput.

Note that these are two different implementation methods, with different reflections in design and source code.

2.4.1 AsyncAppender

AsyncAppender is implemented by referencing other Appenders. When log events arrive, another thread will be opened to process them. It should be noted that if an exception occurs during Appender, it will not be perceived by the application. AsyncAppender should be configured after the Appender it refers to. It is implemented by default using java.util.concurrent.ArrayBlockingQueue without the need for other external class libraries. When using this Appender, you need to be aware that blocking queues are susceptible to lock contention in a multi-threaded environment, which may have an impact on performance.

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="warn">
 <Appenders>
 <!--正常的Appender配置，此处配置的RollingFile会在下面AsyncAppender被通过name引用-->
 <RollingFile name="RollingFileError" fileName="${Log_Home}/error.${date:yyyy-MM-dd}.log" immediateFlush="true"
filePattern="${Log_Home}/$${date:yyyy-MM}/error-%d{MM-dd-yyyy}-%i.log.gz">
     <PatternLayout pattern="%d{yyyy-MM-dd 'at' HH:mm:ss z} %-5level %logger{36} : %msg%xEx%n"/>
       <ThresholdFilter level="error" onMatch="ACCEPT" onMismatch="DENY"/>
       <Policies>
        <TimeBasedTriggeringPolicy modulate="true" interval="1"/>
        <SizeBasedTriggeringPolicy size="10MB"/>
        </Policies>
 </RollingFile>
 <!--一个Appender配置完毕-->
 <!--异步AsyncAppender进行配置直接引用上面的RollingFile的name-->
 <Async name="Async">
        <AppenderRef ref="RollingFileError"/>
 </Async>
 <!--异步AsyncAppender配置完毕，需要几个配置几个-->
 </Appenders>

 <Loggers>
         <Root level="error">
          <!--此处如果引用异步AsyncAppender的name就是异步输出日志-->
         <!--此处如果引用Appenders标签中RollingFile的name就是同步输出日志-->
          <AppenderRef ref="Async"/>
</Root>
 </Loggers>
</Configuration>

2.4.2 AsyncLogger

AsyncLogger is the highlight of log4j2 and is also the officially recommended asynchronous method. It can make calling Logger.log return faster. AsyncLogger in Log4j2 uses the Disruptor framework internally.

Introduction to Disruptor
Disruptor is a high-performance queue developed by LMAX, a British foreign exchange trading company. The system developed based on Disruptor can support 6 million orders per second in a single thread.
Currently, many well-known projects including Apache Strom and Log4j2 have applied Disruptor to obtain high performance.
The internal core data structure of the Disruptor framework is RingBuffer, which is a lock-free ring queue.

Why is Disruptor so fast?

lock-free - uses CAS to achieve thread safety
Using cache line padding to resolve spurious sharing issues

There are two options for asynchronous logging: global asynchronous and hybrid asynchronous.

1) Global asynchronous

Global asynchronous means that all logs are recorded asynchronously. There is no need to make any changes to the configuration file. You only need to add a parameter when starting the jvm; this is the simplest configuration and provides the best performance.
Then add the log4j2.component.properties configuration file in the src/java/resources directory

Set asynchronous logging system properties

log4j2.contextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector

2) Mixed asynchronous

Mixed asynchronous means that you can use synchronous logs and asynchronous logs at the same time in your application, which makes the log configuration more flexible. Because the Log4j document also says that although Log4j2 provides a set of exception handling mechanisms that can cover most states, there are still a small number of special situations that cannot be fully handled. For example, if we record audit logs, then the official It is recommended to use synchronous logs. For other places that only record a program log, using asynchronous logs will greatly improve performance and reduce the impact on the application itself. The mixed asynchronous method needs to be implemented by modifying the configuration file and using AsyncLogger to mark the configuration.

Step 1: Add relevant jar packages to pom

<dependency>
  <groupId>com.lmax</groupId>
  <artifactId>disruptor</artifactId>
  <version>3.4.2</version>
</dependency>

Step 2: An example of mixed configuration of log4j2.xml synchronous log and asynchronous log (configuring AsyncLogger) is as follows:

<?xml version="1.0" encoding="UTF-8"?>
<!--日志级别以及优先级排序: OFF > FATAL > ERROR > WARN > INFO > DEBUG > TRACE > ALL -->
 <!--status="WARN" :用于设置log4j2自身内部日志的信息输出级别，默认是OFF-->
 <!--monitorInterval="30" :间隔秒数,自动检测配置文件的变更和重新配置本身-->
<configuration status="WARN" monitorInterval="30">
   <Properties>
       <!--1、自定义一些常量，之后使用${变量名}引用-->
       <Property name="logFilePath">log</Property>
       <Property name="logFileName">test.log</Property>
   </Properties>
   <!--2、appenders:定义输出内容,输出格式,输出方式,日志保存策略等,常用其下三种标签[console,File,RollingFile]-->
   <!--Appenders中配置日志输出的目的地
           console只的是控制台 system.out.println
           rollingFile 只的是文件大小达到指定尺寸的时候产生一个新的文件-->
   <appenders>
       <!--console :控制台输出的配置-->
       <console name="Console" target="SYSTEM_OUT">
           <!--PatternLayout :输出日志的格式,LOG4J2定义了输出代码,详见第二部分 %p 输出优先级，即DEBUG，INFO，WARN，ERROR，FATAL-->
           <PatternLayout pattern="[%d{HH:mm:ss:SSS}] [%p] - %l - %m%n"/>
       </console>
       <!--File :同步输出日志到本地文件-->
       <!--append="false" :根据其下日志策略,每次清空文件重新输入日志,可用于测试-->
       <File name="log" fileName="${logFilePath}/${logFileName}" append="false">
           <!-- 格式化输出：
           %d表示日期，%thread表示线程名，
           %-5level：级别从左显示5个字符宽度
           %thred: 输出产生该日志事件的线程名
           %class：是输出的类
           %L: 输出代码中的行号
           %M：方法名
           %msg：日志消息，
           %n是换行符
           %c: 输出日志信息所属的类目，通常就是所在类的全名 
           %t: 输出产生该日志事件的线程名 
           %l: 输出日志事件的发生位置，相当于%C.%M(%F:%L)的组合,包括类目名、发生的线程，以及在代码中的行数。举例：Testlog4.main(TestLog4.Java:10) 
           %p: 输出日志信息优先级，即DEBUG，INFO，WARN，ERROR，FATAL,

           2020.02.06 at 11:19:54 CST INFOcom.example.redistest.controller.PersonController 40 setPerson - 添加成功1条数据
           -->
           <!-- %class{36} 表示 class 名字最长36个字符 -->
           <PatternLayout pattern="%d{HH:mm:ss.SSS} %-5level %class{36} %L %M - %msg%xEx%n"/>
       </File>
       <!--
       关键点在于 filePattern后的日期格式，以及TimeBasedTriggeringPolicy的interval，日期格式精确到哪一位，interval也精确到哪一个单位.
       1） TimeBasedTriggeringPolicy需要和filePattern配套使用，由于filePattern配置的时间最小粒度如果设置是dd天，所以表示每一天新建一个文件保存日志。
       2） SizeBasedTriggeringPolicy表示当文件大小大于指定size时，生成新的文件保存日志。与%i配合使用-->

       <RollingFile name="RollingFileInfo" fileName="${sys:user.home}/logs/info.log"
                    filePattern="${sys:user.home}/logs/$${date:yyyy-MM}/info-%d{yyyy-MM-dd}-%i.log">
           <!--ThresholdFilter :日志输出过滤-->
           <!--level="info" :日志级别,onMatch="ACCEPT" :级别在info之上则接受,onMismatch="DENY" :级别在info之下则拒绝-->
           <ThresholdFilter level="info" onMatch="ACCEPT" onMismatch="DENY"/>
           <PatternLayout pattern="[%d{HH:mm:ss:SSS}] [%p] - %l - %m%n"/>
           <!-- Policies :日志滚动策略-->
           <Policies>
               <!-- TimeBasedTriggeringPolicy :时间滚动策略,默认0点小时产生新的文件,interval="6" : 自定义文件滚动时间间隔,每隔6小时产生新文件, modulate="true" : 产生文件是否以0点偏移时间,即6点,12点,18点,0点-->
               <TimeBasedTriggeringPolicyinterval="6" modulate="true"/>
               <!-- SizeBasedTriggeringPolicy :文件大小滚动策略-->
               <SizeBasedTriggeringPolicysize="100 MB"/>
           </Policies>
           <!-- DefaultRolloverStrategy属性如不设置，则默认为最多同一文件夹下7个文件，这里设置了20 -->
           <DefaultRolloverStrategy max="20"/>
       </RollingFile>

       <RollingFile name="RollingFileWarn" fileName="${sys:user.home}/logs/warn.log"
                    filePattern="${sys:user.home}/logs/$${date:yyyy-MM}/warn-%d{yyyy-MM-dd}-%i.log">
           <ThresholdFilter level="warn" onMatch="ACCEPT" onMismatch="DENY"/>
           <PatternLayout pattern="[%d{HH:mm:ss:SSS}] [%p] - %l - %m%n"/>
           <Policies>
               <TimeBasedTriggeringPolicy/>
               <SizeBasedTriggeringPolicy size="100 MB"/>
           </Policies>
       </RollingFile>
       <RollingFile name="RollingFileError" fileName="${sys:user.home}/logs/error.log"
                    filePattern="${sys:user.home}/logs/$${date:yyyy-MM}/error-%d{yyyy-MM-dd}-%i.log">
           <ThresholdFilter level="error" onMatch="ACCEPT" onMismatch="DENY"/>
           <PatternLayout pattern="[%d{HH:mm:ss:SSS}] [%p] - %l - %m%n"/>
           <Policies>
               <TimeBasedTriggeringPolicy/>
               <SizeBasedTriggeringPolicy size="100 MB"/>
           </Policies>
       </RollingFile>
   </appenders>
   <!--3、然后定义logger，只有定义了logger并引入的appender，appender才会生效-->
   <loggers>
       <!--过滤掉spring和mybatis的一些无用的DEBUG信息-->
       <!--Logger节点用来单独指定日志的形式，name为包路径,比如要为org.springframework包下所有日志指定为INFO级别等。 -->
       <logger name="org.springframework" level="INFO"></logger>
       <logger name="org.mybatis" level="INFO"></logger>
       <!-- Root节点用来指定项目的根日志，如果没有单独指定Logger，那么就会默认使用该Root日志输出 -->
       <root level="all">
           <appender-ref ref="Console"/>
           <appender-ref ref="RollingFileInfo"/>
           <appender-ref ref="RollingFileWarn"/>
           <appender-ref ref="RollingFileError"/>
       </root>
       <!--AsyncLogger :异步日志,LOG4J有三种日志模式,全异步日志,混合模式,同步日志,性能从高到底,线程越多效率越高,也可以避免日志卡死线程情况发生-->
       <!--additivity="false" : additivity设置事件是否在root logger输出，为了避免重复输出，可以在Logger 标签下设置additivity为”false”只在自定义的Appender中进行输出
-->
       <AsyncLogger name="AsyncLogger" level="trace" includeLocation="true" additivity="false">
           <appender-ref ref="RollingFileError"/>
       </AsyncLogger>
   </loggers>
</configuration>

2.5 Things to note when using asynchronous logs

There are some things you need to pay attention to when using asynchronous logs, as follows:

Do not use AsyncAppender and AsyncLogger at the same time. That is to say, do not use the Async logo and configure AsyncLogger at the same time when configuring Appender. This will not cause an error, but it will not have any benefit in improving performance.
Do not still use AsyncAppender and AsyncLogger when global synchronization is turned on. This has the same meaning as the previous one, that is, if you use asynchronous logs, AsyncAppender, AsyncLogger and global logs should not appear at the same time.
If it is not absolutely necessary, regardless of whether it is synchronous or asynchronous, set immediateFlush to false, which will greatly help improve performance.
If it is not really needed, do not print location information, such as HTML location, or %C or $class, %F or %file, %l or %location, %L or %line, %M or %method, in pattern mode. Wait, because Log4j needs to take a snapshot of the stack when printing logs to obtain this information, which is a huge loss in performance.

3 Summary

During the stress test, we try our best to explore and excavate the root causes of the problems, and constantly accumulate and summarize practical experience. Especially when using some open source components, you need to learn and understand the usage specifications and best practices in detail. If necessary, you can add performance testing to ensure that we meet our quality and performance requirements.

4 Reference

https://www.yisu.com/zixun/623058.html
https://www.jianshu.com/p/9f0c67facbe2
https://blog.csdn.net/thinkwon/article/details/101625124
https://zhuanlan.zhihu.com/p/386990511

Author: JD Logistics Liu Jiangbo Lu Yihao

Source: JD Cloud Developer Community Ziyuanqishuo Tech Please indicate the source when reprinting

Performance issues caused by log4j2 synchronized logs | JD Logistics Technical Team