Quoted from: http://zhanghua.1199.blog.163.com/blog/static/464498072012112142121681/
1. log4j has become an indispensable part of large-scale systems. log4j can easily help us output the information to be printed anywhere in the program, which is convenient for us to analyze and locate the problem in the debugging stage and the formal operation stage of the system. Due to the different log levels, there is also a big gap in the performance impact of the system. The higher the log level, the higher the performance. 2. log4j is mainly divided into four levels: error, warn, info, and debug, which are also the four most used. The log level increases from left to right. 3. The impact of log4j on system performance is mainly reflected in the following aspects: a. The destination of log output, the speed of output to the console is slower than the speed of output to the file system. b. Different log output formats will also have an impact on performance. For example, the output speed of SimpleLayout is faster than that of PatternLayout. You can use a simple output layout format to output log information as needed. c. The lower the log level, the more log content is output, which has a great impact on the system performance. d. Different log output methods also have a certain impact on the system performance. The asynchronous output method has higher performance than the synchronous output method. e. Printing a log content every time a log output event is received is lower than when the log content reaches a certain size. 4. For points 4 and 5 in the impact of the above points on system performance, configure the log configuration file as follows: a. Set the log cache and cache size
- log4j.appender.A3.BufferedIO=true
- #Buffer unit is byte, the default is 8K, the default size of IO BLOCK is also 8K
- log4j.appender.A3.BufferSize=8192
The above configuration shows that when the log content reaches 8k, the log will be output to the log output destination.
b. Set the log output to asynchronous mode
- <appender name="DRFOUT"class="org.apache.log4j.DailyRollingFileAppender">
- <param name="File" value="logs/brws.log" />
- <param name="Append" value="true" />
- <param name="DatePattern" value="yyyy_MM_dd'.'" />
- <layoutclass="org.apache.log4j.PatternLayout">
- <param name="ConversionPattern" value="%d [%t] %-5p %l %x - %m%n" />
- </layout>
- </appender>
- <appender name="ASYNCOUT"class="org.apache.log4j.AsyncAppender">
- <param name="BufferSize" value="512" />
- <appender-ref ref="DRFOUT" />
- </appender>
Synchronization: each thread directly obtains the output stream for output (no synchronization between threads is required).
Asynchronous situation: 1. Each thread writes the log to the cache, and continues to perform the following tasks (this is asynchronous)
2. The log thread finds that it needs to record the log, and the cache is exclusive (at the same time, each thread waits, and each thread is blocked at this time live), take the log information from the cache, obtain the output stream for output, and unlock the cache (each thread receives a reminder and can continue to write the log).
As we all know, disk IO operations, network IO operations, JDBC operations, etc. are very expensive At times, the main performance bottleneck of log output is when writing files, writing networks, and writing JDBC. The log must be recorded, but it must be recorded in an asynchronous way. Only by separating these time-consuming operations from the main thread can real performance be improved, and the asynchronous method can only be used when the synchronization overhead between threads is less than the time-consuming operation. It really works!
Now let's take a look at these several ways of recording logs: a. Recording logs to local files is also written to local files. Log4j itself has a buffer for processing storage, and using asynchronous methods does not necessarily improve performance (mainly How to configure the cache size); and the synchronization overhead between threads is very large! Therefore, it is not recommended to use asynchronous mode when using local file logging. b. Logging to JMS JMS itself supports asynchronous messages. If the overhead of JMS message creation is not considered, it is not recommended to use asynchronous methods. c. Record the day to SOCKET Send the log through Socket, pure network IO operation does not require feedback, so it will not take time d. Recording the log to the database It is well known that JDBC is the most time-consuming among several methods: network, disk, Database transactions make JDBC operations unusually time-consuming, and it is a good choice to use asynchronous storage here. e. Logging to SMTP is the same as JDBC 5. Working principle of asynchronous output log
AsyncAppender uses the producer-consumer model to asynchronously send Logging Events to the corresponding Appender.
a. Producer: The real-time thread of the system that applies Log4j externally, and transmits Logging Event into AsyncAppender in real time. b. Transit: Buffer and DiscardSummary c. Consumer: Dispatcher thread and appenders
work principle: 1) Logging Event enters AsyncAppender, AsyncAppender The append method will be called. In the append method, the logging event will be filled into the Buffer. When the consumption capacity is not as good as the production capacity, AsyncAppender will put the Logging Event that exceeds the capacity of the Buffer into the DiscardSummary, as the consumption speed cannot keep up with the production speed. , a solution for the overflow processing of the transit buffer. 2) AsyncAppender has a thread class Dispatcher, which is a simple thread class that implements the Runnable interface. It is the background thread of AsyncAppender. The work that the Dispatcher has to do is: ① Lock the Buffer and block other threads that want to operate on the Buffer. ② Check whether the capacity of the Buffer is full. If it is full, take out all the Logging Events in the Buffer, and empty the Buffer and DiscardSummary; if it is not full, wait for the Buffer to be filled with Logging Events, and then notify the Disaptcher thread. ③ Hand over all the Logging Events taken out to the corresponding appender for subsequent log information push.
The above are the two key points of the AsyncAppender class: the append method and the Dispatcher class. Through these two key points, the function of asynchronously pushing log information is realized, so that if a large number of Logging Events enter the AsyncAppender, the log information can be handled with ease.
************************************************************************************************************************************************************************
************************************************************************************************************************************************************************
Not long ago, the monitoring function was completed in the system. The monitoring system has a large amount of information. Every click of the user on the page will generate a record. The amount of logs downloaded every day is more than 2G. I use log4j to put these monitoring records in the log. , and then perform asynchronous processing, but even so, logging will cause frequent access to disk IO, and IO is usually the bottleneck of the system. So some tuning of log4j configuration becomes necessary. Below is the log4j configuration in my system:
log4j.logger.monitorLogger=INFO,monitorAppender
log4j.additivity.monitorLogger=false
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d (%F:%L) %-5p %c - %m%n
log4j.appender.fileout=org.apache.log4j.DailyRollingFileAppender
log4j.appender.fileout.File=logs/server_log.txt
log4j.appender.fileout.layout=org.apache.log4j.PatternLayout
log4j.appender.fileout.layout.ConversionPattern=%d [ %t ] (%F:%L) %-5p %c - %m%n
log4j.appender.monitorAppender=org.apache.log4j.DailyRollingFileAppender
log4j.appender.monitorAppender.File=mtlogs/mt_log.txt
log4j.appender.monitorAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.monitorAppender.layout.ConversionPattern=%m%n
log4j.appender.monitorAppender.DatePattern='.'yyyy-MM-dd-HH
log4j.appender.monitorAppender.BufferedIO=true
#Buffer单位为字节，默认是8K
log4j.appender.monitorAppender.BufferSize=8192
1）log4j.additivity.monitorLogger=false
这个选项用于控制监控logger的日志不会输出到rootlogger，否则无论会产生许多重复的数据，同时也会影响性能；
2）log4j.appender.monitorAppender.DatePattern='.'yyyy-MM-dd-HH
这个选项用于告诉DailyRollingFileAppender每小时输出日志，而不是默认的一天输出一次，因为监控日志的数据量很巨大，如果以天为单位输出，日志文件会很大（G级），而且再处理会很耗时。
其他一些输出选项还有：
1)'.'yyyy-MM: 每月
2)'.'yyyy-ww: 每周
3)'.'yyyy-MM-dd: 每天
4)'.'yyyy-MM-dd-a: 每天两次
5)'.'yyyy-MM-dd-HH: 每小时
6)'.'yyyy-MM-dd-HH-mm: 每分钟
3）log4j.appender.monitorAppender.BufferedIO=true
log4j.appender.monitorAppender.BufferSize=8192
这 个选项用于告诉log4j输出日志的时候采用缓冲的方式，而不是即时flush方式，并且设定了缓冲为8K，8K是默认值，可以根据日志输出的情况来修 改。这个选项很重要，在测试中发现，当并发访问很高，例如每一秒100个并发以上，使用缓存跟不使用缓冲差距很大。具体数字我这里就不列出来了。
另外我想说的是，log4j输出缓冲日志是以8K为单位的，因为磁盘的一个block为8K，这样可以减少碎片，也就是说假设你设置缓存为18K，log4j在16K（8K*2)的时候就会输出），而不是18K。
4）组装输出内容之前可对logger的输出级别先进行判断而不要完全依赖log4j控制，因为组装输出日志内容也是要损耗效率的。
//若log4j并未开启info级日志记录，直接返回
if(!monitorLogger.isInfoEnabled()){
return;
}
StringBuilder log = new StringBuilder();
logSql.append(logPk+" ");
...
5）使用异步输出 org.apache.log4j.AsyncAppender，异步输出必须使用xml方式配置才能支持，我把上面properties形式的配置文件用xml表达一下：
< log4j:configuration debug ="true" >
< appender name ="stdout"
class ="org.apache.log4j.ConsoleAppender" >
< layout class ="org.apache.log4j.PatternLayout" >
< param name ="ConversionPattern" value ="%d (%F:%L) %-5p %c - %m%n" />
</ layout >
</ appender >
< appender name ="fileout"
class ="org.apache.log4j.DailyRollingFileAppender" >
< layout class ="org.apache.log4j.PatternLayout" >
< param name ="ConversionPattern" value ="%d [%t] (%F:%L) %-5p %c - %m%n" />
</ layout >
< param name ="File"
value ="logs/server_log.txt" />
</ appender >
< appender name ="monitorAppender"
class ="org.apache.log4j.DailyRollingFileAppender" >
< layout class ="org.apache.log4j.PatternLayout" >
< param name ="ConversionPattern" value ="%m%n" />
</ layout >
< param name ="DatePattern" value ="'.'yyyy-MM-dd-HH" />
< param name ="File" value ="mtlogs/mt_log.txt" />
< param name ="BufferedIO" value ="true" />
<!-- 8K为一个写单元 -->
< param name ="BufferSize" value ="8192" />
</ appender >
< appender name ="async" class ="org.apache.log4j.AsyncAppender" >
< appender-ref ref ="monitorAppender" />
</ appender >
< root >
< priority value ="error" />
< appender-ref ref ="stdout" />
< appender-ref ref ="fileout" />
</ root >
< category name ="com.danga.MemCached" >
< priority value ="error" />
< appender-ref ref ="fileout" />
</ category >
< category name ="com.opensymphony" >
< priority value ="error" />
< appender-ref ref ="fileout" />
</ category >
< category name ="monitorLogger" additivity ="false" >
< priority value ="info" />
< appender-ref ref ="async" />
</ category >
</ log4j:configuration >
The red part of the configuration is used to support asynchronous output. In the process of testing with jmeter, it is found that the asynchronous method is used, and the work is not very stable. The performance improvement is also not significant. So it was not adopted in the end.
try {
in = Log4jConfigLocator.class .getResourceAsStream ( fileName);
if (fileName.endsWith(".xml")){
// Load configuration file in XML format
Document doc = DocumentBuilderFactory.newInstance().newDocumentBuilder ().parse(in);
DOMConfigurator.configure(doc.getDocumentElement());
} else {
// Load the configuration file in properties format
Properties props = new Properties(); props.load(in); PropertyConfigurator.configure( props); }