Written in 2019-05-17
1, dynamic query condition is determined without making non-empty and cause 100% CPU cause an accident
Code Example:
//message为解析外部存管行返回的消息内容并封装成的对象
TppCgbankReq cgbankReq = new TppCgbankReq();
cgbankReq.setBusiness(getTrxCode());
cgbankReq.setOrderNo(message.getPreOrderNo());
List<TppCgbankReq> list = tppCgbankReqService.getAll(cgbankReq);Reasons:
First, in the normal (successful business) circumstances, message.getPreOrderNo () has a value, in the case of service failures, message.getPreOrderNo () is null;
two, tppCgbankReqService.getAll dynamic condition is a query method, the not to do orderNo sentenced empty process, once the results of the entire table trxCode qualifying record (hundreds of thousands) are out and query load to the heap;
by reason of the heap space long-term stress, frequent trigger full gc cause 100% CPU utilization .
Solution:
a, orderNo should take message.getHeaderOrderNo (), this time regardless of business success or failure message.getHeaderOrderNo () has a value;
Second, defensive coding principles, because the message parsing orderNo is made by an external system, still It may be present in an empty, plus the non-empty judgment.
2, Rabbitmq as consumers slow process due to high CPU and memory usage
reasons:
RabbitMQ use push mode, when a consumer process speed can not keep up speed producers, since no QoS restrictions, the message consumer side accumulation, leading consumers to stack space occupied rise, full gc also followed the rise, cpu and memory dual tension.
Solution:
Because no way in a short time will increase the speed of consumers and producers to speed matching, so the set qos, limit the number of messages each made of 10 reference: channel.basicQos (10) ;.
3, too many threads lead to too high cpu usage
code examples:
//请求开始
//主业务操作
threadPool.execute(new Runnable() {
@Override
public void run() {
//生成合同
}
});
//请求响应The reason:
a multi-threaded model is unreasonable, although the use of thread pools, some non-core businesses from excessive threading, cpu usage is too high;
Second, the main business process is completed, use the thread pool asynchronous generation contract, Since the contract was generating IO-intensive operations, which at its peak threadPool soon reach the maximum number of threads;
Third, because threadPool set value is too large, since too many threads lead to high CPU.
Solution:
First, short-term adjustment threadPool value of appropriate size;
Second, long-term non-core asynchronous tasks using MQ asynchronous architecture, using asynchronous consumption threading;
4, causing slow SQL database CPU rises, other SQL is suspended, the system short unavailable.
The reason:
Reports slow Ali cloud RDS database sql cause 100% CPU
Solution:
1, KILL out the task in a timely manner;
2, optimized slow SQL;
3, separate read and write, report commuting library.
After 5, Netty wrote stable TCP services running more than a month, suddenly 100% CPU utilization
reasons:
a, ByteBuf not caused by external manual release heap memory leaks slowly accumulated to a certain extent, due to the lack of space, cause and cause full gc 100% CPU utilization;
two, Netty some older indeed there is a risk of other causes of memory leaks.
Solution:
1, Netty upgrade to the latest stable version;
2, take the initiative to release ByteBuf created by hand;
3, plus a log pile outside memory and print reports.
6, the connection pool is set too small, resulting in the system response timeout reasons:
First, the beginning of the CPU and the memory is normal, the system response timeout, log large database connection timeout error;
Second, since the core operational interface is a synchronous architecture, the assets end large number of requests backlog and frequent overtime, leading to capital end users can not use the system properly.
Solution:
First, adjust the connection pool, turn up the maximum number of connections;
II [architecture adjust] assets end funds ends separated from each other;
III [architecture for Adjustment core business operations modified from synchronous to asynchronous architecture, the introduction of MQ peak cut process.
Summarize
general, cause high CPU utilization has:
1, FULL GC;
2, excessive Thread, frequent switching thread will consume a lot of CPU resources;
3, irrational cycle, including infinite loop, too many cycles and so on;
4, some third-party lib and so on.
In general, cause high memory usage are:
1, a query too much data from the database, this can be checked with the slow sql initiated;
2, after the release of resources to forget or unaccounted release after throwing an exception;
3 , a cycle is created excessive object;
4, use the global collection class data cache excessive.
Online troubleshooting process is to combine top, process jstat, jmap, jstack command to verify the above conjecture if the line can not locate the problem quickly, timely Dump file to a local, then MemoryAnalyzer, Jvisualvm and other analysis.