Performance and stress testing

Performance and stress testing

One, performance monitoring

1. JVM memory model

• Program Counter Register:

  • Records the address of the bytecode instruction of the virtual machine being executed

  • This memory area is the only area that does not specify any OutOfMemoryError in the JAVA virtual machine specification

• Virtual machine stack VMStack:

2. Heap

All 对象实例and 数组all 在堆上分配. 堆It is the main area managed by the garbage collector, also known as " GC堆"; it is also the place we consider most for optimization. The heap can be subdivided into:

• 新生代
  • Eden空间
  • From Survivor空间
  • To Survivor空间
• 老年代
• 永久代/元空间
  • The permanent generation before Java8 is managed by jvm, and the metaspace after Java8 uses physical memory directly. Therefore, by default, the size of the metaspace is limited only by local memory.

Garbage collection

Starting from Java 8, Hotspot has completely removed the Permanent Generation (Permanent Generation) and replaced it with a new regional unary space ( MetaSpace)

3、jconsole与jvisualvm

Two small tools of JDK jconsole, jvisualvm(upgraded version of jconsole); Start by command line to monitor local and remote applications. Remote application requires configuration

jconsole：

1. What can jvisualvm do

监控内存泄露，跟踪垃圾回收，执行时内存、cpu分析，线程分析

启动jvisualvm：


监控内存泄露，跟踪垃圾回收，执行时内存、cpu分析：

线程分析：

• 运行: Running
• 休眠: sleep
• 等待: wait
• 驻留: Idle threads in the thread pool
• 监视: Blocked thread, waiting for lock

2. Install plug-in to view GC

• cmdstart up jvisualvm
• 工具→插件
  
• If the 503 error is resolved:
  • Open the URL https://visualvm.github.io/pluginscenters.html
  • cmd Check your own jdk version and find the corresponding
    

You need to restart after installing the plugin jvisualvm

4. Monitoring indicators

Development environment configuration:

实体机： Development and service operation

虚拟机： Run middleware and database, memory 3G, 1 core

CentOS-7、Docker、redis、mysql:5.7、elasticsearch:7.4.2、kibana:7.4.2、nginx:1.10

1. Middleware indicators

nginx：

• Computational
• Consume cpu
  
  
  

Gateway：

• Computational
• Compare CPU consumption
  
  
  
  

2. Database indicators

• The less time it takes for SQL, the better, usually in microseconds.
• The higher the hit rate, the better, under normal circumstances it cannot be lower than 95%
• The lower the number of lock waits, the better, and the shorter the wait time, the better.

Service stress test:

简单服务(不请求数据库)：

Gateway+简单服务(不请求数据库)：

全链路(不请求数据库)：

首页一级菜单渲染(不过网关)：

三级分类数据获取：

首页全量数据获取：

Nginx+Gateway：

Summary of results:

Pressure test content	Number of threads	Throughput/s	90% response time	99% response time
Nginx	50	6564	4ms	172ms
Gateway	50	12356	6	19
Simple service	50	14177	4	65
Home page first-level menu rendering (thymeleaf cache is not enabled)	50	515	130	200
Home menu level rendering (open thymeleaf cache, 优化数据库, 关日志)	50	607/700	101/86	118/151
Three-level classification data acquisition	50	5/19(加索引)	11141/2764	11366/2956
Three-level classification data acquisition (business logic optimization)	50	200	398	644
Three-level classification data acquisition (using redis cache)	50	684	95	128
Home page full data acquisition (before optimization // 优化后after dynamic and static separation)	50	20/22/550	3426/2522/103	4272/3133/804
Nginx+Gateway	50
Gateway+ simple service	50	4919	21	47
Full link	50	1439	50	80

简单服务: The reason for the slowness is DB and thymeleaf rendering. Through comparison, we can see:

• 首页一级菜单渲染The throughput performance improvement before and after the thymeleaf cache is turned on 17.86%is quite obvious.
• 首页一级菜单渲染During 关闭debug日志and 给where条件列加索引after the visit , the DB request time was 6-1085msincreased from the original to the 2-8msspeed 3-135倍.
• 首页一级菜单渲染In 开启thymeleaf缓存, 优化数据库（加索引）, 关日志before and after comparison can be seen to enhance the throughput of about 35.92%.

三级分类数据获取: The main reason for the slowness is the DB. There is a process of traversal query, multiple requests for services, frequent database IO, and the development environment has not closed the debug level log; currently the service has not used the database connection pool, and the database connection pool can be configured for optimization. In addition For three-level classification of data that does not change frequently, you can choose to use redis to cache it to improve the efficiency of access.

首页全量数据获取: The reason for the slowness 首页一级菜单渲染can be known by comparison. The main reason for the slowness is the slow loading of static resources. At present, our service has not done the request separation operation of dynamic and static resources. You can use nginx as a static resource server for optimization.

Conclusion and optimization:

• 中间件越多，性能损失越大，大多都损失在网络交互。可以优化中间件配置，使用更快的硬件资源
• 业务：
  • DB (MySQL service optimization, index optimization)
  • The rendering speed of the template (the cache is turned off during the development of thymeleaf, and the cache is turned on after the service is online; template rendering requires CPU to calculate, so a better CPU can be used; server memory)
  • Static resources

5. JVM analysis & tuning

1. Several commonly used tools

2. Command example

3. Tuning items

1. Performance pressure test-optimization-nginx dynamic and static separation

1) Upload static resources to nginx server

• mydata/nginx/html/Create static/ folder under / path
• src\main\resources\staticUpload all the static resource files in the folder in the local code path to the nginx server.

2) Modify the nginx configuration file and restart

• Nginx modified from the conf.ddirectory default.confcopied files pafcmall.conffile, location / { add a static resource configuration file access paths before:

# 配置静态资源访问路径 
location /static/ {
    
    
	root   /usr/share/nginx/html;
}  

• Restart nginx:docker restart nginx

3) Modify the local code, delete src\main\resources\staticthe static resource file under the folder in the local code path , and modify index.htmlthe reference path of the static resource in the file.

• js：

<script src="/static/index/js/xxx.js" type="text/javascript" charset="utf-8"></script>

• css：

<link rel="stylesheet" href="/static/index/css/xxx.css">

• img：

<img src="/static/index/img/xxx.jpg" />

2. Performance stress test-optimization-simulate online application memory crash and downtime

1. JVM tuning-setting commodity service startup parameters for comparison, taking 50 threads as an example:

1）-Xmx100m

2）-Xmx512m

Set the maximum size of heap memory to 512m

3）-Xmx1024m -Xms1024m -Xmn512m

Set the maximum and minimum size of the heap memory to 1024m; and set the young generation (Eden+S0+S1) to 512m to reduce the frequent Minor GC of the garbage collector

2. Simulate online application memory crash and downtime

Set the number of threads of jmeter to 200, and set the maximum heap memory of the service to to -Xmx100mperform a simulation test
服务不可用:

OOM异常：

3. Performance stress measurement-optimization-optimization of three-level classification data acquisition

Change the database multiple queries into one query, reduce the frequent IO of the database, you can see 50 threads, when the Xmx is 100m, the throughput of the service reaches 200, which is an improvement in performance compared to the index 19 in the previous icon About 10倍.

If you want to further enhance performance, we need to consider 使用数据库连接池, 使用缓存in the circumstances, this will be introduced to later.

2. Stress test

压力测试：

Stress testing examines what the system can withstand under the current hardware and software environment 最大负荷and helps identify system bottlenecks. Stress testing is for the system to be online 处理能力and 稳定性maintain in one 标准范围内, so that you can know it well.

Using stress testing, we hope to find many errors that are more difficult to find with other testing methods. There are two types of errors: 内存泄漏, 并发与同步.

Effective stress testing the system applies the following key conditions 重复, 并发, 量级, 随机变化.

1. Performance indicators

• Response Time (Response Time: RT): Response time refers to the time it takes for the entire process from the moment the client initiates a request to the end of the client receiving the response returned from the server.

• HPS (Hits Per Second): The number of hits per second, in units of seconds.

• TPS (Transaction per Second) : The number of transactions processed by the system per second, in units of transactions per second.

• QPS (Query per Second) : The number of queries processed by the system per second, the unit is times/second.

  For Internet services, if some services have and only have one request connection, then TPS=QPS=HPS. In general, TPS is used to measure the entire business process, QPS is used to measure the number of interface queries, and HPS is used to indicate the number of server requests. Click the request.

• Regardless of TPS, QPS, or HPS, this indicator is a very important indicator to measure the processing capacity of the system. The bigger the better, based on experience, in general:

  金融行业: 1000TPS~50000TPS, Excluding Internet-based activities

  保险行业: 100TPS~100000TPS, Excluding Internet-based activities

  制造行业: 10TPS~5000TPS

  互联网电子商务: 10000TPS-1000000TPS

  互联网中型网站: 1000TPS-50000TPS

  互联网小型网站: 500TPSM10000TPS

• Max Response Time (Max Response Time): refers to the maximum time from when a user sends a request or instruction to the system responds (responds).

• Minimum Response Time (Mninum ResponseTime): Refers to the minimum time from when the user sends a request or instruction to the system responds (responds). 90% Response Time (90% Response Time) refers to the response time of all users sorted, the 90th% response time.

• From the outside, performance testing mainly focuses on the following three indicators:

  吞吐量: The number of requests and tasks that the system can handle per second.

  响应时间: The time it takes for the service to process a request or a task.

  错误率: The percentage of requests with incorrect results in a batch of requests.

2、JMeter

1. JMeter installation

https://jmeter.apache.org/download_jmeter.cgi Download the corresponding compressed package, unzip and run jmeter.bat

2. JMeter pressure test example

Add thread group-add the number of requests:

Add sampler-add request interface:

Add a listener-observe the execution result:

3. JMeter Address Already in use error resolution

The port access mechanism provided by windows itself.

Windows provides ports 1024-5000 for TCP/IP links, and it takes four minutes to recycle them. This caused us to fill up the port when running a large number of requests in a short time.

1 cmd, treated with regeditthe command to open注册表

2. HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\ParametersUnder

1.Right click parametersand add a new one DWORDwith the nameMaxUserPor

2. Then double-click MaxUserPort, enter the numerical data as the 65534base selection 十进制(if it is distributed operation, the control machine and the load machine need to do this)

3. Remember to restart the machine after modifying the configuration to take effect

https://support.microsoft.com/zh-cn/help/196271/when-you-try-to-connect-from-tcp-ports-greater-than-5000-you-receive-t

TCPTimedWaitDelay: 30