1. Introduction
Performance is a non-functional characteristic of software . The focus is not on whether the software can complete a specific function, but the timeliness displayed when the function is completed; the definition of performance testing : refers to simulating a variety of normal and peak values through automated testing tools And abnormal load conditions to test various performance indicators of the system, observe whether the performance of the system in a given environment and scenario is consistent with expectations, verify whether the system has performance defects, and identify performance bottlenecks based on test results to improve the system The complete course of performance.
2. Performance test category
Server:
Interface testing based on the protocol level: By simulating a large number of client requests sent to the server, evaluate whether the server’s load processing capability, hardware resource utilization efficiency, network transmission response time and other indicators meet the performance requirements of the application system , or through stress testing, to simulate the stability and reliability of the server system under extreme conditions.
Code-level interface testing: Similar to protocol-level interface testing, it just calls the API interface of the client or server more directly, and initiates requests to the server through concurrent threads. It is suitable for performance testing at the white box level, and is used to test the performance of a certain unit. (Slightly different from the real scene: for example, web page requests, including many static resource requests, when testing the interface, there is little control)
Client
stand-alone application, client performance test: such as the resource consumption of the APP application on the current mobile terminal, such as the utilization of CPU, memory, traffic, power and other resources. Or the running efficiency of js scripts on the web front end, etc.
3. The principle of performance testing
Based on the protocol, based on multi-thread/multi-process/coroutine, simulate the user's usage scenario (try to simulate all the user's page requests, try to realize all the steps of the user operation scenario -- transaction) simulate the user usage scenario Note: the size of the
request
block (Simulate the size of the data submitted by the user -- parameterization)
Parametrically construct user data or take data from the production environment (at this time, user information needs to be desensitized)
Thinking time (simulating the waiting time for thinking pauses between two operations of the user; Random waiting time)
cache (simulate browser cache data -- in memory, hard disk)
bandwidth (through router or network device speed limit settings, or test tool settings to simulate bandwidth)
concurrent users (through multi-threading, process, protocol Process to simulate)
Database capacity: (simulate the data of the database in the real use scenario of the user, pre-embed, desensitize the production data, etc.)
Test environment: the index conversion between the test environment and the production environment. Rent a cloud server;
Fourth, the purpose of performance testing
Determine if the application meets performance requirements (for example, the system should be able to handle up to 1,000 concurrent users). Locate computational bottlenecks in your application. Determine if the performance levels claimed by the software vendor are true. Compare two or more systems and determine which one performs best. Measure stability under peak traffic events.
Five, performance test classification
Load test : under certain software, hardware, and network environment, send requests to the server by simulating different numbers of users until one or more performance indicators of the system reach the safety critical value, so as to detect the occurrence of the maximum performance bottleneck of the server where. For example: to test the performance of the login interface, first simulate 10 user visits, then 100 user visits, and then 1000 user visits, pay attention to the response time of the program, the resources consumed, until the timeout or the exhaustion of key resources; the main purpose is:
detection The maximum number of users and the optimal number of users of the server. (Refer to the load test model diagram)
Stress test : Under certain software, hardware, and network environment, by simulating a large number of virtual users to generate load on the server, the server's resources are in the limit state and continue to run for a long time to test whether the server can work stably under high load conditions ; Stress testing can also be seen as a specific type of load testing. For example: check whether the program can run for 1 day, 1 month, and 1 year, whether all indicators are within the normal indicators, whether there will be memory overflow, whether there will be functional failures, etc.
The main purpose is: to detect the stability and robustness of the server;
the difference between the two: performance testing is to obtain or verify system performance indicators. Stress testing is to test the stability of the system under high load conditions.
Benchmark test : Under a certain software, hardware, and network environment, apply low pressure to the system, simulate a small number of concurrent users to send requests to the server, check the operating status of the system and record relevant data as a basic reference.
The main purpose is: to provide a reference standard for subsequent performance testing; also to debug scripts to ensure normal operation; somewhat similar to smoke testing.
Configuration test : In different software and hardware environments, by simulating a certain number of users to send requests to the server, it is used to detect the performance consumption of the server under different server or client configurations.
The main purpose is: to find the best performance; (server performance, or client performance)
Concurrency test : Under certain software, hardware, and network environment, by simulating a large number of users accessing a system at the same time, modules or data records and other concurrent operations, pay attention to possible performance bottlenecks in the system, such as memory leaks, thread deadlocks or resource competition And other issues.
The main purpose is: to detect whether the server will have thread deadlock, resource preemption, memory leak and other problems.
Limit test : Under a certain software, hardware, and network environment, by simulating a large number of concurrent users, sending requests to the server for a long time without interruption (one week, one month, one year) to test whether the server will problem appear. For example: Tmall Double Eleven, etc.;
Capacity Test : Test the database. Collect stress metrics on the database. and analyze vials
The common ones are load testing, stress testing, and benchmarking, while others are less common.
6. Indicators
Response time (response time): The time from when the user sends a request to when the user receives the response data returned by the server is the response time. The industry standard is generally 258 principles; 2 seconds is the best; 2-5 seconds is average; more than 5-8 seconds is poor.
Throughput (troughput): the number of client requests processed by the system per unit time; generally, the number of requests/second is used as the unit of throughput. General concerns include:
throughput in: TI throughput, the amount of requests entering the server. (request)
throughput out: TO vomit, the amount returned from the server; (response)
Transaction per second (transaction per second) : the number of transactions that the system can process per second;
transaction: a series of user operations according to a certain logic, a set of one or several steps of operations. The characteristics of a transaction (atomicity, consistency, isolation, durability) are either fully executed or fully rolled back.
Requests per second-HPS (hits per second): The number of requests sent by the client per second.
Response per second-RPS (response per second): The number of responses returned by the server per second.
Transaction success rate: the probability of transaction success, industry standards generally require a success rate of more than 99.5%.
Resource utilization: cpu usage: 75-80% or more; memory usage: 70%, disk: 70%, network: 70%
7. Related concepts
Correlation: the current request parameters are obtained from the previous request response;
rendezvous point: when doing stress testing or concurrency testing, you can set a rendezvous point, and execute it immediately after the number of started threads reaches the requirements of the rendezvous point.
The optimal number of users: the number of users corresponding to the load when all system resources are in an optimal state is the optimal number of users.
Maximum number of users: the critical value for the system to operate normally; (the number of users in the limit saturation state)
concurrent users: the total number of users who make requests to the server at the same time; (usually about 8%-12% of the number of online users)
online users Number: The number of users visiting within a certain period of time. These users are only online and do not necessarily do a certain thing at the same time.
Number of system users: software system registration is the total number of users;
single user single iteration: SUSI, used to debug the basic functions of the code.
Single user multiple iterations: SUMI, used to debug some data duplication issues.
Multi-user single iteration: MUSI, used to debug multi-threaded concurrency.
Multi-User Multi-Iteration: MUMI, for formal performance test execution.
8. Performance testing process
Analysis: Analyze performance test requirements, determine performance indicators, and implement specific data (eg: RT: 2s, cpu: 75%)
RT: Response time; if there is a clear demand, the demand shall prevail; if there is no clear demand, the industry standard 258 Second.
TPS: If there is historical data, it is calculated based on historical online data analysis; such as system PV and transaction volume of e-commerce projects. No historical data: use the formula: TPS = VU/(Tt + Tr); respectively: the number of concurrent users, thinking time, and response time. Industry standards: Finance: 5,000-100,000; Insurance: 100-5,000; Large e-commerce: 100,000-1,000,000; Small e-commerce: 5,000-10,000
Transaction success rate: The probability of transaction success, industry standards generally require a success rate of 99.5% or more.
Resource utilization: cpu usage: 75-80% or more; memory usage: 70%, disk: 70%, network: 70%
Design: design performance test plan;
project background, test purpose, test scope, test strategy, test method, test environment, test risk, test output, test start, pause, stop, restart criteria.
Realization: preparation of performance test scripts, test cases, business models, data, test scenarios and environments;
test tools, test scripts; SUSI, SUMI, MUSI, MUMI;
Execution: Execute test scripts (online load), organize test data
Analysis: analyze test data and graphics, find system bottlenecks, perform performance tuning, and regression testing
Maintenance: maintain test scripts, test tools (framework)
Nine, python implementation
Simulate user usage scenarios:
import random
import re
import time
import requests
import threading
#做一个响应时间列表，保存每个操作的时间；用以分析时，查看平均的响应时间、最快、最慢等
time_list = []
#静态资源，如果已经缓存的，放入这里做个记录
uri_li = []
class Deom:
def __init__(self):
self.session = requests.session()
#模拟用户打开首页信息
def homepage(self):
resp = self.session.get(url='http://localhost:8080/homepage')
response = str(resp.headers) + resp.text
#获取静态资源，
resp_html = resp.text
url1 = re.findall('type="text/css" href="(.+?)"',resp_html)
url2 = re.findall('src="(.+?)"',resp_html)
background = re.findall('backaground-image:url\(\'(.+?)\'\)',resp_html)
url_li = url1+url2+background
for uri in url_li:
url = 'http://localhost:8080'+uri
#如果这个静态资源时第一次访问，则下载静态资源到缓存
if uri not in uri_li:
self.session.get(url = url)
uri_li.append(uri)
print(f'首页数据响应大小是： {len(response)/1024}KB')
#模拟用户登录账户
def login(self):
account={'username':'lijian','password':'123456'}
resp = self.session.post(url='http://localhost:8080/login',data=account)
response = str(resp.headers)+resp.text
print(f'登录数据响应大小是： {len(response)/1024}KB')
#模拟添加会员
def add_customer(self):
customer={'cus_Name':'lijian','sex':'男'，'data':'2022-05-29'}
resp = self.session.post(url='http://localhost:8080/add',data=customer)
response = str(resp.headers)+resp.text
print(f'添加用户数据响应大小是： {len(response)/1024}KB')
#主调方法，把模拟用户操作整个流程，先打开主页,然后登录，最后添加用户
def main_test(self):
#不同操作之间有个思考时间
self.homepage()
time.sleep(round(random.uniform(1, 3)))
self.login()
time.sleep(round(random.uniform(1, 3)))
self.add_customer()
if __name__ == '__main__'
demo = Deom()
#通过多线程模拟多用户,多迭代
for i in range(10):
threading.Thread(target=demo.main_test()).start()
#demo.main_test()10. Monitoring System Resources
Doing performance testing requires us to monitor the status of the system at all times, such as ps, top, free, etc.; a good way to obtain system information in Python is to use the third-party module psutil. As the name suggests, psutil = process and system utilities, it can not only realize system monitoring through one or two lines of code, but also can be used across platforms, supporting Linux/UNIX/OSX/Windows, etc., it is a must for test engineers, system administrators and operation and maintenance partners Mandatory modules that are missing.
Install psutils:
pip install psutilCommonly used methods to check content, CPU, network usage, etc.
import psutil
# 打印CPU信息,看多少核的
print(psutil.cpu_count())
# 打印CPU信息,看利用率
print(psutil.cpu_percent())
# 获取内存信息
print(psutil.virtual_memory())
# 获取系统交换内存的统计信息
print(psutil.virtual_memory())
# 磁盘
print(psutil.disk_usage('E:').percent)
# 网络
print(psutil.net_io_counters())For the above simulated user usage scenarios, you can add code to view resource usage, so as to monitor the usage of resources such as CPU and memory at each stage.