1. Theoretical knowledge:
1.1 Definition
1. Benchmarking is an activity of measuring and evaluating software performance indicators, which is used to establish a performance baseline at a certain moment, so that when the system undergoes software and hardware changes, the benchmarking test is re-conducted to evaluate the impact of changes on performance
2. Benchmark test is a kind of stress test for system settings, but it is different from stress test
Benchmark test: direct, simple, and easy to compare, used to evaluate the processing power of the server Benchmark test: may not care about business logic, the authenticity of the query and business used may
have nothing to do with the business environment Stress
test: Test real business data to obtain the pressure that the real system can withstand
1.2 Test steps
Select sysbench as the test tool
1. Install sysbench
1) Download the installation package
wget https://github.com/akopytov/sysbench/archive/0.5.zip2) Install dependent tools
apt -y install make automake libtool pkg-config libaio-dev vim-common
# For MySQL support
apt -y install libmysqlclient-dev
# For PostgreSQL support
apt -y install libpq-dev3) Compile and install
unzip 0.5.zip
cd sysbench-0.5
./autogen.sh
# Add --with-pgsql to build with PostgreSQL support
./configure
make
make install2. Prepare to test
1) Create a test library
create database sysbenchtest;2) Create a test user
grant all privileges on sysbenchtest.* to sysbench@'localhost' identified by '4rfv$RFV';
flush privileges;3) Prepare test data: build 5 tables, each with 100,000 pieces of data
sysbench --test=./sysbench/tests/db/oltp.lua --mysql-host=127.0.0.1 --mysql-db=sysbenchtest --mysql-user=sysbench \
--mysql-password='4rfv$RFV' --oltp-test-mode=complex --oltp-tables-count=5 --oltp-table-size=100000 --threads=10 --time=120 \
--rand-init=on --report-interval=10 --mysql-table-engine=innodb prepare
* Parameter Description
mysql-db=dbtest1a：测试使用的目标数据库，这个库名要事先创建
--oltp-tables-count=10：产生表的数量
--oltp-table-size=500000：每个表产生的记录行数
--oltp-dist-type=uniform：指定随机取样类型，可选值有 uniform(均匀分布), Gaussian(高斯分布), special(空间分布)。默认是special
--oltp-read-only=off：表示不止产生只读SQL，也就是使用oltp.lua时会采用读写混合模式。默认 off，如果设置为on，则不会产生update,delete,insert的sql。
--oltp-test-mode=nontrx：执行模式，这里是非事务式的。可选值有simple,complex,nontrx。默认是complex
simple：简单查询，SELECT c FROM sbtest WHERE id=N
complex (advanced transactional)：事务模式在开始和结束事务之前加上begin和commit， 一个事务里可以有多个语句，如点查询、范围查询、排序查询、更新、删除、插入等，并且为了不破坏测试表的数据，该模式下一条记录删除后会在同一个事务里添加一条相同的记录。
nontrx (non-transactional)：与simple相似，但是可以进行update/insert等操作，所以如果做连续的对比压测，你可能需要重新cleanup,prepare。
--oltp-skip-trx=[on|off]：省略begin/commit语句。默认是off
--rand-init=on：是否随机初始化数据，如果不随机化那么初始好的数据每行内容除了主键不同外其他完全相同
--num-threads=12： 并发线程数，可以理解为模拟的客户端并发连接数
--report-interval=10：表示每10s输出一次测试进度报告
--max-requests=0：压力测试产生请求的总数，如果以下面的max-time来记，这个值设为0
--max-time=120：压力测试的持续时间，这里是2分钟。
注意，针对不同的选项取值就会有不同的子选项。比如oltp-dist-type=special，就有比如oltp-dist-pct=1、oltp-dist-res=50两个子选项，代表有50%的查询落在1%的行（即热点数据）上，另外50%均匀的(sample uniformly)落在另外99%的记录行上。
再比如oltp-test-mode=nontrx时, 就可以有oltp-nontrx-mode，可选值有select（默认）, update_key, update_nokey, insert, delete，代表非事务式模式下使用的测试sql类型。
以上代表的是一个只读的例子，可以把num-threads依次递增（16,36,72,128,256,512），或者调整my.cnf参数，比较效果。另外需要注意的是，大部分mysql中间件对事务的处理，默认都是把sql发到主库执行，所以只读测试需要加上oltp-skip-trx=on来跳过测试中的显式事务。
ps1: 只读测试也可以使用share/tests/db/select.lua进行，但只是简单的point select。
ps2: 我在用sysbench压的时候，在mysql后端会话里有时看到大量的query cache lock，如果使用的是uniform取样，最好把查询缓存关掉。当然如果是做两组性能对比压测，因为都受这个因素影响，关心也不大。3. Run the Benchmark
1) Run the Test: Mixed Operations Test
sysbench --test=./sysbench/tests/db/oltp.lua --mysql-table-engine=innodb --mysql-host=127.0.0.1 --mysql-db=sysbenchtest --mysql-user=sysbench \
--mysql-password='4rfv$RFV' --num-threads=8 --oltp-table-size=100000 --oltp_tables_count=5 --oltp-read-only=off --report-interval=10 \
--rand-type=uniform --max-time=300 --max-requests=0 --percentile=99 run >> ./log/sysbench.log2) If multiple tests clean up records
sysbench --test=./sysbench/tests/db/oltp.lua --mysql-table-engine=innodb --mysql-host=127.0.0.1 --mysql-db=sysbenchtest --num-threads=8 \
--oltp-table-size=100000 --oltp_tables_count=5 --oltp-read-only=off --report-interval=10 --rand-type=uniform --max-time=600 --max-requests=0 \
--mysql-user=test --mysql-password='4rfv$RFV' cleanup
* Parameter Description
--num-threads=8 表示发起 8个并发连接
--oltp-read-only=off 表示不要进行只读测试，也就是会采用读写混合模式测试
--report-interval=10
表示每10秒输出一次测试进度报告 --rand-type=uniform
表示随机类型为固定模式，其他几个可选随机模式：uniform(固定),gaussian(高斯),special(特定的),pareto(帕累托)
--max-time=120 表示最大执行时长为 120秒,实际环境中建议30分钟
--max-requests=0 表示总请求数为 0，因为上面已经定义了总执行时长，所以总请求数可以设定为 0；也可以只设定总请求数，不设定最大执行时长
--percentile=99 表示设定采样比例，默认是 95%，即丢弃1%的长请求，在剩余的99%里取最大值4. Analyze the tuning effect: Control the test through scripts, and take the average value for multiple tests. To run the script, you need to clear the previous log.
1) Parameter analysis
queries performed:
read: 938224 -- 读总数
write: 268064 -- 写总数
other: 134032 -- 其他操作总数(SELECT、INSERT、UPDATE、DELETE之外的操作，例如COMMIT等)
total: 1340320 -- 全部总数
transactions: 67016 (1116.83 per sec.) -- 总事务数(每秒事务数)
deadlocks: 0 (0.00 per sec.) -- 发生死锁总数
read/write requests: 1206288 (20103.01 per sec.) -- 读写总数(每秒读写次数)
other operations: 134032 (2233.67 per sec.) -- 其他操作总数(每秒其他操作次数)
General statistics: -- 一些统计结果
total time: 60.0053s -- 总耗时
total number of events: 67016 -- 共发生多少事务数
total time taken by event execution: 479.8171s -- 所有事务耗时相加(不考虑并行因素)
response time: -- 响应时长统计，主要参考指标
min: 4.27ms -- 最小耗时
avg: 7.16ms -- 平均耗时
max: 13.80ms -- 最长耗时
approx. 99 percentile: 9.88ms -- 超过99%平均耗时
Threads fairness:
events (avg/stddev): 8377.0000/44.33
execution time (avg/stddev): 59.9771/0.005. Test analysis script
Analysis_mysql.py, control the number of tests and get the average value, but clear the sysbench.log log file before each test
#!/usr/bin/env python
import re
import os
#logFile = open('C:\\Users\\Administrator\\Desktop\\sysbench.log','r')
logFile = open('/root/sysbench-0.5/log/sysbench.log','r')
analysisFile = open('/root/sysbench-0.5/log/analysis.log','a+')
#analysisFile = open('C:\\Users\\Administrator\\Desktop\\analysis.log','a+')
# 结果分析
def closeFile(file):
file.close()
def analysis():
print("开始分析...")
transacion = []
readeWrite = []
mintime = []
avgtime = []
maxtime = []
approxtime = []
# 读取文件数据
for line in logFile:
transacion_persec = re.search(r'transactions:.*\((\d+\.*\d*).*\).*',line)
readeWrite_persec = re.search(r'read/write requests:.*\((\d+\.*\d*).*\).*',line)
mintime_response = re.search(r'min:\s*(\d+\.?\d*).*',line)
avgtime_response = re.search(r'avg:\s*(\d+\.?\d*).*',line)
maxtime_response = re.search(r'max:\s*(\d+\.?\d*).*',line)
approxtime_response = re.search(r'approx.*?(\d+\.{1}\d*).*',line)
if transacion_persec:
transacion.append(transacion_persec.group(1))
elif readeWrite_persec:
readeWrite.append(readeWrite_persec.group(1))
elif mintime_response:
mintime.append(mintime_response.group(1))
elif avgtime_response:
avgtime.append(avgtime_response.group(1))
elif maxtime_response:
maxtime.append(maxtime_response.group(1))
elif approxtime_response:
approxtime.append(approxtime_response.group(1))
sum = 0.0
# 统计事务
for data in transacion:
sum += float(data)
avgtransaction = ('%.2f'%(sum / len(transacion)))
analysisFile.write("transacion每秒：" + str(avgtransaction) + "\n")
sum = 0.0
# 统计读写
for data in readeWrite:
sum += float(data)
read_write = ('%.2f'%(sum / len(readeWrite)))
analysisFile.write("read/write每秒：" + str(read_write) + "\n")
sum = 0.0
# 统计最小时间
for data in mintime:
sum += float(data)
if len(mintime) !=0:
min_time = ('%.2f'%(sum / len(mintime)))
analysisFile.write("min time：" + str(min_time) + "\n")
sum = 0.0
# 统计最大时间
for data in maxtime:
sum += float(data)
max_time = ('%.2f'%(sum / len(maxtime)))
analysisFile.write("max time：" + str(max_time) + "\n")
sum = 0.0
# 统计平均时间
for data in avgtime:
sum += float(data)
avg_time = ('%.2f'%(sum / len(avgtime)))
analysisFile.write("avg time：" + str(avg_time) + "\n")
analysisFile.write("="*20 + "\n")
print("分析完成...")
closeFile(analysisFile)
# 数据准备
def prepare():
print("准备数据...")
os.system("sysbench --test=/root/sysbench-0.5/sysbench/tests/db/oltp.lua --mysql-host=127.0.0.1 --mysql-db=sysbenchtest --mysql-user=sysbench --mysql-password='4rfv$RFV' --oltp-test-mode=complex --oltp-tables-count=5 --oltp-table-size=100000 --threads=10 --time=120 --rand-init=on --report-interval=10 --mysql-table-engine=innodb prepare")
# 测试函数
def experiment():
print("测试数据...")
os.system("sysbench --test=/root/sysbench-0.5/sysbench/tests/db/oltp.lua --mysql-table-engine=innodb --mysql-host=127.0.0.1 --mysql-db=sysbenchtest --mysql-user=sysbench --mysql-password='4rfv$RFV' --num-threads=8 --oltp-table-size=100000 --oltp_tables_count=5 --oltp-read-only=off --report-interval=10 --rand-type=uniform --max-time=30 --max-requests=0 --percentile=99 run >> /root/sysbench-0.5/log/sysbench.log")
#
# # 清空数据
def clean():
print("清空测试数据库...")
os.system("sysbench --test=/root/sysbench-0.5/sysbench/tests/db/oltp.lua --mysql-table-engine=innodb --mysql-host=127.0.0.1 --mysql-db=sysbenchtest --num-threads=8 --oltp-table-size=100000 --oltp_tables_count=5 --oltp-read-only=off --report-interval=10 --rand-type=uniform --max-time=600 --max-requests=0 --mysql-user=test --mysql-password='4rfv$RFV' cleanup")
if __name__ == '__main__':
for i in range(0,3):
prepare()
experiment()
clean()
analysis()Finally, I would like to thank everyone who has read my article carefully. Reciprocity is always necessary. Although it is not a very valuable thing, you can take it away if you need it:
These materials should be the most comprehensive and complete preparation warehouse for [software testing] friends. This warehouse has also accompanied tens of thousands of test engineers through the most difficult journey, and I hope it can help you! Friends in need can click on the small card below to get it