【Storage Test】vdbench storage performance test tool
I. Introduction
1 Introduction
vdbench is an I/O workload generator typically used to verify data integrity and measure direct-attached (or network-attached) storage performance. It can run in windows and linux environments and can be used to test file system or block device benchmark performance.
2. Get the version package
vdbench version package
download link
jdk version package
download link (windows)
download link (linux)
II. Installation and deployment
1. Linux client
Deploy the jdk environment
Upload jdk-8u341-linux-x64.rpm to the /root directory and install the rpm package
#rpm -ivh jdk-8u341-linux-x64.rpm
Check if the installation was successful
#java -version
Configure the key-free
option is not required, if you need to test online, you need to configure this item
The example uses three clients for online testing, using client node1 as the primary node
client hostname
Client IP
node1
10.10.10.1
node2
10.10.10.2
node3
10.10.10.3
Write the mapping relationship between each node IP and host name into the /etc/hosts configuration file
#echo ‘10.10.10.1 node1’ >> /etc/hosts
#echo ‘10.10.10.2 node2’ >> /etc/hosts
#echo ‘10.10.10.3 node3’ >> /etc/hosts
The master node generates a public key file and copies it to other slave nodes (configure the master node to slave node to log in without a secret key)
#ssh-keygen
#ssh-copy-id node2
#ssh-copy-id node3
Use vdbench to
decompress the vdbench installation package to the /root/vdbench50406 directory, switch to the vdbench decompression directory and execute the corresponding parameter file
#tar -zxvf vdbench50406.tar.gz -C /root/
2. Windows client
Deploy the jdk environment
Just install the jdk software directly (environment variables will be added automatically after the installation is complete)
Use vdbench
Unzip the vdbench software package, switch to the vdbench decompression directory and execute the corresponding parameter file
3. Precautions
During the online test, the system time of the client must be consistent, otherwise there will be a clock synchronization alarm (this can lead to heartbeat issues). The firewall of the client should be closed (or set the open
program to specify port 5570, 5560 access)
to close the system log service rsyslog, Avoid printing information in other log files during operation
Add messagescan=no to the parameter file to filter out redundant system logs
3. Common operations
1. Common operation commands
Check vdbench environment
#cd /root/vdbench50406/
#./vdbench -t
run test model
Note: -f is followed by the test parameter file name, -o is followed by the export test result path
#./vdbench -f {filename} -o {exportpath}
establish rsh communication
Note: This command is used when running vdbench online with multiple hosts in the windows system, because the windows operating system does not support ssh, so vdbench provides the rsh communication method. After executing this tool on the target host, vdbench will start a java socket for communication between vdbench slave and master
./vdbench rsh
4. Parameter description
vdbench can be used for file system and block device benchmark performance testing. The following mainly introduces the common test parameters of file system and block storage. All test parameters of vdbench are
defined in a parameter file, which are read in sequence at runtime and perform corresponding operations. The order of execution needs to be defined when the file is defined
1. File system
The file system parameter file definition order is: HD, FSD, FWD, RD
1.1、HD（Host Define）
Optional, HD parameters do not need to be configured when a single machine is running, and generally only need to be configured when multi-host online testing
hd=default,vdbench=/root/vdbench50406,user=root,shell=ssh
hd=hd1,system=node1
hd=hd2,system=node2
hd=hd3,system=node3
hd= identifies the name defined by the host, when multiple hosts are running, you can use hd1, hd2, hd3... to distinguish
system = host IP address or host name
vdbench = vdbench execution file storage path, when the multi-host storage path is different, you can define it in hd When specifying
user=slave and master communication separately, user
shell= can be selected as rsh, ssh or vdbench, and the default value is rsh. During multi-host online testing, the communication mode between master and slave hosts
needs to be configured when the parameter value is rsh. The master and slave host rsh mutual trust, considering that rsh uses plaintext transmission, the security level is not enough, usually this communication method is not recommended when the parameter
value is ssh, you need to configure the master and slave host ssh mutual trust, usually used when the Linux host is online This communication method
When the parameter value is vdbench, you need to run vdbench rsh on all slave hosts to enable the rsh daemon process of vdbench itself. Usually, this communication method is used when the Window host is online.
1.2, FSD (File System Define)
fsd=default,openflags=directio,depth=2,width=3,files=2,size=128k
fsd=fsd1,anchor=/mnt/client1
fsd=fsd2,anchor=/mnt/client2
fsd=fsd3,anchor=/mnt/client3
fsd= identifies the name of the file system definition. When there are multiple file systems (fsd1, fsd2, fsd3...), you can specify default (the same parameter is used as the default value of all fsds) openflags= is set to o_direct or directio to use unbuffered
caching Anchor= file is written into the
directory, the specified path of linux is /dir01; the specified path of windows is E:\dir01;
depth= the number of levels of the created directory (that is, the depth of the directory)
width= the subfolder of each layer of folders number of
files= number of test files (multi-level directory structure will be generated during the vdbench test process, in fact, only the last layer of directory will generate test files)
size= size of each test file
**distribution= ** The optional value is bottom or all , the default is bottom
– when the parameter value is bottom, the program only writes the test file in the last layer directory
– when the parameter value is all, the program writes the test file in every layer directory
shared= The optional value is yes or no, the default value is no, generally only when multi-host online testing is specified,
vdbench does not allow different slaves to share all files in the same directory structure, because this will bring a lot of overhead, but they allow sharing the same Directory Structure. If shared=yes is set, then different slaves can share all the files in a directory equally for access, which means that each slave has its own equally divided access area, so it is not possible to test the read and write of the same file by multiple customers
– When the multi-host online test, the written root directory anchor is the same path, you need to specify the parameter value to be yes
hd=default,vdbench=/root/vdbench50406,user=root,shell=ssh
hd=hd1,system=node1
hd=hd2,system=node2
hd=hd3,system=node3
fsd=fsd1,anchor=/client/,depth=2,width=100,files=100,size=4k,shared=yes
The calculation formula is as follows:
Number of generated folders in the last layer = width^depth
Number of test files = (width^depth)*files
fsd=fsd1,anchor=/dir1,depth=2,width=3,files=2,size=128k
Taking the above parameters as an example, the generated directory structure and test files are as follows:
The number of folders in the last layer=3^2=9 The number of files in the last layer=9*2=18
/dir1/
├── no_dismount.txt
├── vdb.1_1.dir
│ ├── vdb.2_1.dir
│ │ ├── vdb_f0001.file
│ │ └── vdb_f0002.file
│ ├── vdb.2_2.dir
│ │ ├── vdb_f0001.file
│ │ └── vdb_f0002.file
│ └── vdb.2_3.dir
│ ├── vdb_f0001.file
│ └── vdb_f0002.file
├── vdb.1_2.dir
│ ├── vdb.2_1.dir
│ │ ├── vdb_f0001.file
│ │ └── vdb_f0002.file
│ ├── vdb.2_2.dir
│ │ ├── vdb_f0001.file
│ │ └── vdb_f0002.file
│ └── vdb.2_3.dir
│ ├── vdb_f0001.file
│ └── vdb_f0002.file
├── vdb.1_3.dir
│ ├── vdb.2_1.dir
│ │ ├── vdb_f0001.file
│ │ └── vdb_f0002.file
│ ├── vdb.2_2.dir
│ │ ├── vdb_f0001.file
│ │ └── vdb_f0002.file
│ └── vdb.2_3.dir
│ ├── vdb_f0001.file
│ └── vdb_f0002.file
└── vdb_control.file
12 directories, 20 files
1.3、FWD（FileSystem Workload Defile）
fwd=default,operation=read,xfersize=4k,fileio=sequential,fileselect=random,threads=2
fwd=fwd1,fsd=fsd1,host=hd1
fwd=fwd2,fsd=fsd2,host=hd2
fwd=fwd3,fsd=fsd3,host=hd3
fwd= identifies the name of the file system workload definition. When defining multiple file system workloads, you can use fwd1, fwd2, fwd3... to distinguish fsd
= identify this workload using the name of the file storage definition
host= identify this workload using the host
operation= The optional value is read or write, and the file operation method
rdpct= The optional value is 0~100, the percentage of read operation, which generally needs to be specified when mixed reading and writing. When the value is 60, the mixed reading and writing ratio is 6:4
fileio= The optional value is random or sequential, which identifies the way file I/O will be performed.
fileselect= random or sequential, which identifies the way to select files or directories.
xfersizes= The size of data processed by data transfer (read and write operations) (ie Single IO size)
threads= the number of concurrent threads of this workload
Note:
1. In the case of default format=yes, the prefill data workload is defined as threads=8, xfersize=128k, that is, the prefill data uses 8 threads, and the IO size
If you need to change the default pre-filled data workload definition for 128K , you need to increase the parameters to specify the number of threads and IO size (fwd=format, threads=nn, xfersize=nn)
1.4、RD（Run Define）
rd=rd1,fwd=(fwd1-fwd3),fwdrate=max,format=restart,elapsed=604800,interval=10
rd= Identifies the name of the file system run definition.
fwd= Identifies the name of the file system workload definition.
fwdrate= Number of filesystem operations performed per second.
If it is set to max, it means that there is no limit, and it will
be adaptive according to the maximum strength . If there is no file in the test directory, vdbench will exit abnormally because there is no available file to read and write
yes: means to delete the existing file structure of the test directory, and recreate a new file structure
restart: means to create only ungenerated directories or files, and Increase the file that does not reach the actual size
elapsed= The default value is 30, the test run duration (in seconds)
interval= The result output printing interval (in seconds)
2. Block device
The block device parameter file definition sequence is: HD, SD, WD, RD
1.1、HD（Host Define）
Optional, HD parameters do not need to be configured when a single machine is running, and generally only need to be configured when multi-host online testing
hd=default,vdbench=/root/vdbench50406,user=root,shell=ssh
hd=hd1,system=node1
hd=hd2,system=node2
hd=hd3,system=node3
hd= identifies the name defined by the host, when multiple hosts are running, you can use hd1, hd2, hd3... to distinguish
system = host IP address or host name
vdbench = vdbench execution file storage path, when the multi-host storage path is different, you can define it in hd When specifying
user=slave and master communication separately, user
shell= can be selected as rsh, ssh or vdbench, and the default value is rsh. During multi-host online testing, the communication mode between master and slave hosts
needs to be configured when the parameter value is rsh. The master and slave host rsh mutual trust, considering that rsh uses plaintext transmission, the security level is not enough, usually this communication method is not recommended when the parameter
value is ssh, you need to configure the master and slave host ssh mutual trust, usually used when the Linux host is online This communication method
When the parameter value is vdbench, you need to run vdbench rsh on all slave hosts to enable the rsh daemon process of vdbench itself. Usually, this communication method is used when the Window host is online. 1.2,
SD (Storage Define)
sd=sd1,hd=hd1,lun=/dev/sdb,openflags=o_direct,threads=6
sd=sd3,hd=hd2,lun=/dev/sdb,openflags=o_direct,threads=6
sd=sd6,hd=hd3,lun=/dev/sdb,openflags=o_direct,threads=6
sd= identifies the name defined by the storage
hd= identifies the name defined by the host
lun= writes to the block device, if linux uses the sdb disk, the specified path is /dev/sdb; if windows uses the G disk, the specified path is \.\G:;
openflags= By setting it to o_direct or directio, read and write operations are performed in an unbuffered cache.
threads= The maximum number of concurrent I/O requests to SD
1.3, WD (Workload Define)
wd=wd1,sd=sd*,seekpct=100,rdpct=100,xfersize=8k,skew=40
wd=wd2,sd=sd*,seekpct=100,rdpct=0,xfersize=8k,skew=10
wd=wd3,sd=sd*,seekpct=100,rdpct=100,xfersize=1024k,skew=40
wd=wd4,sd=sd*,seekpct=100,rdpct=0,xfersize=1024k,skew=10
wd= identifies the name of the workload definition
sd= identifies the name of the storage definition
seekpct= The optional value is 0 or 100 (sequential or random can also be used), the default value is 100, the percentage of random seek, when set to 0, it means Order, when set to 100 means random.
rdpct= The percentage of read requests to the total number of requests. When set to 0, it means write, and when set to 100, it means read.
xfersize= The size of the data to be transferred. The default setting is 4k
skew= is not a mandatory option, and generally needs to be specified when there are multiple workloads, indicating that the workload accounts for the percentage of the total workload (the sum of skew is 100)
1.4, RD (Run Define)
rd=rd1,wd=wd*,iorate=max,maxdata=400GB,warmup=30,elapse=604800,interval=5
rd= identifies the name of the run definition
wd= identifies the name of the workload definition
iorate= common optional values are 100, max, the fixed I/O rate of this workload
– when the parameter value is 100, the rate is 100 I/O per second Run the workload at the speed of Os. When the parameter value is set to a value lower than the maximum rate, the effect of limiting the read and write speed can be achieved –
when the parameter value is max, the workload is run at the maximum I/O rate, general testing When reading and writing the maximum performance, the value of this parameter is max
warmup= warmup time (in seconds). By default, vdbench will exclude the output data of the first time interval, and the test of the program during the warmup time will not be included in the final test. In the test results (that is, the formal test starts after the warm-up is over)
– when the interval is 5 and elapsed is 600, the test performance is the average performance in the time interval of 2~elapsed/interval (avg_2-120)
– when the interval is 5 , warmup is 60, and elapsed is 600, the test performance is 1+(warmup/interval)~(warmup+elapsed)/interval(avg_13-132) The average performance maxdata= read and write data size in the time interval, usually
, The test ends after running the elapsed time; when specifying the elapsed and maxdata parameter values at the same time, the parameter that runs the fastest shall prevail (that is, when the maxdata test time is less than elapsed, the program ends after writing the elapsed data amount) – when the parameter value
is When it is less than 100, it means that the read and write data volume is a multiple of the total storage definition size (for example, maxdata=2, 2 storage definitions (each storage definition data volume is 100G), then the actual read and write data size is 400G) – when the parameter
value When it is more than 100, it means that the amount of data is the actual amount of read and written data (you can use the unit M, G, T, etc.)
elapsed= The default value is 30, the duration of the test run (in seconds)
interval= The report interval (in seconds)
5. Running usage
1. Stand-alone operation
The Linux
example is as follows, a single node is tested for a bare disk, 1M sequential write, the test time is 600s, the warm-up time is 60s, and the report interval is 2s
[root@node1 vdbench50406] cat Single-RawDisk.html
sd=sd1,lun=/dev/sdb,openflag=o_direct
wd=wd1,sd=sd1,seekpct=0,rdpct=0,xfersize=1M
rd=rd1,wd=wd1,iorate=max,warmup=60,elapsed=600,interval=2
[root@node1 vdbench50406]
[root@node1 vdbench50406] ./vdbench -f Single-RawDisk.html
The example of Window
is as follows, a single node is tested for the file system, 1M is written sequentially, the directory depth is 2, the number of directories in each layer is 3, the number of files in each directory is 10, the size of each file is 200M, the test time is 600s, and the report time 2s
E:\vdbench50406>more “Single FileSystem.txt”
fsd=fsd1,anchor=E:\Sigle-FileSystem,depth=2,width=3,files=10,size=200M
fwd=fwd1,fsd=fsd1,operation=write,xfersize=1M,fileio=sequential,fileselect=rando
m,threads=2
rd=rd1,fwd=fwd1,fwdrate=max,format=yes,elapsed=600,interval=5
E:\vdbench50406>
E:\vdbench50406>vdbench -f “Single FileSystem.txt”
2. Online operation
Linux
1. According to 2. Installation and deployment, configure multi-host ssh mutual trust
2. The master host can run the test parameter file. The
example is as follows, the three nodes are for the bare disk online test, write in 1M sequence, the test data volume is 400G, and the warm-up time is 30s. Report interval 5s
[root@node1 vdbench50406] cat Multi-RawDisk
hd=default,vdbench=/root/vdbench50406,user=root,shell=ssh
hd=hd1,system=node1
hd=hd2,system=node2
hd=hd3,system=node3
sd=sd1,hd=hd1,lun=/dev/sdb,openflag=o_direct
sd=sd2,hd=hd2,lun=/dev/sdb,openflag=o_direct
sd=sd3,hd=hd3,lun=/dev/sdb,openflag=o_direct
wd=wd1,sd=sd*,seekpct=0,rdpct=0,xfersize=1M
rd=rd1,wd=wd1,iorate=max,maxdata=100M,elapsed=64800,warmup=30,interval=5
[root@node1 vdbench50406]
[root@node1 vdbench50406] ./vdbench -f Multi-RawDisk
Window
1. All slave hosts run vdbench itself rsh daemon process
E:\vdbench50406>vdbench rsh
2. The master host can run the test parameter file. The
example is as follows. The three nodes are for the online test of the file system, 1M sequential writing, the directory depth is 2, the number of directories in each layer is 3, the number of files in each directory is 10000, and the size of each file is 200M, test time is 600s, report interval is 1s
E:\vdbench50406>more “Multi FileSystem.txt”
hd=default,vdbench=E:\vdbench50406,user=Micah,shell=vdbench
hd=hd1,system=10.10.10.1
hd=hd2,system=10.10.10.2
fsd=fsd1,anchor=Z:\Sigle-FileSystem-01,depth=2,width=3,files=10000,size=200M
fsd=fsd2,anchor=Z:\Sigle-FileSystem-02,depth=2,width=3,files=10000,size=200M
fwd=default,operation=write,xfersize=1M,fileio=sequential,fileselect=random,threads=16
fwd=fwd1,fsd=fsd1,host=hd1
fwd=fwd2,fsd=fsd2,host=hd2
rd=rd1,fwd=fwd*,fwdrate=max,format=yes,elapsed=600,interval=1
E:\vdbench50406>
E:\vdbench50406>vdbench -f “Multi FileSystem.txt”
6. Results Analysis
After vdbench runs the load test, it will generate an output folder in the installation directory, which contains the test result file
1. Output file
errorlog.html
When running tests with data validation enabled, it may contain error messages such as:
invalid key read
invalid lba read (logical byte address of a sector)
invalid SD or FSD name read Get
data corruption
bad sector
flatfile.html
vdbench generates a column-by-column ASCII format information, you can use the parseflat parameter to parse the results./vdbench
parseflat -i <flatfile.html> -o output.csv [-c col1 col2 … ] [-a] [-f col1 value1 col2 value2...]
-i input flatfile, e.g. output/flatfile.html
-o output CSV file name (default stdout)
-c which column to write to CSV. Columns are written in the order specified
-f filters: 'if (colX == valueX) ... ...' (Alphabetic compare)
-a include only the 'avg' data. Default: include only non-avg data.
-i是表示待分析的文件，这里写vdbench输出目录里的flatfile.html这个文件，写其它文件不能正常解析；
-o是解析后的输出文件，可以手动指定存放目录。文件格式为CSV，此文件的列由-c参数指定，列的顺序为-c参数的顺序
-a是表示csv文件中只记录测试过程中的avg值
Examples are as follows:
.\vdbench.bat parseflat -i D:\vdbench50406\output\flatfile.html -c run rate MB/sec seekpct rdpct bytes/io threads resp -o d:\output.csv -a
vdbench parseflat arguments:
Argument 0: -i
Argument 1: D:\vdbench50406\output\flatfile.html
Argument 2: -c
Argument 3: run
Argument 4: rate
Argument 5: MB/sec
Argument 6: seekpct
Argument 7: rdpct
Argument 8: bytes/io
Argument 9: threads
Argument 10: resp
Argument 11: -o
Argument 12: D:\output.csv
Argument 13: -a
14:12:49.265 ParseFlat completed successfully.
histogram.html
A file containing the response time for the report histogram, in text format.
logfile.html
Contains a copy of each line of information that the Java code writes to the console window. logfile.html is mainly used for debugging
parmfile.html
contains test run configuration parameter information
summary.html
records all data information, shows the overall performance and workload in each report interval, and all intervals except the first interval Weighted average
totals.html
records the average value after all data calculations, the general test results are taken from this file, and the weighted average value of all intervals except the first interval
2. Results analysis
2.1. File system
The test parameters are as follows:
hd=default,vdbench=E:\vdbench50406,user=Micah,shell=vdbench
hd=hd1,system=10.10.10.1
hd=hd2,system=10.10.10.2
fsd=fsd1,anchor=Z:\Sigle-FileSystem-01,depth=2,width=3,files=10,size=4M
fsd=fsd2,anchor=Z:\Sigle-FileSystem-02,depth=2,width=3,files=10,size=4M
fwd=default,operation=write,xfersize=1M,fileio=sequential,fileselect=random,threads=2
fwd=fwd1,fsd=fsd1,host=hd1
fwd=fwd2,fsd=fsd2,host=hd2
rd=rd1,fwd=fwd*,fwdrate=max,format=yes,elapsed=600,interval=5
The test results are as follows:
18:47:03.001 Starting RD=format_for_rd1
六月 04, 2020 .Interval. .ReqstdOps… …cpu%… read …read… …write… …mb/sec… mb/sec .xfer… …mkdir… …rmdir… …create… …open… …close… …delete…
rate resp total sys pct rate resp rate resp read write total size rate resp rate resp rate resp rate resp rate resp rate resp
18:48:40.218 avg_2-20 57.6 6.244 13.4 2.99 0.0 0.0 0.000 57.6 6.244 0.00 7.20 7.20 131072 0.2 104.49 0.2 41.526 1.8 7527.0 1.8 192.01 1.8 7134.3 1.8 21.984
18:48:42.000 Starting RD=rd1; elapsed=600; fwdrate=max. For loops: None
18:58:42.205 avg_2-120 6.2 1.063 13.0 2.80 0.0 0.0 0.000 6.2 1.063 0.00 6.24 6.24 1048576 0.0 0.000 0.0 0.000 0.0 0.000 1.6 47.864 1.6 2401.1 0.0 0.000
totals.html generally includes two parts. The first part is the average performance value of the file storage directory structure and data filling. The second part is the average performance value of all time intervals during the execution of the test except the first time interval. Mainly see the second part of the content
Interval
reports the interval number. The test result is generally the weighted average of all time intervals except the first interval. For
example, elapsed=600, interval=5, the performance result is the average value from the second interval to the 120th interval (avg_2- 120)
ReqstdOps
rate
The number of read and write I/O per second (read and write IOPS), which can be controlled by defining the parameter fwdrate through rd operation.
When fwdrate is max, the workload is run at the maximum I/O rate.
When fwdrate is lower than the maximum I/O When the rate is 0, the read and write speed can be limited, and the workload can be run at a fixed I/O rate. Resp Read and
write
request response time (read and write delay), the unit is ms
cpu%
tatol
Total cpu occupancy
sys
system cpu occupancy rate
read pct
The percentage of read requests to the total number of requests, when it is 0, it means write, when it is 100, it means read read
rate
Number
of read I/Os per second (read IOPS)
resp
Read request response time (read delay ), the unit is ms
write
rate
write I/O number per second (write IOPS)
resp
write request response time (write delay), the unit is ms
mb/sec
read
read speed per second
write
write speed per second
total total read
and write speed per second
xfersize
each read and write I/O transfer data volume (that is, the size of a single read and write I/O), the unit is byte B
2.2, block device
The test parameters are as follows:
messagescan=no
hd=default,vdbench=/root/vdbench50406,user=root,shell=ssh
hd=hd1,system=10.10.10.1
hd=hd2,system=10.10.10.2
hd=hd3,system=10.10.10.3
sd=sd1,hd=hd1,lun=/dev/sdb,openflag=o_direct
sd=sd2,hd=hd2,lun=/dev/sdb,openflag=o_direct
sd=sd3,hd=hd3,lun=/dev/sdb,openflag=o_direct
wd=wd1,sd=sd*,seekpct=0,rdpct=0,xfersize=1M
rd=rd1,wd=wd1,iorate=max,elapsed=600,warmup=30,interval=5
The test results are as follows:
19:02:15.001 Starting RD=rd1; I/O rate: Uncontrolled MAX; elapsed=600 warmup=30; For loops: None
Jun 04, 2020 interval i/o MB/sec bytes read resp read write resp resp queue cpu% cpu%
rate 1024**2 i/o pct time resp resp max stddev depth sys+u sys
19:12:46.068 avg_7-126 82.74 82.74 1048576 0.00 289.158 0.000 289.158 2092.803 155.103 23.9 16.3 14.2
interval
reports the interval number. The test result is generally the weighted average of all time intervals except the first one. If a warm-up time is set, this part of the test data also needs to be excluded. For example, elapsed=600,
warmup=30, interval= 5, the performance test result is the average value from the 7th interval to the 126th interval (avg_7-126)
i/o rate
The number of read and write I/O per second (read and write IOPS), you can define the parameter iorate through rd operation Control
When iorate is max, run the workload at the maximum I/O rate.
When iorate is a value lower than the maximum I/O rate, you can limit the read and write speed, and run the workload at a fixed I/O rate per
MB/sec.
Read and write speed per second (read and write bandwidth)
Note: According to the official manual,
bytes i/0
transmits the amount of data per read and write I/O (that is, the size of a single read and write I/O), the unit is byte B, and can be passed through wd The workload definition parameter xfersize controls the percentage of
read pct
read requests to the total number of requests, which can be controlled by the wd workload definition parameter rdpct.
When rdpct is 0, it means that the test model is write
. When rdpct is 100, it means that the test model is read
resp time
Request response time (reading and writing delay), in milliseconds
read resp
read request response time in milliseconds
write resp
write request response time in milliseconds ms
resp max
Maximum request response time, in milliseconds
resp stddev
Request response time standard deviation, in milliseconds
queue depth
Read and write I/0 queue depth
cpu% sys+u
kernel mode space and user mode space CPU usage
cpu% sys
kernel mode Space CPU occupancy 2020-05-29 16:00:16 Clear
7. Test description
1. Test summary
The number of threads (thread) is generally set to the total size of the number of client CPU threads. grep 'processor' /proc/cpuinfo | sort -u | wc -l The total data volume of the test needs to be twice the size of the client memory. The client needs to be cleaned up when testing the
read
model End cache information sync; echo 3 > /proc/sys/vm/drop
2. Common test parameters
4M sequential write
directory depth 2, single-level directory number 100, single directory file number 100, single file size 4M, IO block size 1M, sequential write
hd=default, vdbench=/root/vdbench50406, user=root, shell=ssh
hd=hd1,system=node1
hd=hd2,system=node2
hd=hd3,system=node3
hd=hd4,system=node4
hd=hd5,system=node5
hd=hd6,system=node6
fsd=fsd1,anchor=/client/test01,depth=2,width=100,files=100,size=4M,shared=yes
fwd=format,threads=24,xfersize=1m
fwd=default,xfersize=1m,fileio=sequential,fileselect=sequential,operation=write,threads=24
fwd=fwd1,fsd=fsd1,host=hd1
fwd=fwd2,fsd=fsd1,host=hd2
fwd=fwd3,fsd=fsd1,host=hd3
fwd=fwd4,fsd=fsd1,host=hd4
fwd=fwd5,fsd=fsd1,host=hd5
fwd=fwd6,fsd=fsd1,host=hd6
rd=rd1,fwd=fwd*,fwdrate=max,format=restart,elapsed=600,interval=1
4M sequential read
directory depth 2, single-level directory number 100, single directory file number 100, single file size 4M, IO block size 1M, sequential read
hd=default, vdbench=/root/vdbench50406, user=root, shell=ssh
hd=hd1,system=node1
hd=hd2,system=node2
hd=hd3,system=node3
hd=hd4,system=node4
hd=hd5,system=node5
hd=hd6,system=node6
fsd=fsd1,anchor=/client/test02,depth=2,width=100,files=100,size=4M,shared=yes
fwd=format,threads=24,xfersize=1m
fwd=default,xfersize=1m,fileio=sequential,fileselect=sequential,operation=read,threads=24
fwd=fwd1,fsd=fsd1,host=hd1
fwd=fwd2,fsd=fsd1,host=hd2
fwd=fwd3,fsd=fsd1,host=hd3
fwd=fwd4,fsd=fsd1,host=hd4
fwd=fwd5,fsd=fsd1,host=hd5
fwd=fwd6,fsd=fsd1,host=hd6
rd=rd1,fwd=fwd*,fwdrate=max,format=restart,elapsed=600,interval=1
8M mixed reading and writing
, directory depth 2, number of single-level directories 100, number of files in a single directory 100, single file size 8M, IO block size 1M, mixed reading and writing (reading and writing ratio is 6:4) hd=default,vdbench=/
root /vdbench50406,user=root,shell=ssh
hd=hd1,system=node1
hd=hd2,system=node2
hd=hd3,system=node3
hd=hd4,system=node4
hd=hd5,system=node5
hd=hd6,system=node6
fsd=fsd1,anchor=/client/test03,depth=2,width=100,files=100,size=8M,shared=yes
fwd=format,threads=24,xfersize=1m
fwd=default,xfersize=1m,fileio=random,fileselect=random,rdpct=60,threads=24
fwd=fwd1,fsd=fsd1,host=hd1
fwd=fwd2,fsd=fsd1,host=hd2
fwd=fwd3,fsd=fsd1,host=hd3
fwd=fwd4,fsd=fsd1,host=hd4
fwd=fwd5,fsd=fsd1,host=hd5
fwd=fwd6,fsd=fsd1,host=hd6
rd=rd1,fwd=fwd*,fwdrate=max,format=restart,elapsed=600,interval=1
One, Q&A
1、open failed for xxx
- Problem Description:
During the test using vdbench, the test is interrupted, and the error message is as follows:
19:37:41.155 19:37:44.813 error: 20
19:37:41.155 19:37:44.813 file_open(), open /trash/64M/vdb.1_47.dir/vdb.2_24.dir/vdb_f0002.file failed
19:37:41.155 19:37:44.814 error: 20
19:37:41.155 19:37:44.814 file_open(), open /trash/64M/vdb.1_47.dir/vdb.2_24.dir/vdb_f0003.file failed
19:37:41.156 19:37:44.814 Maximum native memory allocation: 1,048,576; Current allocation: 1,048,576
19:37:41.156 19:37:44.814 Maximum native memory allocation: 1,048,576; Current allocation: 1,048,576
19:37:41.156 19:37:44.814 error: 20
19:37:41.156 19:37:44.814
19:37:41.156 19:37:44.814 open failed for /trash/64M/vdb.1_47.dir/vdb.2_24.dir/vdb_f0002.file
19:37:41.156 19:37:44.814
19:37:41.158 java.lang.RuntimeException: open failed for /trash/64M/vdb.1_47.dir/vdb.2_24.dir/vdb_f0002.file
- Cause Analysis:
By default, vdbench limits the amount of memory used by java. During the test period, the running memory of java is insufficient to cause the test to be interrupted abnormally.
- Treatment measures:
Increase the java memory parameter of vdbench running script
[root@node40 ~] cat vdbench50406/vdbench
if [ “$1” == “SlaveJvm” ]; then
$java -client -Xmx10240m -Xms128m -cp $cp Vdb.SlaveJvm $*
exit $?
else
$java -client -Xmx10240m -Xms64m -cp $cp Vdb.Vdbmain $*
exit $?
fi
Note: Xmx indicates the largest memory allocation pool of jvm, Xms indicates that the initial memory allocation pool
process initially starts with Xms memory size, when the free heap memory is less than 40%, jvm will increase the memory to Xmx; when the free heap memory is greater than 70%, jvm will reduce the memory to Xms; in order to avoid the JVM to reallocate memory after each GC, you can set Xmx and Xms to the same parameter value
2、linux/aarch64.so does not exist
- Problem Description:
The arm server runs vdbench, and there is a shared library mismatch problem linux/aarch64.so does not exist, prompting that manual compilation is required
[root@node163 vdbench50407] ./vdbench -t
14:07:47.486 Created output directory ‘/root/vdbench50407/output’
14:07:47.737 input argument scanned: ‘-f/tmp/parmfile’
14:07:47.866 Starting slave: /root/vdbench50407/vdbench SlaveJvm -m localhost -n localhost-10-220209-14.07.47.444 -l localhost-0 -p 5570
14:07:47.890
14:07:47.890 File /root/vdbench50407/linux/aarch64.so does not exist.
14:07:47.891 This may be an OS that a shared library currently
14:07:47.891 is not available for. You may have to do your own compile.
14:07:47.891 t: java.lang.UnsatisfiedLinkError: Can’t load library: /root/vdbench50407/linux/aarch64.so
14:07:47.892
14:07:47.892 Loading of shared library /root/vdbench50407/linux/aarch64.so failed.
14:07:47.892 There may be issues related to a cpu type not being
14:07:47.892 acceptable to Vdbench, e.g. MAC PPC vs. X86
14:07:47.893 Contact me at the Oracle Vdbench Forum for support.
14:07:47.893
14:07:48.395
14:07:48.396 Failure loading shared library
14:07:48.396
java.lang.RuntimeException: Failure loading shared library
at Vdb.common.failure(common.java:350)
at Vdb.common.get_shared_lib(common.java:1103)
at Vdb.Native.<clinit>(Native.java:31)
at Vdb.common.signal_caller(common.java:737)
at Vdb.ConnectSlaves.connectToSlaves(ConnectSlaves.java:98)
at Vdb.Vdbmain.masterRun(Vdbmain.java:814)
at Vdb.Vdbmain.main(Vdbmain.java:628)
- Cause Analysis:
The dynamic library file linux/linux64.so in the root directory of vdbench is compiled based on x86, and the dynamic library file linux/linux64.so needs to be recompiled based on aarch64
[root@node163 vdbench50407] ls linux/
config.sh linux32.so linux64.so sparc64.so
[root@node163 vdbench50407] file linux/linux64.so
linux/linux64.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=34a31f32956f21153c372a95e73c02e84ddd29f8, not stripped
- Treatment measures:
Download vdbench50407 source code package, download address: vdbench50407.src.zip
decompress source code package, create linux directory
mkdir vdbench50407-src
unzip vdbench50407.src.zip -d vdbench50407-src/
cd vdbench50407-src/src/
mkdir linux
Enter the Jni directory and modify the make.linux file. The main modifications are as follows:
– Specify the vdbench source code path (vdb=xx)
– Specify the jdk path (java=xx), usually under the /usr/lib/jvm/ path
– Delete the full text – m32 and -m64 characters
cd Jni/
sed -i ‘svdb=$mine/vdbench504vdb=/root/vdbench50407-src/srcg’ make.linux
sed -i ‘sjava=/net/sbm-240a.us.oracle.com/export/swat/swat_java/linux/jdk1.5.0_22/java=/usr/lib/jvm/java-1.8.0-openjdk/g’ make.linux
sed -i ‘s/-m32//g’ make.linux
sed -i ‘s/-m64//g’ make.linux
Execute the source code compilation operation.
After the successful execution of /make.linux, the linux32.so and linux64.so files will be generated in the .../linux directory, copy the linux64.so file to the root directory of the vdbench test tool linux/aarch64.so, and run again vdbench test
[root@node163 Jni] ./make.linux
target directory: /root/vdbench50407-src/src
Compiling 32 bit
Linking 32 bit
Compiling 64 bit
Linking 64 bit
[root@node163 Jni] ll …/linux/
total 160
-rwxrwxrwx 1 root root 78656 Feb 9 14:54 linux32.so
-rwxrwxrwx 1 root root 78656 Feb 9 14:54 linux64.so
[root@node163 Jni] cp …/linux/linux64.so /root/vdbench50407/linux/aarch64.so
3、Raw device ‘sd=sd1,lun=.\G:’ does not exist, or no permissions.
- Problem Description:
Using vdbench to test the performance of the bare disk under the windows system, the script fails to run, prompting Raw device 'sd=sd1,lun=\.\G:' does not exist, or no permissions.
D:\vdbench50406>more raw-test.txt
sd=sd1,lun=\.\G:
wd=wd1,sd=sd1,seekpct=0,rdpct=0,xfersize=1M
rd=rd1,wd=wd1,iorate=max,maxdata=100M,elapsed=64800,warmup=30,interval=5
D:\vdbench50406>vdbench.bat -f raw-test.txt
Copyright © 2000, 2016, Oracle and/or its affiliates. All rights reserved.
Vdbench distribution: vdbench50406 Wed July 20 15:49:52 MDT 2016
For documentation, see ‘vdbench.pdf’.
17:35:24.328 input argument scanned: ‘-fraw-test.txt’
17:35:24.375 Starting slave: D:\vdbench50406\vdbench SlaveJvm -m localhost -n localhost-10-220119-17.35.24.293 -l localhost-0 -p 5570
17:35:24.834 All slaves are now connected
17:35:25.263 Raw device ‘sd=sd1,lun=\.\G:’ does not exist, or no permissions.
17:35:25.764
17:35:25.764 Please check above failures
17:35:25.765
java.lang.RuntimeException: Please check above failures
at Vdb.common.failure(common.java:335)
at Vdb.InfoFromHost.matchDataWithSds(InfoFromHost.java:674)
at Vdb.InfoFromHost.receiveInfoFromHost(InfoFromHost.java:485)
at Vdb.SlaveOnMaster.processSlave(SlaveOnMaster.java:151)
at Vdb.SlaveOnMaster.run(SlaveOnMaster.java:42)
- Treatment measures:
1. Check whether the bare disk path is correct. For example, to test the performance of the G disk bare disk, specify the test parameter as lun=\.\G:
2. Check whether there is execution permission, and the cmd window needs to be run as an administrator
4、Minimum required Java version for Vdbench is 1.7.0
- Problem Description:
Running vdbench under Linux fails, prompting Minimum required Java version for Vdbench is 1.7.0; Vdbench terminated.
root@node65:/home/vdbench50406 ./vdbench -t
Copyright © 2000, 2016, Oracle and/or its affiliates. All rights reserved.
Vdbench distribution: vdbench50406 Wed July 20 15:49:52 MDT 2016
For documentation, see ‘vdbench.pdf’.
-
Minimum required Java version for Vdbench is 1.7.0;
-
You are currently running 11.0.9.1
-
Vdbench terminated.
CTRL-C requested. vdbench terminating
- Treatment measures:
Check the official instructions, the previous version of vdbench50407 has java detection function (vdbench50407 version has been removed), and the version can be upgraded to vdbench50407 without changing the java version of the existing environment.
// Removed as per 50407 because of java 1.10.x
//checkJavaVersion();
//....
private static void checkJavaVersion()
{
if (common.get_debug(common.USE_ANY_JAVA))
return;
if (!JVMCheck.isJREValid(System.getProperty("java.version"), 1, 7, 0))
{
System.out.print("*\n*\n*\n");
System.out.println("* Minimum required Java version for Vdbench is 1.7.0; \n" +
"* You are currently running " + System.getProperty("java.version") +
"\n* Vdbench terminated.");
System.out.println("*\n*\n*\n");
System.exit(-99);
}
}
5. Running vdbench online is interrupted
- Problem Description:
Interruption occurs when running vdbench online under Linux
- Treatment measures:
Due to security hardening, the ssh connection will be automatically disconnected after timeout, edit /etc/ssh/sshd_config
Remove
#ClientAliveInterval 0
#ClientAliveCountMax 3
and change ClientAliveInterval 0 to ClientAliveInterval XX before the following two sentences
Unit s, modified according to vdbench online running time
After the modification is complete, restart the sshd service
systemctl restart sshd
Supplementary instructions on using vdbench for data consistency verification
Recommended parameter: -jn
The parameter to open the vdbench verification data is -v or -j. This process will record a log for each write operation for subsequent verification.
If the -v parameter is used, the generated verification log will be directly saved in the memory; if the -j parameter is used, a verification log file will be generated. In the second verification, -jr can restore the log for verification. -v is directly recorded in the memory, which is faster, but if the storage system restarts or the memory is cleaned up, the verification log recorded by the -v parameter will be lost; -j is directly written to the disk, which is safe but the speed will be reduced. Slow down, you can choose -jn at this time, write to the disk asynchronously, the speed and safety are guaranteed to a certain extent. Principle of data verification
The workflow of data verification is as follows: each first write operation in the storage system is recorded in a table, assuming that the block size of the write operation is 1m, then the two items contained in every 512 bytes in this block size – 8 bytes of logical byte address (LBA) and a byte of data verification key value (the mark is the number of writes, the range is 0-125, 00 represents the creation of writes, 01 represents the first overwrite, and By analogy, when it reaches 126 and returns to 00, a new round) will be recorded, this process is to generate a verification log; the second rerun of the script (using the parameter -jr or -vr) will be based on the log recorded for the first time Perform data verification vdbench to filter redundant system logs