After deploying the object storage, we need to conduct performance testing, using cosbench, and then summarize it here.
1. Introduction:
COSBench is a distributed benchmarking tool for testing cloud object storage systems and also allows users to create adapters for additional storage systems.
Consists of two main components
Driver（Load Generator）：
- Responsible for generating workloads and issuing operations to target cloud object storage; performance statistics
- Accessible via http://:18088/driver/index.html
Controller：
- Responsible for coordinating the collective execution of drivers, collecting and aggregating runtime status or benchmark results from driver instances
- Accessible via http://:19088/controller/index.html
The two components can be mixed and deployed on one node, or they can be deployed separately
2. Install dependency packages:
1. java: requires java1.6 version and above
2. nc (nmap-ncat): used to detect service ports
[root@ceph-node1 ~]# yum install java nc
[root@ceph-node1 ~]# java -version #测试java是否可用
openjdk version "1.8.0_312"
OpenJDK Runtime Environment (build 1.8.0_312-b07)
OpenJDK 64-Bit Server VM (build 25.312-b07, mixed mode)
3. Install cosbench:
1. Download, unzip and view
[root@ceph-node1 ~]# wget https://github.com/intel-cloud/cosbench/releases/download/v0.4.2.c4/0.4.2.c4.zip
[root@ceph-node1 ~]# unzip 0.4.2.c4.zip
官方英文使用文档（超详细，可以翻翻看）
[root@ceph-node1 0.4.2.c4]# ll *.pdf
-rw-r--r--. 1 root root 348912 Jul 9 2014 3rd-party-licenses.pdf
-rw-r--r--. 1 root root 985318 Jul 9 2014 COSBenchAdaptorDevGuide.pdf
-rw-r--r--. 1 root root 2516640 Apr 27 2016 COSBenchUserGuide.pdf
服务调用脚本说明
[root@ceph-node1 0.4.2.c4]# ll *.sh
-rw-r--r--. 1 root root 2639 Oct 30 00:25 cli.sh #通过命令行操作cosbench
-rw-r--r--. 1 root root 2944 Apr 27 2016 cosbench-start.sh #其他启动脚本调用的脚本
-rw-r--r--. 1 root root 1423 Dec 30 2014 cosbench-stop.sh #其他启动脚本调用的脚本
-rw-r--r--. 1 root root 727 Apr 27 2016 start-all.sh #在当前节点启动driver和controller组件
-rw-r--r--. 1 root root 724 Apr 27 2016 stop-all.sh #在当前节点停止driver和controller组件
-rw-r--r--. 1 root root 1062 Jul 9 2014 start-controller.sh #在当前节点启动controller
-rw-r--r--. 1 root root 1912 Oct 30 00:22 start-driver.sh #在当前节点启动driver
-rw-r--r--. 1 root root 809 Jul 9 2014 stop-controller.sh #在当前节点关闭controller
-rw-r--r--. 1 root root 1490 Apr 27 2016 stop-driver.sh #在当前节点关闭driver
2. Modify the driver configuration (can be ignored if it is local access)
默认driver配置的页面地址为127.0.0.1，要修改为当前服务器IP，用以其他节点访问
[root@ceph-node1 0.4.2.c4]# cat /root/0.4.2.c4/start-driver.sh
#!/bin/bash
ip=192.168.1.51 #默认为127.0.0.1，将之修改为部署节点的IP，供其他节点访问
num=1
base_port=18088
……
3. Start cosbench
启动前为了保证driver和controller之间交互正常，需要关闭HTTP代理
[root@ceph-node1 0.4.2.c4]# unset http_proxy
启动controller和driver
[root@ceph-node1 0.4.2.c4]# sh start-all.sh
Launching osgi framwork ...
Successfully launched osgi framework!
Booting cosbench driver ...
.
Starting cosbench-log_0.4.2 [OK]
.
Starting cosbench-tomcat_0.4.2 [OK]
Starting cosbench-config_0.4.2 [OK]
Starting cosbench-http_0.4.2 [OK]
Starting cosbench-cdmi-util_0.4.2 [OK]
Starting cosbench-core_0.4.2 [OK]
Starting cosbench-core-web_0.4.2 [OK]
Starting cosbench-api_0.4.2 [OK]
Starting cosbench-mock_0.4.2 [OK]
Starting cosbench-ampli_0.4.2 [OK]
Starting cosbench-swift_0.4.2 [OK]
Starting cosbench-keystone_0.4.2 [OK]
Starting cosbench-httpauth_0.4.2 [OK]
Starting cosbench-s3_0.4.2 [OK]
Starting cosbench-librados_0.4.2 [OK]
Starting cosbench-scality_0.4.2 [OK]
Starting cosbench-cdmi-swift_0.4.2 [OK]
Starting cosbench-cdmi-base_0.4.2 [OK]
Starting cosbench-driver_0.4.2 [OK]
Starting cosbench-driver-web_0.4.2 [OK]
Successfully started cosbench driver!
Listening on port 0.0.0.0/0.0.0.0:18089 ...
Persistence bundle starting...
Persistence bundle started.
----------------------------------------------
!!! Service will listen on web port: 18088 !!!
----------------------------------------------
======================================================
Launching osgi framwork ...
Successfully launched osgi framework!
Booting cosbench controller ...
.
Starting cosbench-log_0.4.2 [OK]
.
Starting cosbench-tomcat_0.4.2 [OK]
Starting cosbench-config_0.4.2 [OK]
Starting cosbench-core_0.4.2 [OK]
Starting cosbench-core-web_0.4.2 [OK]
Starting cosbench-controller_0.4.2 [OK]
Starting cosbench-controller-web_0.4.2 [OK]
Successfully started cosbench controller!
Listening on port 0.0.0.0/0.0.0.0:19089 ...
Persistence bundle starting...
Persistence bundle started.
----------------------------------------------
!!! Service will listen on web port: 19088 !!!
----------------------------------------------
4. Verification (check java process and port)
[root@ceph-node1 0.4.2.c4]# netstat -nplt |grep 18088
tcp6 0 0 :::18088 :::* LISTEN 6084/java
[root@ceph-node1 0.4.2.c4]# netstat -nplt |grep 19088
tcp6 0 0 :::19088 :::* LISTEN 6235/java
[root@ceph-node1 0.4.2.c4]# ps aux |grep java
root 6084 5.3 4.4 3562620 178000 pts/1 Sl 02:26 0:10 java -Dcosbench.tomcat.config=conf/driver-tomcat-server.xml -server -cp main/org.eclipse.equinox.launcher_1.2.0.v20110502.jar org.eclipse.equinox.launcher.Main -configuration conf/.driver -console 18089
root 6235 5.2 3.8 3555108 153032 pts/1 Sl 02:26 0:10 java -Dcosbench.tomcat.config=conf/controller-tomcat-server.xml -server -cp main/org.eclipse.equinox.launcher_1.2.0.v20110502.jar org.eclipse.equinox.launcher.Main -configuration conf/.controller -console 19089
root 6327 0.0 0.0 112812 980 pts/1 S+ 02:29 0:00 grep --color=auto java
Use cli.sh to view startup information
[root@ceph-node1 0.4.2.c4]# sh cli.sh info
Drivers:
driver1 http://127.0.0.1:18088/driver
Total: 1 drivers
Active Workloads:
Total: 0 active workloads
From the above information, we know that both controller and driver have been started.
报错提示：
如果不安装nc，会报错
which: no nc in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin)
No appropriate tool found to detect cosbench driver status.
如果不安装java，会报
Ncat: Connection refused.
Visit http://:19088/controller/index.html to verify
4. Configure multiple drivers in cosbench (can be ignored if you do not use multiple drivers)
The controller can manage multiple drivers. In addition to the default one driver, multiple drivers can be deployed to average the load. Driver has multiple deployment methods
- Can be deployed on the same node as the controller
- Can be deployed on the same node using different ports
- Deployed on other nodes
Here is the experimental environment, use method 2 experiment
修改controller配置文件：
[root@ceph-node1 0.4.2.c4]# cat conf/controller.conf
[controller]
drivers = 3
log_level = INFO
log_file = log/system.log
archive_dir = archive
[driver1]
name = driver1
url = http://192.168.1.51:18088/driver
[driver2]
name = driver2
url = http://192.168.1.51:18188/driver
[driver3]
name = driver3
url = http://192.168.1.51:18288/driver
1. Start the basic controller and driver
[root@ceph-node1 0.4.2.c4]# sh start-all.sh
…………
Successfully started cosbench driver!
Listening on port 0.0.0.0/0.0.0.0:18089 ...
Persistence bundle starting...
Persistence bundle started.
----------------------------------------------
!!! Service will listen on web port: 18088 !!!
----------------------------------------------
…………
Successfully started cosbench controller!
Listening on port 0.0.0.0/0.0.0.0:19089 ...
Persistence bundle starting...
Persistence bundle started.
----------------------------------------------
!!! Service will listen on web port: 19088 !!!
----------------------------------------------
2. Start multiple drivers (using different local ports)
The default driver uses port 18088, and other ports here use 18188/18288
命令格式：sh start-driver.sh n ip base-port （默认会创建出来两个driver节点）
参数详解：
n：第几个节点（不重复即可）
ip：部署节点ip
base-port：部署使用的端口
启动
[root@ceph-node1 0.4.2.c4]# sh start-driver.sh 2 192.168.1.51 18188
………………
Successfully started cosbench driver!
Listening on port 0.0.0.0/0.0.0.0:18189 ...
Persistence bundle starting...
Persistence bundle started.
----------------------------------------------
!!! Service will listen on web port: 18188 !!!
----------------------------------------------
……………………
Successfully started cosbench driver!
Listening on port 0.0.0.0/0.0.0.0:18289 ...
Persistence bundle starting...
Persistence bundle started.
----------------------------------------------
!!! Service will listen on web port: 18288 !!!
----------------------------------------------
3. Check the process (one controller, three drivers)
[root@ceph-node1 0.4.2.c4]# ps aux |grep java
root 8103 2.8 4.3 3568764 175888 pts/1 Sl 03:35 0:13 java -Dcosbench.tomcat.config=conf/driver-tomcat-server.xml -server -cp main/org.eclipse.equinox.launcher_1.2.0.v20110502.jar org.eclipse.equinox.launcher.Main -configuration conf/.driver -console 18189
root 8263 2.4 4.2 3564640 172748 pts/1 Sl 03:35 0:11 java -Dcosbench.tomcat.config=conf/driver-tomcat-server.xml -server -cp main/org.eclipse.equinox.launcher_1.2.0.v20110502.jar org.eclipse.equinox.launcher.Main -configuration conf/.driver -console 18289
root 8431 18.5 4.0 3563672 161632 pts/1 Sl 03:42 0:11 java -Dcosbench.tomcat.config=conf/driver-tomcat-server.xml -server -cp main/org.eclipse.equinox.launcher_1.2.0.v20110502.jar org.eclipse.equinox.launcher.Main -configuration conf/.driver -console 18089
root 8582 15.0 3.5 3550292 142732 pts/1 Sl 03:42 0:08 java -Dcosbench.tomcat.config=conf/controller-tomcat-server.xml -server -cp main/org.eclipse.equinox.launcher_1.2.0.v20110502.jar org.eclipse.equinox.launcher.Main -configuration conf/.controller -console 19089
root 9092 0.0 0.0 112812 976 pts/1 S+ 03:43 0:00 grep --color=auto java
4. View the page
5. Configure the pressure test file
The default template test configuration file: conf/s3-config-sample.xml
[root@ceph-node1 0.4.2.c4]# cat conf/s3-config-sample.xml
<?xml version="1.0" encoding="UTF-8" ?>
<workload name="s3-sample" description="sample benchmark for s3">
<storage type="s3" config="accesskey=<accesskey>;secretkey=<scretkey>;proxyhost=<proxyhost>;proxyport=<proxyport>;endpoint=<endpoint>" />
<workflow>
<workstage name="init">
<work type="init" workers="1" config="cprefix=s3testqwer;containers=r(1,2)" />
</workstage>
<workstage name="prepare">
<work type="prepare" workers="1" config="cprefix=s3testqwer;containers=r(1,2);objects=r(1,10);sizes=c(64)KB" />
</workstage>
<workstage name="main">
<work name="main" workers="8" runtime="30">
<operation type="read" ratio="80" config="cprefix=s3testqwer;containers=u(1,2);objects=u(1,10)" />
<operation type="write" ratio="20" config="cprefix=s3testqwer;containers=u(1,2);objects=u(11,20);sizes=c(64)KB" />
</work>
</workstage>
<workstage name="cleanup">
<work type="cleanup" workers="1" config="cprefix=s3testqwer;containers=r(1,2);objects=r(1,20)" />
</workstage>
<workstage name="dispose">
<work type="dispose" workers="1" config="cprefix=s3testqwer;containers=r(1,2)" />
</workstage>
</workflow>
</workload>
Parameter Description:
- workload name: The name of the task displayed during the test, which can be defined here
- description: description information, you can define it yourself here, the example is written (10 concurrently write 100 4kb files)
- storage type: storage type, here it can be configured as s3
- config: connection information configuration, accesskey and secretkey fill in the user’s key; proxyhost and proxyport fill in the IP and port of the http proxy host, fill in if necessary, if not, you can delete it; endpoint fill in the access address of rgw
- workstage name: the tasks to be executed, cosbench is executed sequentially in stages
- init: initialization phase, mainly to create buckets,
- workers: Indicates how many worker threads are enabled when executing this stage. Bucket creation will not be calculated as performance, so single threading is also acceptable;
- config: Configures the name prefix of the bucket bucket;
- containers: indicates the number of polls. In the above example, buckets with s3testqwer as the prefix and suffixes of 1 and 2 will be created.
- prepare: initialization phase, the data written to the bucket is configured, workers, config and containers are the same as the init phase
- object: indicates how many objects are written in one round, and the size of the object. In the example, objects=r(1,10);sizes=c(64)KB" means writing 1 to 10 objects with a size of 64KB
- main stage: the stage for testing,
- workers: how many threads to open to perform operations
- runtime: indicates the running time, the default time is seconds (can be used in conjunction with totalOps or alone)
- totalOps: Indicates how many operations are performed. For example, when the value is 10, object=(1,00), then 100 objects are originally to be created. After adding totalOps, it is necessary to create 10 times in a loop from 1-100, creating a total of 1000 files, but the actual number of files created is still only 100, like overwriting creation
- operation type: operation type, which can be read, write, delete, etc.
- Ratio: Indicates the proportion of operations occupied by this operation. For example, in the above example, the proportion of read and write in the test is 80%, and the proportion of write is 20%;
- config: Configure the prefix and suffix information of the bucket. Note that the sizes of write can be modified according to the actual test.
- cleanup: The environment cleanup phase mainly deletes the data in the bucket to ensure that the data after testing will not be retained in the cluster.
- dispose: This stage is to delete the bucket
The main method is to write the read and write tasks in a work, and the mixed read and write
can also skip the prepare and main stages directly, directly perform the write operation, and then directly read
Example:
<workstage name="write">
<work name="write" workers="64" totalOps="100000">
<operation type="write" config="cprefix=cosbench;containers=r(1,1);objects=r(1,100000);sizes=c(4)KB" />
</work>
</workstage>
<workstage name="read">
<work name="read" workers="64" totalOps="100000">
<operation type="read" config="cprefix=cosbench;containers=r(1,1);objects=r(1,100000)" />
</work>
</workstage>
Write test files based on template files
[root@ceph-node1 0.4.2.c4]# cat conf/s3-config-test.xml
<?xml version="1.0" encoding="UTF-8" ?>
<workload name="s3-test1" description="10-100-4kb-test">
<storage type="s3" config="accesskey=OZGQE03XOX2DDD77M9OX;secretkey=yvqhvuWg7mGKWfx3JGggtgM6LdDnkeJhxj8MHJeg;endpoint=http://192.168.1.51:7480" />
<workflow>
<workstage name="init">
<work type="init" workers="1" config="cprefix=cosbench;containers=r(1,1)" />
</workstage>
<workstage name="prepare">
<work type="prepare" workers="1" config="cprefix=cosbench;containers=r(1,1);objects=r(1,100);sizes=c(4)KB" />
</workstage>
<workstage name="main">
<work name="main" workers="8" runtime="30">
<operation type="read" ratio="80" config="cprefix=cosbench;containers=u(1,1);objects=u(1,100)" />
<operation type="write" ratio="20" config="cprefix=cosbench;containers=u(1,1);objects=u(100,200);sizes=c(4)KB" />
</work>
</workstage>
<workstage name="cleanup">
<work type="cleanup" workers="1" config="cprefix=cosbench;containers=r(1,1);objects=r(1,200)" />
</workstage>
<workstage name="dispose">
<work type="dispose" workers="1" config="cprefix=cosbench;containers=r(1,1)" />
</workstage>
</workflow>
</workload>
6. Execution startup
Two ways to start
1. Start by script
[root@ceph-node1 0.4.2.c4]# sh cli.sh submit conf/s3-config-test.xml
Accepted with ID: w3
2. Start by uploading the configured xml file on the page
Upload files and execute submit
7. Results analysis
1. Running tasks
2. Ended tasks
3. Click [view details] in the end task column to view the test results.
Description of each parameter of the test result:
- Op-Type: operation type
- Op-Count: total number of operations
- Byte-Count: the total amount of data generated by the operation
- Avg-ResTime: the average time generated by the operation
- Avg-ProcTime: The average time of the operation. This is the main delay reference. It reflects the average delay of each request.
- Throughput: Throughput, which mainly reflects the degree of concurrency of operations and is also an important reference (operations completed in one second)
- Bandwidth: Bandwidth, reflecting the average bandwidth during operation (amount of data transmitted per second)
- Succ-Ratio: The success rate of the operation
Mainly look at the last four values! !
You can also view each defined task to see if there are any failed tasks.
End……