Comparison tool
Mainstream open source performance testing tools have the following main
Point of comparison |
JMETER |
nGrinder |
Gatling | Tsung | result |
Open source | Free, fully open source | Free, fully open source | Free, fully open source | Free, fully open source | = |
Implementation language |
JAVA |
JAVA |
Written in Scala, supports JAVA library | erlang | jmeter=ngrinder>gatling>Tsung |
Use |
C / S or Command |
B/S |
Command | Command | = |
Support for distributed |
master/slave |
controller/agent |
not support | master/slave | jmeter=ngrinder=Tsung>gatling |
Resource Monitoring |
monitor/plugin |
monitor mode, there is a direct source available |
no | Erlang or SNMP protocol can be used to remotely monitor machine and to generate a corresponding graph | jmeter=ngrinder=Tsung>gatling |
Community activity |
Documents perfect, multi-user |
There are Chinese Community |
Have community support | Have community support | jmeter=ngrinder>gatling>Tsung |
The need for coding |
Basic need |
Needs, Jython / Groovy |
Needs, scala | need | jmeter>Tsung>ngrinder=gatling |
Maintenance script |
local |
Built-in SVN |
local | local | = |
Script recording |
Support http proxy recording, support for third-party record |
Plug-ins can be recorded by PTS |
Support http proxy record | Record support script | = |
Ease of use | Mature template, components, controllers directly introduced is relatively easy to use, low programming requirements | Control logic, parameterized, the checkpoint-dependent programming | Scala less familiar people, logic control, parameterization, dependent checkpoint programming | The script is the same format and loadrunner | jmeter>Tsung>ngrinder>gatling |
Protocol Support | Multi-protocol support | http protocol, others need to develop their own extensions | http protocol, others need to develop their own extensions | Multi-protocol support | jmeter>Tsung>ngrinder>gatling |
Scalability |
Increase the plugin, and strong expansion |
Supports plug-ins |
Gatling DSL APT based on a set of open source, easy to extend functionality | Supports plug-ins | = |
installation |
Based jdk, absolute lightweight, with a copy that is |
Two service controller and agent |
Based jdk, absolute lightweight, with a copy that is | You need to install and configure three services | jmeter=gatling>ngrinder>Tsung |
Pressure measurement platform coding amount | Big | small | Great | Great | ngrinder>jmeter>gatling=Tsung |
通过从下面各个维度对比可以看出,工具jmeter都具有优势,但压测平台的开发编码量比较大,nGrinder有现成的压测平台,开发工作量少,但是易用性不如jmeter,脚本需要大量的编码工作,推广难道比较大,两者各有优缺点。
平台开发工作量对比
如下为完全平台化需要完成的核心功能点
比较点 |
JMETER |
nGrinder |
平台开发主要实现功能 | 核心功能点如下
|
核心功能如下
|
可以看出JMETER的工作量要远大于nGrinder
易用性对比
在易用性主要是在脚本的编码工作量方面,通过对比可以看出jmeter无非常明显的优势,前提是测试人员具备一定的编码能力
比较点 |
JMETER |
nGrinder |
参数化 | 通过界面操作完成,简单 | 需要手功编写代码,编码工作量少,简单 |
关联 | 界面操作,各个接口之间的数据可以直接传递,简单 | 各个方法之间的值可以直接传递,简单 |
检查点 | 界面操作完成,简单 | 有检查点的方法调用,简单 |
综合场景 | 有丰富的逻辑控制器,简单 | 需要大量的编码工作,复杂 |
压力机硬件资源消耗对比
压力机对资源的占用对比,已压测某服务为例,使用相同的并发用户数500,对比压力机资源占用可以看出,ngrinder的cpu占用是jmeter的4倍,内存占用是jmeter5倍
比较点 |
JMETER |
nGrinder |
cpu | jmeter只需要1个进程,cpu占用为320% |
agent 如下10个进程CPU占用达到1200%
|
内存 | 进程内存占用为0.96G | agent10个进程内存占用5.3G |
nGrinder 压测时,监控agent资源利用情况
jmeter
测试指标对比
通过对同一服务,相同并发数和测试时间,对比业务指标TPS和平均响应时间结果如下,jmeter的值要远大于nGrinder,后续我们对nGrinder的压测节点进行扩容
最终也只能压测到18000多TPS
比较点 |
JMETER |
nGrinder |
TPS | 24002 |
14778 |
响应时间 | 20ms | 33.1ms |
nGrinder测试结果截图
JMETER 测试结果截图
综合分析
在工具对比方面,jmeter最具优越性,而且目前公司的性能测试都是用jmeter工具,免去替换成平台后需要重新编写测试脚本的问题,nGrinder在平台二次开发上最简单,在易用性方面也比较好(前提是要具备一定的编码能力),但是在调研的过程中发现非常耗压力机资源,在相同并发下cpu占用是jmeter的4倍,内存占用是jmeter5倍以上,同时在测试kong的时候发现无法压测到kong的极限值。所以最终推荐压测平台使用集成jmeter的方式。