In capacity testing, "quantity control" is very important. JMeter controls the pressure according to the number of threads, but the indicators in the pressure test targets we set are often throughput (QPS/TPS), which gives Testers have brought inconvenience. They have to adjust the number of threads while observing the level of QPS/TPS. In order to solve this problem, JMeter provides a plug-in for throughput timer. We can set the throughput limit to achieve Limit QPS/TPS to achieve the effect of quantity control.
The above approach can ensure that the throughput is controlled at a fixed value, but it is not enough. In practice, we hope that the throughput can be adjusted at any time when each pressure test is executed. For example, under a certain pressure, the service capacity does not Question, we hope to add some pressure without stopping the pressure measurement. How to implement such a function?
1. Dynamic throughput scheme
The solution provided is also very simple. It is still based on a fixed throughput timer (also called a constant throughput timer). The basic implementation principle is to set the throughput limit value as a global variable (${__P(throughput, 99999999)}, throughput is a variable, and gives a large default value), using the BeanShell function of JMeter, by executing external commands, injecting specific values at runtime to achieve the purpose of dynamically adjusting throughput. The specific implementation is as follows:
First run the Beanshell service and open ports 9000 and 9001:
Use JMeter Beanshell as the server to execute Beanshell commands. Dynamically update the previously defined "throughput" parameter by calling the beanshell function. Beanshell is a Java source code interpreter built into JMeter
Uncomment the line for beanshell.server.port on jmeter.properties
#---------------------------------------------------------------------------
# BeanShell configuration
#---------------------------------------------------------------------------
# BeanShell Server properties
#
# Define the port number as non-zero to start the http server on that port
beanshell.server.port=9000
# The telnet server will be started on the next port
#
# Define the server initialisation file
beanshell.server.file=../extras/startup.bshRestart jmeter, you can see the startup log of the port:
Or open cmd in the home directory to view the port running status:
Netstat -ano | findstr “900”
You can see that ports 9000 and 9001 are monitored, port 9000 will be used for http access, and port 9001 will be used for telnet access (both ports must be open under the firewall). Then we can implant our BeanShell script and global variables by accessing the BeanShell service through an external command:
java -jar <jmeter_path>/lib/bshclient.jar localhost 9000 update.bsh <参数>The local test uses localhost, and the actual application can be replaced by IP (the distribution node starts this port by starting jmeter-server, so the IP is for the node IP, and multiple nodes need to be adjusted separately).
The custom throughput update script update.bsh is as follows:
import org.apache.jmeter.util.JMeterUtils;
getprop(p){ // get a JMeter property
return JMeterUtils.getPropDefault(p,"");
}
setprop(p,v){ // set a JMeter property
print("Setting property '"+p+"' to '"+v+"'.");
JMeterUtils.getJMeterProperties().setProperty(p, v);
}
setprop("throughput", args[0]);In fact, we can also unpack the source code of Jmeter's BeanShellClient, so that we can understand the relationship between these parameters (3 mandatory parameters, the fourth and above are the parameters to be dynamically changed), and the use of port 9001 (telnet), The following code:
private static final int MINARGS = 3;
public static void main(String [] args) throws Exception{
if (args.length < MINARGS){
System.out.println("Please provide "+MINARGS+" or more arguments:");
System.out.println("serverhost serverport filename [arg1 arg2 ...]");
System.out.println("e.g. ");
System.out.println("localhost 9000 extras/remote.bsh apple blake 7");
return;
}
String host=args[0];
String portString = args[1];
String file=args[2];
int port=Integer.parseInt(portString)+1;// convert to telnet port
System.out.println("Connecting to BSH server on "+host+":"+portString);
try (Socket sock = new Socket(host,port);
InputStream is = sock.getInputStream();
OutputStream os = sock.getOutputStream()) {
SockRead sockRead = new SockRead(is);
sockRead.start();
sendLine("bsh.prompt=\"\";", os);// Prompt is unnecessary
sendLine("String [] args={", os);
for (int i = MINARGS; i < args.length; i++) {
sendLine("\"" + args[i] + "\",\n", os);
}
sendLine("};", os);
int b;
try (BufferedReader fis = Files.newBufferedReader(Paths.get(file))) {
while ((b = fis.read()) != -1) {
os.write(b);
}
}
sendLine("bsh.prompt=\"bsh % \";", os);// Reset for other users
os.flush();
sock.shutdownOutput(); // Tell server that we are done
sockRead.join(); // wait for script to finish
}
}Let's test it now, the test can simulate high throughput (excessive requests will be discarded by the controller, and too few requests will not show the control effect), I set up 2000 concurrent threads (prefer to do more tasks and less tasks), and then enter Jmeter In the directory, call the command to modify the throughput as follows:
java -jar lib/bshclient.jar localhost 9000 update.bsh 1200
You can see the effect of the throughput being dynamically limited:
We can repeatedly execute the above command to arbitrarily modify the value of throughput, and we will find that the value of TPS changes accordingly:
2. Throughput control conversion TPS
Since the unit of the fixed throughput timer used above is per minute, which is not intuitive to use, we can convert it to per second (TPS), using the following formula ${__jexl3(${__P(throughput , 999999999)}*60,)} :
Then we do the experiment again, first dynamically set the TPS to 600, and then set it to 1200 after a while:
Looking at the comparison of the results as follows, the throughput per second (TPS) is more in line with the usage habit of performance testing, and the analysis graph will be more intuitive:
3. Dynamic TPS application of thread group
The main idea of the above dynamic adjustment of throughput comes from a paid article by Wu Junlong, a former senior expert on local life in Alibaba, but in fact this function is not new, but many people do not know it yet. Many Jmeter-based performance testing platforms or full-link stress testing platforms use this idea to control the throughput of each stress testing instance on the page. Since the throughput timer can be controlled, we can also use this method. This idea dynamically controls the TPS of the thread group, which is very common in the TPS control implementation of the stress test platform.
Write a beanshell script named tpschange.bsh:
import org.apache.jmeter.util.JMeterUtils;
setprop(p,v){// set your TPS
print("Setting link '"+p+"' tps to '"+v+"'.");
JMeterUtils.getJMeterProperties().setProperty(p, v);
}
if (Integer.parseInt(args[1]) <= Integer.parseInt(args[2])){
setprop(args[0],args[1]);}
else{
setprop(args[0],args[2]);}Prepare a JMX script with global variables (using bzm - Arrivals Thread Group thread group):
Run the command with the number of threads dynamically set:
java -jar %JMETER_HOME%/lib/bshclient.jar localhost 9000 %JMETER_HOME%/tpschange.bsh api1 5 500- %JMETER_HOME%/tpschange.bsh, the beanshell script path, can maintain a unique path, the content remains unchanged, and should be updated in batches through the unified entry when it needs to be changed.
- api1 is the interface ID. This ID should be permanently unique on the entire platform. After the user confirms the adjustment of TPS, this ID needs to be substituted into args[0] to adjust the TPS of the specified interface.
- The number 5 is the expected TPS value. The situation of distributed testing of multiple machines should be considered here. For example, if the user uses 5 machines for distributed testing, the unified conversion formula of the args[1] value in the command should be: TPS input by the user /N (N is the number of pressure measuring machines applied).
- The number 500 is to limit the TPS value of the interface link. The TPS that a single machine can support is limited. The user can customize the input TPS value within a certain range. After the limit value is exceeded, the input TPS should be equal to the limit TPS value, corresponding to For the value of args[2] in the command, the unified conversion formula should be: args[2]=(TPS of the current thread group entered by the user / TotalTps of all thread groups included in the current JMX script) * The TPS of the single-machine limit agreed upon by the stress testing platform.
- The TPS limited by a single machine should be configured as a maintainable global configuration in the stress test platform, which can be tested and fine-tuned according to specific conditions.
The effect of running with the following parameters is as follows:
interfaceA:5TPS>20TPS>3TPS>END
interfaceB:10TPS>30TPS>1TPS>END
Combined with the left and right parts of the figure above, you can see the dynamic adjustment process after the initial TPS. The following is the effect diagram of configuring TPS in the pressure measurement platform:
4. Adjustment of the number of dynamic concurrent threads
The dynamic adjustment process of TPS is described. Have you considered how to adjust the number of concurrent threads? In general, whether it is capacity testing or full-link stress testing, we focus more on TPS, that is, traffic pressure. In traditional thread groups, the number of concurrent threads may be used to simulate concurrent users. Dynamic adjustment is also required. The same idea can be used, we use bzm - Concurrency Thread Group thread group:
Here we can use ${__P(threads, 1)}, the default value is 1 (and replace the throughput in the previous update.bsh script with threads) to set the number of dynamic threads, and then we dynamically pass in the number of threads 1, 5, 10 to see the effect:
Regarding the configuration of the number of dynamic threads, it is not recommended to use the default thread group Thread Group, and the original mechanism may be different (the DynamicThread dynamic thread technology was introduced in the later JMeterPlugins - Standard plug-in, refer to the plug-in source code undera / jmeter-plugins ), the default The Number of Threads of the thread group is the maximum preset number of threads (not the current maximum number of active threads). In the process of script running, modifying the thread value through Beanshell does not take effect in real time (because it belongs to all threads pre-created in memory). , not a Dynamic Thread), you need to stop the script to restart the script to take effect, which loses the meaning of dynamic configuration.
Also explain why the number of concurrent threads (VU) is just a means, not the point of our performance emphasis, first of all to understand from this general formula: TPS=(1/RT)*VU, RT is the response time, 1 is 1 second , if the unit of RT is seconds, then the number in the formula is 1, if the unit of RT is milliseconds, then the number in the formula is 1000, and VU is the number of virtual users corresponding to the number of concurrent threads in JMeter. In the pressure test, the strong indicators we focus on are TPS and RT. VU is only a means of pressure (in order to quickly reach the amount of TPS, such as gradient pressure), and TPS is the value we expect a system to reach. , we directly and dynamically adjust this value to test whether the system can achieve it. At the same time, RT will also know through JMeter's pressure test results under different TPS values. There are a total of 3 variables in the above formula. We know two, then the VU value will naturally be Got it. Therefore, we say that the core indicators of performance testing are TPS and RT.
Five, the realization principle of platform
For the Jmeter-based stress testing platform, the above principles can be used to dynamically update and configure the Jmeter script parameters during the stress testing process (the script running test is not interrupted), but the specific implementation may even be better than the above methods. It's simple, because we don't need to use beanshellServer to establish a connection. We can use Java to directly call Jmeter inside the platform to achieve this, so that we can bypass the Beanshell script and directly modify the global variables of the stress test thread:
import org.apache.jmeter.util.JMeterUtils;
// 设置全局变量
JMeterUtils.setProperty(p, v);
// 获取全局变量
JMeterUtils.getPropDefault(p, "");Generally, the method of calling global variables in Jmeter ${__P(variable, "")} can be completely used to call the configuration of the jmeter.properties configuration file, so changing global variables is the same as changing the configuration. You can load and modify the configuration file of Jmeter and set parameters, as follows:
public void setJmeterProperties() {
String jmeterHomeBin = getJmeterHomeBin();
JMeterUtils.loadJMeterProperties(jmeterHomeBin + File.separator + "jmeter.properties");
JMeterUtils.setJMeterHome(getJmeterHome());
JMeterUtils.initLocale();
Properties jmeterProps = JMeterUtils.getJMeterProperties();
// Add local JMeter properties, if the file is found
String userProp = JMeterUtils.getPropDefault("user.properties", ""); //$NON-NLS-1$
if (userProp.length() > 0) { //$NON-NLS-1$
File file = JMeterUtils.findFile(userProp);
if (file.canRead()) {
try (FileInputStream fis = new FileInputStream(file)) {
Properties tmp = new Properties();
tmp.load(fis);
jmeterProps.putAll(tmp);
} catch (IOException e) {
}
}
}
// Add local system properties, if the file is found
String sysProp = JMeterUtils.getPropDefault("system.properties", ""); //$NON-NLS-1$
if (sysProp.length() > 0) {
File file = JMeterUtils.findFile(sysProp);
if (file.canRead()) {
try (FileInputStream fis = new FileInputStream(file)) {
System.getProperties().load(fis);
} catch (IOException e) {
}
}
}
jmeterProps.put("jmeter.version", JMeterUtils.getJMeterVersion());
// 配置client.rmi.localport，如配置端口60000
if(MasterClientRmiLocalPort() >= 0) {
JMeterUtils.setProperty("client.rmi.localport", Integer.toString(MasterClientRmiLocalPort()));
}
}Therefore, in a platform system, it is no longer technically difficult to dynamically configure variables and modify parameters. If you cooperate with the hook program of Jmeter, you can change the parameters after each request is sent. The changed parameters are called by the stress test thread and take effect. It still depends on the above beanshellServer solution. For beanshellServer, you can look at the code of Jmeter. It is started when the Jmeter program is started, as shown below:
The above startOptionalServers() call is to start beanshellServer. Remember that it must be called after startGui or startNonGui. If the order is wrong, even if it is started, the parameters cannot be changed through beanshellServer. The code for startOptionalServers() is as follows:
private void startOptionalServers() {
int bshport = JMeterUtils.getPropDefault("beanshell.server.port", 0);// $NON-NLS-1$
String bshfile = JMeterUtils.getPropDefault("beanshell.server.file", "");// $NON-NLS-1$ $NON-NLS-2$
if (bshport > 0) {
log.info("Starting Beanshell server ({},{})", bshport, bshfile);
Runnable t = new BeanShellServer(bshport, bshfile);
t.run(); // NOSONAR we just evaluate some code here
}
runInitScripts();
int mirrorPort=JMeterUtils.getPropDefault("mirror.server.port", 0);// $NON-NLS-1$
if (mirrorPort > 0){
log.info("Starting Mirror server ({})", mirrorPort);
try {
Object instance = ClassTools.construct(
"org.apache.jmeter.protocol.http.control.HttpMirrorControl",// $NON-NLS-1$
mirrorPort);
ClassTools.invoke(instance,"startHttpMirror");
} catch (JMeterException e) {
log.warn("Could not start Mirror server",e);
}
}
}The code that calls BeanShellServer is as follows:
package org.apache.jmeter.util;
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* Implements a BeanShell server to allow access to JMeter variables and
* methods.
*
* To enable, define the JMeter property: beanshell.server.port (see
* JMeter.java) beanshell.server.file (optional, startup file)
*
*/
public class BeanShellServer implements Runnable {
private static final Logger log = LoggerFactory.getLogger(BeanShellServer.class);
private final int serverport;
private final String serverfile;
/**
* Constructor which sets the port for this server and the path to an
* optional init file
*
* @param port
* the port for the server to use
* @param file
* the path to an init file, or an empty string, if no init file
* should be used
*/
public BeanShellServer(int port, String file) {
super();
serverfile = file;// can be the empty string
serverport = port;
}
// For use by the server script
static String getprop(String s) {
return JMeterUtils.getPropDefault(s, s);
}
// For use by the server script
static void setprop(String s, String v) {
JMeterUtils.getJMeterProperties().setProperty(s, v);
}
@Override
public void run() {
ClassLoader loader = Thread.currentThread().getContextClassLoader();
try {
Class<?> interpreter = loader.loadClass("bsh.Interpreter");//$NON-NLS-1$
Object instance = interpreter.getDeclaredConstructor().newInstance();
Class<String> string = String.class;
Class<Object> object = Object.class;
Method eval = interpreter.getMethod("eval", string);//$NON-NLS-1$
Method setObj = interpreter.getMethod("set", string, object);//$NON-NLS-1$
Method setInt = interpreter.getMethod("set", string, int.class);//$NON-NLS-1$
Method source = interpreter.getMethod("source", string);//$NON-NLS-1$
setObj.invoke(instance, "t", this );//$NON-NLS-1$
setInt.invoke(instance, "portnum", serverport);//$NON-NLS-1$
if (serverfile.length() > 0) {
try {
source.invoke(instance, serverfile);
} catch (InvocationTargetException ite) {
Throwable cause = ite.getCause();
if (log.isWarnEnabled()) {
log.warn("Could not source, {}. {}", serverfile,
(cause != null) ? cause.toString() : ite.toString());
}
if (cause instanceof Error) {
throw (Error) cause;
}
}
}
eval.invoke(instance, "setAccessibility(true);");//$NON-NLS-1$
eval.invoke(instance, "server(portnum);");//$NON-NLS-1$
} catch (ClassNotFoundException e) {
log.error("Beanshell Interpreter not found");
} catch (Exception e) {
log.error("Problem starting BeanShell server", e);
}
}
}
Therefore, if you really want to modify variables through BeanShellServer like Jmeter, you can completely rewrite the code according to the above logic or directly call the above methods.
Reference documentation:
07 | Tool Evolution: How to Implement a Distributed Pressure Testing Platform - Geek Time