Best practices for using Alibaba Cloud performance testing tool JMeter to stress test RocketMQ in scenarios

12.23 Yuanchuanghui·Shanghai Station, let’s talk about LLM infrastructure

Author: Morimoto

Demand background

Before a new business goes online, we usually need to conduct a stress test on different middleware of the system to find the upper limit of the traffic that the middleware can withstand under the current configuration, so as to determine the current limiting rules of the upstream link to protect the system from collapse due to sudden traffic. Alibaba Cloud PTS's JMeter stress test can support users to upload custom JMeter scripts, and use the powerful distributed stress test capabilities of PTS to perform stress tests on different middleware of the system according to custom logic. Below, we will take the JMeter5.5 and RocketMQ5.0 series as examples to introduce in detail how to stress test RocketMQ using the JMeter scenario of PTS.

Preconditions

JMeter is installed locally.
RocketMQ has been deployed on Alibaba Cloud ECS (this article chose an ECS with 8C32G specifications).
The PTS service has been activated on Alibaba Cloud.

Stress testing process

JMeter provides a highly extensible JavaSampler. We can customize the logic executed in the JavaSampler by inheriting the AbstractJavaSamplerClient class to implement stress testing of RocketMQ.

Step 1: Create a Maven project and introduce dependencies

Create a new Maven project and introduce the following dependencies in the pom file:

<dependency>
  <groupId>org.apache.jmeter</groupId>
  <artifactId>ApacheJMeter_java</artifactId>
  <version>5.5</version>
  <scope>provided</scope>
</dependency>
<dependency>
  <groupId>org.apache.rocketmq</groupId>
  <artifactId>rocketmq-client</artifactId>
  <version>4.9.5</version>
</dependency>

ApacheJMeter_java is a dependency of JMeter JavaSampler, and rocketmq-client is a client dependency of RocketMQ (version 4.x is used here because the client of version 4.x is compatible with the server instance of version 5.x, but the version 5.x The client is not compatible with server instances of version 4.x and can be adjusted according to your own needs). Among them, it should be noted that the scope of the dependency of ApacheJMeter_java is defined as provided. The JAR package is already available in the lib/ext directory of JMeter, so it is not necessary to package the dependency together.

Introduce the maven-assembly-plugin plug-in into the pom file, and use the "jar-with-dependencies" packaging method here to package the project's required dependencies and project code into the same JAR package. You can then only upload the JAR package to PTS. In the JMeter environment, there is no need to upload multiple dependent JAR packages:

<build>
  <finalName>jmeter-rocketmq4</finalName>
  <plugins>
    <plugin>
      <artifactId>maven-assembly-plugin</artifactId>
      <version>3.4.2</version>
      <configuration>
        <!-- 打包方式 -->
        <descriptorRefs>
          <descriptorRef>jar-with-dependencies</descriptorRef>
        </descriptorRefs>
      </configuration>
      <executions>
        <execution>
          <id>make-assembly</id>
          <phase>package</phase>
          <goals>
            <goal>single</goal>
          </goals>
        </execution>
      </executions>
    </plugin>
  </plugins>
</build>

Step 2: Create a new subclass of AbstractJavaSamplerClient and rewrite related methods

The AbstractJavaSamplerClient class inherits the JavaSamplerClient interface, which contains four methods: setupTest, runTest, teardownTest and getDefaultParameters:

setupTest

JMeter will create a JavaSamplerClient implementation instance for each thread in the test. When the test starts, setupTest will be called on the JavaSamplerClient instance of each thread to initialize the client. In this case, the RocketMQ producer is initialized.
runTest

Each thread will call the runTest method once per iteration. In this example, the message sending method and the setting logic of the sampling results need to be defined in the runTest method.
teardownTest

After iterating the set number of times or time, this method will be executed. In this example, the producer needs to be closed in this method.
getDefaultParameters

This method defines a parameter list, which is passed to the above method through the JavaSamplerContext. The parameters defined in this method can be set in the GUI interface of the JMeter JavaRequest Sampler. In this example, the broker address and topic name of RocketMQ need to be defined. , message key, message content and other parameters.

The reference for creating a new subclass is as follows:

import java.nio.charset.StandardCharsets;
import org.apache.jmeter.config.Arguments;
import org.apache.jmeter.protocol.java.sampler.AbstractJavaSamplerClient;
import org.apache.jmeter.protocol.java.sampler.JavaSamplerContext;
import org.apache.jmeter.samplers.SampleResult;
import org.apache.rocketmq.client.exception.MQBrokerException;
import org.apache.rocketmq.client.exception.MQClientException;
import org.apache.rocketmq.client.producer.DefaultMQProducer;
import org.apache.rocketmq.client.producer.SendResult;
import org.apache.rocketmq.common.message.Message;
import org.apache.rocketmq.remoting.exception.RemotingException;

public class JavaSamplerForRocketMQ extends AbstractJavaSamplerClient {
    private DefaultMQProducer producer;
    private static final String NAME_SRV_ADDRESS = "nameSrvAddress";
    private static final String TOPIC = "topic";
    private static final String PRODUCER_GROUP = "producer group";
    private static final String MSG_BODY = "messageBody";
    private static final String MSG_KEY = "messageKey";
    private static final String MSG_TAG = "messageTag";
    private static final String ERROR_CODE = "500";

    @Override
    public void setupTest(JavaSamplerContext javaSamplerContext) {

        try {
            // 初始化producer
            producer = new DefaultMQProducer(javaSamplerContext.getParameter(PRODUCER_GROUP));
            producer.setNamesrvAddr(javaSamplerContext.getParameter(NAME_SRV_ADDRESS));
            producer.start();
        } catch (MQClientException e) {
            throw new RuntimeException(e);
        }

    }

    @Override
    public SampleResult runTest(JavaSamplerContext javaSamplerContext) {
        SampleResult sampleResult = new SampleResult();
        sampleResult.setSampleLabel("rocketmq-producer");
        // 请求开始
        sampleResult.sampleStart();
        // 普通消息发送
        Message message = new Message(
            javaSamplerContext.getParameter(TOPIC),
            javaSamplerContext.getParameter(MSG_TAG),
            javaSamplerContext.getParameter(MSG_BODY).getBytes()
        );
        try {
            // 发送消息，需要关注发送结果，并捕获失败等异常。
            SendResult sendResult = producer.send(message);
            // 设置发送请求的字节数
            sampleResult.setSentBytes(message.toString().getBytes(StandardCharsets.UTF_8).length);
            sampleResult.setDataType(SampleResult.TEXT);
            // 设置请求内容
            sampleResult.setSamplerData(message.toString());
            // 设置响应内容
            sampleResult.setResponseData(String.format("Msg Id:%s", sendResult.getMsgId()).getBytes());
            sampleResult.setSuccessful(true);
            sampleResult.setResponseCodeOK();
        } catch (MQBrokerException | InterruptedException | RemotingException | MQClientException e) {
            sampleResult.setSuccessful(false);
            sampleResult.setResponseCode(ERROR_CODE);
            sampleResult.setResponseData(String.format("Error Msg:%s", e).getBytes());
            return sampleResult;
        } finally {
            // 请求结束
            sampleResult.sampleEnd();
        }
        return sampleResult;
    }

    @Override
    public void teardownTest(JavaSamplerContext javaSamplerContext) {
        producer.shutdown();
    }

    @Override
    public Arguments getDefaultParameters() {
        Arguments arguments = new Arguments();
        arguments.addArgument(NAME_SRV_ADDRESS, "");
        arguments.addArgument(PRODUCER_GROUP, "");
        arguments.addArgument(TOPIC, "");
        arguments.addArgument(MSG_KEY, "");
        arguments.addArgument(MSG_TAG, "");
        arguments.addArgument(MSG_BODY, "");
        return arguments;
    }
}

Step 3: Package the project into a JAR file

Package the project through mvn clean package. In the target directory, you can see two JAR packages: jmeter-rocketmq4.jar and jmeter-rocketmq4-jar-with-dependencies.jar. Among them, jmeter-rocketmq4-jar-with-dependencies.jar includes all Required dependencies, use this JAR package in subsequent steps.

.
├── pom.xml
├── src
│   ├── main
│   │   ├── java
│   │   │   └── JavaSamplerForRocketMQ4.java
│   │   └── resources
│   └── test
│       └── java
└── target
    ├── jmeter-rocketmq4-jar-with-dependencies.jar
    ├── jmeter-rocketmq4.jar

Step 4: Use JMeter GUI for scripting and debugging

Copy the packaged JAR package and dependent JAR packages to the JMETER_HOME/lib/ext directory, and then execute the command JMETER_HOME/bin/jmeter to open the JMeter GUI.
After creating a new thread group, add the Java request sampler.

In the drop-down box, select the class added in step 2 (it may not be exactly the same as in the picture, select according to the actual fully qualified name of the class), and fill in the relevant parameters below.

Add "View Result Tree" and "Summary Report" listeners to the thread group, then start the test plan, and verify whether the results of the test are as expected in the result tree and summary report.
Save the test plan as a JMX file.

Step 5: Create a JMeter scenario in PTS for stress testing

Create a JMeter environment in the PTS console, and upload the JAR package packaged in step 3 to the JMeter environment (for more details, please refer to View, Modify, and Create JMeter Environment Management_Performance Testing-Alibaba Cloud Help Center [ 1] ):

a. Enter the PTS console and select "JMeter Environment";

b. Enter the customized environment name;

c. Click to upload the file and select the JAR package packaged in step three;

d. Click Save.

Select the "JMeter Stress Test" scenario in the PTS console creation scenario:

Edit "Scene Configuration":

a. Custom scene name;

b. Click Upload File and select the JMX file saved in step 4;

c. Select "Yes, use dependent environment" in the "Use dependent environment?" drop-down box;

d. Select the JMeter environment just created in the "Select dependent environment" drop-down box.

Pressure configuration:

Small suggestion: Since we want to find the maximum number of concurrent requests that RocektMQ can withstand through stress testing, it is recommended to choose the RPS mode, which can directly measure the pressure-bearing capacity of RocektMQ. At the same time, considering the limitations of public network bandwidth, Alibaba Cloud VPC intranet stress testing should be selected.

a. Select the pressure source as the Alibaba Cloud VPC intranet, and select the region where the ECS where RocketMQ is deployed for stress testing is located;

b. Set the VPC, security group and switch of ECS. Note that the VPC and security group must be the same as ECS, and the corresponding ports must be opened in the security group (set in the ECS console);

c. Set the pressure mode to RPS mode;

d. Set the starting RPS, maximum RPS and stress test duration. This article sets the starting RPS to 90000 and the maximum RPS to 110000 for 2 minutes.

e. The specified loop is generally set to No, which means that it will be executed once and it will end. The specified IP number will be automatically generated according to the set RPS.

For the rest of the settings, please refer to JMeter Stress Test_Performance Test-Alibaba Cloud Help Center [ 2] according to your needs .
Save the configuration and debug the scenario, confirm the connection with RocketMQ, and then start the stress test.

Step 6: View the stress test report

For general interpretation of JMeter's stress test report, please refer to How to View JMeter Stress Test Data, Sampling Logs and Pressure Applicator Performance_Performance Testing-Alibaba Cloud Help Center [ 3] . The next section will introduce how to use PTS's stress test report to find RocketMQ. pressure-bearing capacity.

Report interpretation

First, view the overview information and indicator trends of the entire stress test. As shown in the figure below, the first column of the report shows the request success rate, average RT, average TPS and other indicators of the entire stress testing process. Specific explanations of these indicators can be found in the official documents. At the same time, according to the success rate trend chart, starting from 18:54:05, the success rate gradually fluctuates and decreases. The TPS value at this time is 9.55W, which means that the average TPS in the first 5 seconds calculated at 18:54:05 is about 9.55 W.

Secondly, use the Prometheus monitoring data in the stress test report to further analyze the results. With the help of Alibaba Cloud ARMS's Prometheus and Grafana products, PTS's stress test report can provide timing diagrams including throughput, success rate, and response time. At the same time, users can use PromQL statements to edit the data panel and flexibly query the required data. ,In this article, we can put the success rate and ,throughput in a panel for further analysis.

a. First click "Success Rate (Time Series)", and then click "Edit" to enter the editing interface of the success rate dashboard and copy the success rate query PromQL:

sum(rate(pts_api_response_total{task_id="$task_id", code=~"200|302"}[5s]))/sum(rate(pts_api_response_total{task_id="$task_id"}[5s]))

b. Then enter the editing interface of the throughput dashboard, replace the PromQL of the number of virtual users with the PromQL of the success rate, and change the relevant configuration of Grafana (red box in the figure below), and you will get a panel showing the throughput and success rate.

The statistical accuracy of the data displayed in this panel is 1 second, which can obtain more accurate data. At 18:54:05 seconds, the success rate begins to decrease, and the TPS is 96561.9 at this time.

c. In order to better evaluate the performance of RocketMQ, we can also calculate the average TPS within the time range when the success rate remains 100%. First find the duration when the success rate is 100%, which is 47 seconds in the figure below, and then calculate The time range of the TPS indicator is changed to 47s, so that each point represents the average TPS of the previous 47s. Move the mouse to the last time when the success rate is 100%. The TPS value at the current time is the time range of 100% success rate. The average TPS within is 89357.5.

Finally, in order to compare the impact of different parameter settings on RocketMQ performance and verify the availability of PTS in RocketMQ stress testing, we conducted a simple comparative experiment and used the jstat command to observe the impact of different parameters on garbage collection.

The experimental results show that for RocketMQ deployed with the current ECS configuration, appropriately increasing the heap memory can effectively improve the performance of RocketMQ. When the heap memory is increased to 24g (the ECS memory usage reaches 85.39% in this case), the performance is not significantly improved; appropriately Increasing the value of sendMessageThreadPoolNums can improve the performance of RocketMQ. When sendMessageThreadPoolNums exceeds 16, the performance does not improve significantly, or even decreases slightly. Users can conduct more detailed comparative experiments based on actual conditions to fully evaluate the pressure-bearing capacity of the deployed RocketMQ.

Conclusion

This article introduces the detailed steps of using Alibaba Cloud PTS's JMeter scenario to stress test RocketMQ, explaining each link one by one. Finally, through the customized analysis of the stress test report, it demonstrates the powerful stress test result analysis capabilities of PTS. With the help of JMeter and PTS, users can perform flexible and multi-dimensional analysis on various types of middleware to help them build stable and robust systems.

Latest events & free trials

Related Links:

[1] View, modify and create JMeter environment management_Performance Test-Alibaba Cloud Help Center

https://help.aliyun.com/document_detail/170857.html?spm=a2c4g.103173.0.0.292c20f8wnWyCV

[2] JMeter Stress Test_Performance Test-Alibaba Cloud Help Center

https://help.aliyun.com/document_detail/97876.html?spm=a2c4g.91788.0.0.2fde6f338aHIDI

[3] How to view JMeter stress test data, sampling logs and pressure machine performance_Performance Test-Alibaba Cloud Help Center

https://help.aliyun.com/document_detail/127454.html?spm=a2c4g.94066.0.0.4a5164bepHmzWD