In fact, before the performance test, there are many things to prepare, one of which is about the smoke test. It is to ensure that the test environment built can complete the basic business functions. Before actually raising the business volume, it is best to do the following tasks:
- Ensure that the performance of the operating system layer is suitable for testing; ensure that the update of the kernel and surrounding software will not affect the performance test;
- If the hardware is replaced: CPU, network card; also need to ensure that the performance of the hardware meets expectations; the performance of the disk;
- If the cloud platform is replaced or upgraded, it is also necessary to test the performance of the cloud platform to see if it meets expectations;
- Test the network to see whether the network conditions between each unit meet expectations; TCP, UDP, fragmentation, etc. need to be done; after the
above operations are completed, iperf needs to save the results for future software upgrades. There is a contrast. Although these tests are relatively trivial, it may be possible to use automated scripts to achieve automatic running.
To give an example: I recently encountered a performance problem. After starting the business, I found that the sending queue on the socket was full, but I was not sure which one had the problem. Is it because the business data was actually overloaded, or was there something wrong with the network? Fortunately, you can see the TCP-related retransmission count. Therefore, it is suspected that there is a problem with the TCP service data caused by the network problem of the device. Of course, it is more difficult to debug. You need to capture the network packets at both ends, and then compare them to see which device on the network has a problem.
If you have the performance data of the basic performance test above, it may help us locate the problem.
实例
https://bugzilla.redhat.com/show_bug.cgi?id=2221806
all openmpi benchmarks fail on all RDMA device