Continuous testing is a testing practice that includes testing activities involving all quality characteristics. Therefore, continuous testing also includes non-functional testing. As we all know, software quality characteristics include functionality, reliability, usability, efficiency, etc. We have introduced some functional implementation methods in continuous testing, so let’s explain the non-functional testing in continuous testing.
Performance Testing
Performance testing is probably a term that all test engineers are familiar with. Many people can explain how to perform performance testing. As for what performance testing is, it is difficult to give a definition. Performance testing is an overall concept, which refers to the number of concurrent users that the test system can withstand under certain constraints. The constraints here refer to external constraints such as software, hardware, and network required for the operation of the system under test.
Performance testing also involves concepts such as load testing, stress testing, limit testing, and capacity testing. These concepts are intertwined and there is no generally accepted definition in the industry.
In fact, load testing is a test that simulates the load conditions borne by the actual software system. Stress testing is used to evaluate how a system will perform at or above expected loads. Limit testing is similar to stress testing, and capacity testing is similar to load testing. But these are only conceptual classifications, and it is difficult for us to clearly distinguish between load testing and stress testing during the testing process.
In actual work, performance testing, stress testing, and load testing often refer to one thing, that is, load testing. So at work, when the above concepts are mentioned, unless there are other preconditions, you can prepare according to the load test.
Continuous testing is more inclined to the smooth and continuous product delivery process, so in performance testing, it is more inclined to the application practice of DevOps pipeline bidirectionally driven performance testing technology, especially the implementation of performance testing as code technology.
Overview of performance testing tools
The performance test is done by simulating high concurrent access. This simulation is not really having multiple users accessing the system together, but is done through tools. There are three modes for performance testing tools to simulate high concurrency, namely multi-process, multi-thread and coroutine. The relationship between processes, threads, and coroutines is shown in Figure 3-1.
Processes are defined for better utilization of CPU resources, and are used to allocate system resources and identify tasks. A process is the smallest unit of system resource allocation and mainly occupies resources such as address space, global variables, file descriptions, and hardware. The operating system completes multiple tasks in a multi-process manner, so that system resources can be fully utilized.
Among the current mainstream performance testing tools, LoadRunner supports multi-process, but the authorization fee of this tool is very expensive. In addition, the multi-process mode occupies more system resources, and the scheduling overhead between processes is relatively large, so it is not recommended to use the multi-process mode in performance testing.
Threads are different. Threads reduce the consumption of context switching and improve the concurrency of the system. Threads are like multiple production lines in an automobile production workshop, and each production line processes the same transaction work concurrently. If multiple threads are started in one process to process the same logic, the effect of concurrent processing is achieved. Both LoadRunner and JMeter support the multi-threaded concurrent access model, among which JMeter is an open source performance testing tool.
In the process of performance testing, the multithreading model is often used to simulate concurrent access, so that concurrent access can be completed and system resources can be fully utilized. Coroutine is a relatively new concept. It completes scheduling through user control instead of CPU, thus avoiding the resource consumption caused by context switching at the kernel level and breaking through the performance of threads on I/O. bottleneck. The coroutine does not require a multi-threaded lock mechanism, and the scheduling is implemented based on threads.
Locust is a performance testing tool based on the coroutine access model. Many readers who are engaged in performance testing should have heard of LoadRunner, but there are probably fewer people who know Locust. These two tools have their own advantages, and which one to choose in an actual project depends on the situation.
However, in the current era of containerization technology, LoadRunner has become less useful. Both LoadRunner and Locust provide functions such as UI script editing and recording, scene setting, etc., which leads to only concurrent simulation when using them on the container, and script writing needs to be completed on the client PC. For the performance testing tool on the container, we hope it has the following functions.
Support script editing and debugging on PC.
Supports script editing and debugging on the container.
Support for performance testing on the server side.
It has a server-side performance scenario setting function.
It has the scene setting function without UI on the server side.
It can be integrated with CI (Continuous Integration) system.
In summary, Locust is highly recommended. Even if not used on containers, Locust is a good choice. Locust is an open-source performance testing tool that supports the use of Python code to define user behaviors, and uses pure Python to describe test scripts.
Use Locust to simulate millions of concurrent users accessing the system. In addition to HTTP/HTTPS, Locust can also be used to test systems using other protocols, just use Python to call the corresponding library and describe the request.
Locust is also a distributed user load testing tool that can be used to load test websites or other systems. Locust can be used to test the number of concurrently processed users of the system, which is completely time-based, so a single machine may support thousands of concurrent users.
It is difficult for LoadRunner and other testing tools that use processes and threads to simulate high concurrency pressure on a single machine. Compared with many other event-driven applications, Locust does not use callbacks, but uses a lightweight processing method - coroutines.
A coroutine is a lightweight thread in user mode that is controlled by the user. The coroutine has its own register context and stack. When the switch is in progress, the coroutine will save the register context and stack to other locations. When switching back, the coroutine can not only restore the saved content, thereby reducing the consumption of kernel switching, but also access global variables without locking. The coroutine avoids the scheduling of system-level resources, thus greatly improving the concurrency capability of a single machine.
Among the eight quality characteristics of GB/T 25000.10, the independent quality characteristic of performance efficiency includes quality sub-characteristics such as time characteristics, resource utilization, capacity, and performance efficiency compliance. The process of using technical means to test the performance efficiency of information systems is called performance testing. Performance testing allows us to assess how well an information system meets performance efficiency requirements. Information systems usually contain information on the following aspects.
Number of concurrent users: This is for the server, referring to the number of online users interacting with the server at the same time. During stress testing, the number of concurrent users refers to the number of users who execute one or a series of operations at the same time, or the number of users who execute a certain script at the same time. The concurrency situation in different scenarios is different. In the actual test work, the number of concurrent users should be set according to specific requirements.
The maximum number of concurrent users: used to describe the maximum service capability of the information system.
Throughput: The number of requests that the system can handle per unit time. For interactive systems, the unit of throughput is usually bytes/second, pages/second, or requests/second; for non-interactive systems, the unit of throughput is usually transactions/second.
Response time: divided into user response time and system response time. User response time refers to the response time of the system that the user can perceive for its operation. Human eyes can only detect visual changes of more than 0.1s due to the phenomenon of "persistence of vision", so the user's response time only needs to be no more than 0.1s. System response time is the time it takes for a computer to respond to a user's input or request. Stress testing generally considers problems from the perspective of users, so it measures user response time.
Resource utilization: A series of data indicators describing the performance status of the information system, including CPU utilization, memory utilization, disk I/O rate, network throughput, etc. of the server under test.
Waiting time: the time interval between two consecutive requests issued by information system users when conducting business operations.
Performance testing is used to evaluate the serviceability of the system. Performance testing is mainly divided into the following three types.
Load stress test: By continuously adding load to the system, observe the performance changes of the system and determine that the system meets a series of performance indicators (including response time, CPU utilization, memory usage, network throughput, disk I/O rate, etc., among which The key performance indicators are the maximum load that can be sustained under the premise of response time, CPU utilization and memory utilization.
Failure recovery test: For the system that provides system redundancy backup or load balancing mechanism, after a partial failure of the simulated system, in the case that a large number of users continue to access the system, the recovery of system service capability is tested, mainly used to evaluate the system robustness and recoverability.
Fatigue test: The test of running the system for a long time under the condition of ensuring the total business volume is mainly used to evaluate the ability of the system to operate stably for a long time without failure (the test period is usually 7×24 hours, 3×24 hours or 1× 24 hours).
Finally, I would like to thank everyone who has read my article carefully. Reciprocity is always necessary. Although it is not a very valuable thing, you can take it away if you need it:
Software testing interview applet
The software test question bank maxed out by millions of people! ! ! Who is who knows! ! ! The most comprehensive quiz mini program on the whole network, you can use your mobile phone to do the quizzes, on the subway or on the bus, roll it up!
The following interview question sections are covered:
1. Basic theory of software testing, 2. web, app, interface function testing, 3. network, 4. database, 5. linux
6. web, app, interface automation, 7. performance testing, 8. programming basics, 9. hr interview questions, 10. open test questions, 11. security testing, 12. computer basics
These materials should be the most comprehensive and complete preparation warehouse for [software testing] friends. This warehouse has also accompanied tens of thousands of test engineers through the most difficult journey. I hope it can help you too!