What the test architect is engaged in is not a pure testing technology job, but a comprehensive art that requires a combination of products, communication and coordination, written expression, etc., as shown in Figure 1.
Figure 1 Competencies required by software testing architects
In terms of testing technology, software testing architects need to have the following testing technology capabilities:
Software Product Quality Model
Test type
Test Methods
exploratory testing
automated test
1 Six attributes of software product quality
Figure 2 Six attributes of software product quality
1.1 Functionality
Table 1 Functional sub-attributes
Suitability: For Windows calculators, all "calculation"-related functions provided by the software product to users are suitability.
Accuracy: For Windows calculators, the correctness of the calculation results of the calculator itself is an expression of the accuracy of the calculator software.
Interoperability: For Windows calculators, whether different functions and features in the calculator can correctly cooperate with each other is an expression of the calculator's interoperability; in addition, support for different operating systems, such as Support for different versions of Windows 7 (including different patch versions) and support for different working modes (such as safe mode, safe mode with network connection) are also manifestations of interoperability.
Security: For Windows calculators, the calculator should not contain exploitable security holes and content related to "user permissions", such as "both administrators and guests should have the same usage permissions", etc.
Functional compliance: For Windows calculators, functional compliance can be understood as that, as a calculator, calculation rules (such as square operations, statistical operations, etc.) must be consistent with relevant rules in mathematics.
1.2 Reliability
The following three progressive sentences can help us understand user reliability requirements:
Level 1: It is best not to malfunction the equipment;
Layer 2: If the equipment fails, the main functions and services should not be affected;
Level 3: If main functions and services are affected, the system can be located and restored as soon as possible.
Table 2 Reliability sub-attributes
Maturity: For Windows calculators, maturity can be understood as the probability of product function failure. For example, if the calculator continues to run for a period of time, calculation errors will occur. Generally speaking, this kind of error can be recovered by restarting the software, restarting the device, etc.
Fault tolerance: For Windows calculators, fault tolerance can be understood as the product's ability to handle user "error input"
Recoverability: For Windows calculators, recoverability can be understood as the ability of the calculator to recover once an abnormality that cannot be anticipated by the product itself occurs (such as no response, restart).
Reliability compliance: For Windows calculators, there are no strict and clear standards for reliability compliance, but there are some potential conventions. For example, the calculator needs to be able to identify abnormal input for all mathematical operations, and Give a hint as to the cause of the error. Communication products have relatively strict standards in terms of reliability and compliance, such as the system failure rate cannot be higher than the certain value and the fault recovery time cannot be longer than the certain value, etc.
1.3 Portability
Adaptability: For Windows calculators, adaptability can be understood as whether the layout, size, clarity, arrangement of keys, etc. of the calculator can be displayed normally on displays of different sizes.
Installability: For Windows calculators, installability can be understood as whether the calculator can be successfully installed on different Windows operating systems and run normally.
Coexistence: For Windows calculators, coexistence can be understood as the calculator and other software can coexist in Windows at the same time, and there will be no competition for resources (such as CPU, memory, etc.).
Ease of replacement: For Windows calculators, ease of replacement can be understood as assuming that a new calculator is developed and the new calculator can successfully replace the old calculator. (Note that this does not refer to "product upgrade", but there may be situations where "new" and "old" calculators coexist at the same time.)
Portability compliance: For Windows calculators, portability compliance can be understood as some conventions on portability of Windows products. For example, a calculator is not developed for a specific operating system and needs to support all Windows operating systems.
2 test types
Test types refer to the different perspectives that need to be considered for testing. We can use the software product quality model (hereinafter referred to as "quality model") to quickly define and understand test types.
Figure 3 Relationship between quality attributes and test types
Table 5 Common test types and their relationship with quality attributes
3 Test methods
3.1 Product testing wheel diagram
The test type, that is, the angle from which the test is to test the product, determines the idea of the test. What we want to discuss next is how to do it, that is, the specific testing method.
Figure 4 Product test wheel diagram
The relationship between the six categories of quality attributes and test types depicted in Figure 4 does not go into each quality sub-attribute and the test type corresponding to each sub-attribute (you may wish to draw the wheel of "quality sub-attribute" by yourself picture).
Two key issues in product testing can be analyzed from the "wheel diagram":
The first is the issue of how to ensure the "comprehensiveness" of test verification. Obviously, as long as the testing method we use can cover the six quality attributes, there will be no general directional omissions in our testing.
The second is the issue of how to determine the "depth" of testing. Generally speaking, the more testing methods a testing team uses, the more deeply the product is tested.
These will affect the effect of the test and the evaluation of product quality by the test. In addition, the "wheel diagram" can also help us evaluate the capabilities of the testing team. The more testing methods a software tester can master, the stronger his testing capabilities will be.
3.2 Functional testing method
Functional testing methods include:
Single run normal value input method.
Single run boundary value input method.
Multiple run sequence execution method.
Multi-run interaction method.
definition:
Run: In software testing, the "operation" or "behavior" of the user simulated by the tester.
Single run: In software testing, the tester simulates "one operation" or "one behavior" of the user.
Multi-run: In software testing, the tester simulates "multiple operations" or "multiple behaviors" of the user.
That is, "running" refers to an operation or behavior that makes sense from the user's perspective. From a functional level, a "run" determines a possible situation of "input" and "output", as shown in Figure 5.
Figure 5 Schematic diagram of operation at the functional level
Sometimes, we divide functions from a design perspective and fail to provide users with a complete and meaningful behavior, such as "the user established a new connection with the mail server" and "the mail server deleted the connection with the user." , this kind of fine-grained function does not count as a "run" even if the input and output are determined. At this time, multiple functions can be combined until this function can provide complete and meaningful behavior for users, as shown in Figure 6.
Figure 6 Relationship between function and operation
3.2.1 Single-run normal value input method
It is a test method in which the "A1" and "A2" entered during the test are the "normal values" allowed by the system.
3.2.2 Single-run boundary value input method
This is the test method in which the "A1" and "A2" entered during the test are the "boundary values" of the system.
3.2.3 Multi-run sequence execution method
Consider multiple "single run" operations together, and the result is a "multiple run" operation. When using the multi-run sequential execution method for testing, analyzing the sequence between each run is the key to using this method.
The multi-run sequence execution method is very effective in places related to user operating habits.
In addition, the multi-run sequential execution method is also more suitable for use in functional configuration testing.
Table 6 Multiple running sequence scenarios
Multiple running scenarios with serial numbers are associated with multiple running sequences.
1 The user first sends an email and then receives an email No No
2 After the user receives an email, and then sends the received email, is it
3 After the user receives an email, send another arbitrary email No No
3.2.4 Multi-run interaction method
The multi-run interaction method refers to a method of combining multiple inter-related runs for testing during functional testing, as shown in Figure 7
Figure 7 Multi-run interaction method
Different from the multi-run sequential execution method that emphasizes the sequence between multiple runs, the multi-run interaction method emphasizes the relationship between multiple runs. This relationship can be an external relationship, such as the smooth progress of a certain business. Multiple runs collaborate with each other; it can also be an intrinsic relationship, such as these runs using the same resources (such as memory or other hardware resources).
What needs to be pointed out in particular is that these are all "for one user" operation scenarios, rather than "two different users send emails at the same time" or "one user sends emails and one user receives emails" scenarios. In fact, the former belongs to the category of functional testing, while the latter belongs to the category of reliability testing.
3.3 Reliability test method
3.3.1 Outlier input method
The outlier input method is a reliability testing method that uses values that the system does not allow users to input (i.e. outliers) as test input.
3.3.2 Fault implantation method
The fault implantation method is a reliability testing method that places the system in a problematic environment for testing. The main quality attributes that can be tested are fault tolerance and maturity.
Different from the outlier input method, the outlier input method directly inputs a value that the system considers to be wrong and unsupported; while the fault implantation method places the system in a problematic environment, but the input is still a normal value.
What glitches, errors, or problems are there in the user's business environment? List these scenarios, put the system into these scenarios, run normal business, and analyze whether the system's response at this time is reasonable.
If the system is deployed in the user's hardware environment, consider the hardware resources required by the system, such as CPU, memory, storage space, etc., and whether the system's response is reasonable when insufficient. Let's take the example of "User sends email". Network failure is a common failure for users, such as disconnection, intermittent network, and packet loss.
If the system is installed on the user's system, consider whether the system's response is reasonable in the event of software conflicts, incorrect drivers, etc.
If the system is an independent device, consider whether the system's response is reasonable when problems occur with its key components (such as chassis, single boards, plug-in cards, hard drives, chips, etc.).
3.3.3 Stability test method
The stability testing method is a reliability testing method that runs a certain business at a large capacity for a long time over a period of time. It can very effectively test the "maturity" of the system and is a very important reliability testing method. .
It should be pointed out in particular that there is a certain relationship between the stability testing method, the stress testing method and the performance testing method, and this relationship is the product specification.
Product Specifications: The maximum capacity or capability that a product promises to handle. For example, the system supports up to 100 users to log in concurrently and the system supports the establishment of up to 100 security policies, which are product specifications.
The purpose of performance testing is to test whether the actual specifications of the product are consistent with the required specifications promised in the manual. Obviously, the final performance value we measured is the maximum capacity or capability that the system can really handle.
Stability testing is performed below performance values. In fact, when users use the system, they will not always let the system run at the extreme state. During testing, we can control the load in the test to make it as close as possible to the user's actual usage, so that the test can More accurate and more valuable.
Stress testing is performed at higher than performance values. Although the load during the test exceeded the maximum capacity of the system, it does not mean that the system's functions will fail in this case. We need to analyze whether the system's performance is reasonable based on the actual situation. For example, the system supports up to 100 users to log in concurrently, but at this time 110 users initiate login operations at the same time, then the system should ensure that 100 of these 110 users can log in normally, and it is reasonable for 10 users to be unable to log in. , instead of all users being unable to log in.
Now let us return to the topic of this section - stability testing (performance testing methods and stress testing methods will be introduced to you in the following articles). From a methodological perspective, the stability testing method is the most interesting of all testing methods, and can be summarized as a "four-character formula" - multiple, union, complex, and difference.
The essence of "multiple words" is to test the stability of the system by increasing the number of user operations on functions during the test.
The essence of "Ming Zi Jue" is to allow multiple users to operate this function at the same time during the test to test whether the system is still stable. Sometimes we also call this test concurrency testing.
The essence of "Fu Zi Jue" is to allow one or more users to repeatedly perform operations such as creating, refreshing, deleting, synchronizing, and backing up during the test to test whether the system is stable.
The essence of "different words" is to allow one or more users to repeatedly perform abnormal operations during the test to verify whether the system can continue to respond reasonably. Compared with the abnormal input method and the fault implantation method, "Yi Zi Jue" emphasizes "continuation" and "accumulation". In fact, it is often more effective to use "different words" to test. This is because when developing code, it is easy to consider the application and recovery of resources under correct circumstances and ignore the recovery of resources under abnormal circumstances.
3.3.4 Stress testing method
Stress testing is a reliability testing method that continuously uses loads that exceed system specifications for a period of time.
Although the product has declared specifications and only promises to provide stable and reliable functions within the specification range, and does not promise to provide correct functions when exceeding system specifications, stress testing is still meaningful. An important reason is the sudden phenomenon of business - the user's business load is not average. It may exceed the load in a very short period of time, but on average, it does not exceed the specification, as shown in Figure 9 .
Figure 8 Business emergencies
We hope that the system will not be as fragile as a house of cards in the event of an emergency, but will have practical countermeasures, such as not handling loads that exceed system specifications, recording logs for users to analyze the causes of emergencies, etc. Fatal problems such as crashes and repeated restarts will not occur due to emergencies. This is the real purpose of our stress test. In order to achieve this test goal, we need to use a burst load model when performing stress testing, as shown in Figure 9
Figure 9 Burst load model
It is not recommended to use testing methods that continuously exceed system specification loads to dig out product problems. But for testing, it is not completely meaningless to use load model testing that continuously exceeds specifications. It is an integral part of our other reliability testing method-the recovery testing method.
3.3.5 Recovery test method
The recovery test method refers to a test method that uses a load that continues to exceed the specification for testing, and then reduces the load to within the specification, as shown in Figure 10
Figure 10 Recovery test method
When continuously performing load tests that exceed specifications, the business within the allowed specifications is not 100% correct. When the load drops to within the specification value, the business must be able to recover to 100% accuracy.
In order to deepen everyone's understanding of the stress testing method and the recovery testing method, we might as well compare the two models' expectations for "business" results under different loads, as shown in Figure 11.
Figure 11 Expectations for business results from burst load mode and continuous load mode
It can be seen that the biggest difference between the two test methods lies in the "black" part in the picture. When using the burst load mode for stress testing, the black part in the figure does not allow business failure. When using the continuous load mode for recovery testing, the black part allows business failure.
3.4 Performance testing methods
The purpose of performance testing is to test whether the actual specifications of the product are consistent with the required specifications promised in the manual. The performance value we actually measure is the maximum capacity or capability that the system can really handle.
Generally speaking, the product's required specifications will give performance expectations, and the tester only needs to confirm whether the product can meet the specifications. If the product performance specifications in the requirements specifications are very simple and rough, we can conduct performance testing according to Figure 12
Figure 12 Test process
The first step is to test the best performance value of the system
When performing performance testing, we can first try to test the best performance value of the system. We can use the performance values required in the performance specifications as test items to test the limits of these indicators in the system.
The performance specifications of different products may vary widely, but in general, they can be divided into the following two categories.
1) The maximum ability of the system to correctly handle new business
The system's maximum ability to correctly handle new business, we also call it "new". For example, how many new users can the system allow to log in per second, how many new connections can the system actively initiate per second, etc.
Performance testing is conducted on the system's new capabilities. The test is the speed at which the system allocates resources to completes the processing process for a new business. The total amount of business processing processes and resources will affect the system's new ability. It should be noted that the system cannot be "built" without "tearing down": completed or abnormal services need to be dismantled in time, and the occupied resources must be recycled and used for new services.
2) The maximum business capacity that the system can correctly handle at the same time
The maximum business capability that the system can correctly handle at the same time is also called "concurrency". For example, the maximum number of simultaneous online users that the system can support, the maximum number of connections that the system can initiate simultaneously, etc.
It should be pointed out in particular that there is a relationship between "new" and "concurrency", as shown in Figure 13.
Figure 13 New creation and concurrent testing situation
Note: The two indicators of new creation and concurrency will affect each other. For example, the maximum concurrency capacity is 4000. During the test, it only "builds" and does not "tear down". When 150 new items are created per second, by 24 seconds, the number of parallels is greater than the maximum concurrency capacity. It is 4000, which leads to a decrease in the number of new creations.
Therefore, when we test the best performance value of the system, we need to pay attention to the internal relationship between the test indicators. When testing one indicator, other indicators cannot affect this indicator.
The second step is to analyze various factors that will affect the performance value and test their impact on performance.
Both "Configuration" and "Business" will have an impact on performance indicators. Just imagine, the performance of configuring 1 user policy and configuring 1000 user policies should be different; the system receives 1-byte mail and receives 10M mail The performance values tested are also different. In this step, we need to analyze which factors in the system have an impact on performance (performance degradation), and then conduct tests to analyze whether the performance degradation is in line with expectations and whether the worst-case scenario is still acceptable. Reasonable.
Table 7 The maximum number of emails that can be processed correctly at the sample point
Table 8 The maximum number of emails that can be processed correctly under different filtering strategies
By testing these performance values, we can easily get:
Which factors have a greater impact on system performance and which factors have less impact on system performance.
Trends in the impact of various factors on performance. By selecting test samples and using "curve fitting" technology in mathematics, we can obtain the influence curve of factors. We can use the curve to analyze whether this downward trend is reasonable.
The worst value of performance under each factor. Analyze whether this worst value is reasonable and whether it will become a performance "bottleneck" of the system.
Many times, there is not a single factor that affects a performance indicator, but there may be multiple factors. In this step, only the impact of a single factor on performance can be tested. The impact of multiple factors on performance can be placed in typical scenarios, and typical configurations and services can be selected for performance testing.
The third step is to test performance based on scenarios.
Finally, we use "scenario" as a unit to test the performance values of typical configurations and typical services in this scenario.
Take "User sends email" as an example. Assume that in this scenario, the typical configuration is "200 filtering policies", and the email size is 1KB, 10KB, 2MB mixed in 1:2:1, the email system can receive every second and correctly handle the maximum number of messages.
3.5 Ease of use testing method
3.5.1 Conformance testing method
The conformance testing method focuses on the user interface of the product during testing:
Whether the style, layout, and elements are consistent and unified.
Whether the rationality of layout, rationality of operations, prompts, etc. comply with UI design specifications.
The conformance testing method can test the product's capabilities in terms of ease of understanding and ease of use, but it does not care whether the product function is correct, so it can directly test the product's UI design prototype.
3.5.2 Usability testing method
The test object of the usability testing method is also the user interface, but in usability testing, we focus on whether the functions provided by the product are easy for users to learn, understand and use. Therefore, usability testing needs to be combined with functional testing, using scenarios as Test granularity, testing from the user’s perspective.
Table 9 Focus and standards for usability testing
4 Test design techniques
Table 9 Test points for “User sends email”
4.1 Test points are not equal to test cases
If we take test points for testing, we will find many problems that make us unhappy and confused:
Problem 1: These test points have duplicate content and are redundant.
For example, test point 1 and test point 4 will both test “sending emails correctly”.
Problem 2: The test input of some test points is unclear, and I don’t know what to test during the test.
For example, when testing test point 1, we don't know what "normal input data" we want to test. Similar problems exist at test point 5.
Problem 3: Always operating in similar environments and performing similar operations.
There is a certain execution sequence between some test points, and these test points need to be tested together. For example, executing test point 6 first and then executing test point 11 can maximize the use of the previous test environment and test results.
Question 4: The description of the test points is too rough. I don’t know if the test is correct.
Some test points are written very boldly, and we don't know what the test goals are or where to focus. For example, test point 4, we don't know if we put the two operations "user sends email" and "user receives email" together, where are their "interaction points"?
4.2 Four-step test design method
Processing test points into test cases is called test design, and the method used in this process is called test design method. Path analysis, decision tables, orthogonal analysis, equivalence classes, boundary values, etc. are all common test design methods.
In test analysis, we can get test points by thinking about the object under test according to the test method. Therefore, test analysis is a "discovery" process, as shown in Figure 14, but the test design is different.
Figure 14 Test analysis is a “discovery” process
You can do such an experiment and let two testers analyze the same test object based on the "wheel diagram". The test points they get will not be very different, but the test cases generated in the end will be very different. This is because, from test points to test design, we will process test points, combine and split them, select test data, etc. This is a "creative" process.
Figure 15 The four key steps of the four-step test design method
Step 1: Modeling
In fact, in this step, we do not ask everyone to create some original test models for each test point, but according to the characteristics of the test point, select a model suitable for subsequent test design for the test point. Perhaps we call this step "model selection" more appropriately.
Since "model selection" requires reference to the characteristics of the test points, it is essential to study the test points, analyze the characteristics and classify them. We currently divide them into four categories:
Type 1: "Process";
Type 2: "parameter";
Type 3: "data";
Type 4: "Combined".
For each type of test point, we have given some of the most suitable "modeling" methods:
For the "process" class, you can build a test model by drawing a "flow chart".
For the "Parameter" class, the test model can be established through the "Input and Output Table".
For the "data" class, the test model can be established through the "equivalence class analysis table".
For the "combination" class, the test model can be established through the "factor table".
Step 2: Design basic test cases.
In this step, we will design some basic test cases for the already established test model to cover this test model.
The biggest difference between test cases and basic test cases is that the test case determines the test conditions (a description similar to "under the circumstances of XX, perform XX test") and test data (that is, the input "parameter value" or "numeric value" ), while the basic test case only determines the test conditions.
Step 3: Supplement test data.
In this step, we determine the test input for the basic test case and supplement the test data. At this time, the basic test case is upgraded to a real test case.
Step 4: Expand.
Models are not silver bullets and cannot solve all problems with testing a design. We also need to add some test cases based on experience, especially the understanding of where defects in the system are prone to occur, to increase the effectiveness of the system.
Classify test points
Before using the four-step testing method, we first need to classify the test points. The basis for classification is to see whether the test points have characteristics of the "process" type, characteristics of the "parameter" type, characteristics of the "data" type, and characteristics of the "combination" type.
