Author | Xiong Yiheng Ph.D., School of Software Engineering, East China Normal University
Su Ting Professor, School of Software Engineering, East China Normal University
Section | Jianyuan Forum · Model Observation
Community| Add WeChat ID "TICPShanghai" to join the "Shanghai Kongan 51fusa Security Community"
In the software development life cycle, testing is a crucial part. In order to ensure the quality of software products, development teams need to conduct comprehensive testing to discover and fix potential defects and problems in the software. Although traditional manual testing is effective, it is time-consuming, labor-intensive, and costly. In order to improve testing efficiency and accuracy, automated testing has become an important research area.
Figure 1 Automated testing technology
Figure 1 introduces several existing automated testing methods [1]. The X-axis represents Feature compliance, that is, whether the test covers the functions that the tester wants to test, while the Y-axis represents Input scope covered, which represents the degree of coverage of the test input. It can be seen that there is a piece missing in the upper right corner of the figure, that is, the testing technology that meets both high feature compliance and full input scope covered. This technology is the property-based testing technology to be introduced next.
01
What is automated testing technology based on the nature of software
Property-based testing is a popular automated testing technology. Its principle is that testers write real logical statements (i.e. properties) applicable to the software under test, and then use automated testing tools to generate a large number of Test input to fully test the software under test and verify whether the properties written by the tester are met [2]. If the property is violated, it indicates that the software may have a bug. For example, the figure below is a program to be tested, `my_sort()`, whose function is to sort the input sequence and return the sorted sequence.
Figure 2 Program to be tested
To test `my_sort()`, usually testers will write unit test cases. Writing traditional unit tests requires manually specifying test inputs and expected outputs. As shown in the figure below, there are 6 test cases to test the program under test. However, the disadvantage of this method is that it is very labor-intensive, because for each test input, we need to manually specify the corresponding test output. Furthermore, testing has limited input and relies on the knowledge of the tester, making it difficult to adequately test the program under test.
Figure 3 Unit test case
However, if we use property-based testing, we only need to write a test method to cover all test inputs. At the beginning, we gave the properties (properties) that the program under test should have based on our understanding of the program under test. For example, one of the properties that my_sort() should satisfy is that each element in the returned list should not be greater than the element after them. Based on this property, we can write a test case. Note: We use hypothesis, a property-based testing framework for python language, as an example to write.
Figure 4 Property-based testing test case
As shown in the picture above, the first line of code `@given(st.lists(st.integers())` is a decorator, indicating that once the test starts, Hypothesis will generate a large number of random lists as the `test_prop_ordered()` function Input. Each generated list will be passed to the `test_prop_ordered()` function as parameter xs, and the function body will be executed, and then the assert statement will be verified to be true. In this way, we can automatically generate enough lists to verify the program under test Whether it meets the properties we set without having to manually specify specific test input and output data, this greatly improves testing efficiency.
02
How to conduct automated testing based on the nature of software
The previous section introduced the concept of Property-based testing, and this section introduces how to conduct automated testing based on the nature of the software. We use a common template to explain how to perform property-based testing:
Figure 5 property-based testing template
Step 1: Determine the nature of the program you want to test. As shown in the third row of Figure 5, testers determine the nature of the program they want to test through their understanding of the program under test. For example, in Figure 4, the test case is designed to verify the following program property: Each element in the sorted list should not be larger than the element after it. If this property is violated during testing, it indicates that a bug may have occurred in the program under test.
Step 2: Determine the test input type and range. After completing the first step, the tester should determine the type and range of test input that will be passed to the program under test, as shown in the first and second rows of Figure 5. For example, in Figure 4, since the function of the program under test is to sort the elements in the list by size, the test input should be a list. At the same time, since sorting involves size comparison of elements, the elements in the list should be numbers. Here, we have chosen integer as the element type.
Step 3: Write and run test cases. Currently, developers have developed a variety of property-based testing frameworks for different software or programming languages (Java: QuickTheories, Python: Hypothesis, C++: RapidCheck, Scala: ScalaCheck, JavaScipt : fast-check, Ruby: Rantly, Swift: Swiftcheck, etc.) [3-9]. Once a suitable testing framework is selected, we can use the testing framework to write corresponding test cases and run them.
03
HowHow to choose the appropriate software properties
The first two sections introduced the concept of property-based testing and how to perform property-based testing. We can see that property-based testing is a very efficient testing technology. However, when testers actually start writing test cases, they often encounter a question: What properties should be chosen for testing? Therefore, in this section, we will introduce the more general properties [10], hoping to inspire testers on how to choose appropriate software properties.
Property 1: Symmetry (Symmetry). As shown below. If you can convert some value to another value and back again, it should remain consistent with the original value. For example, Serialization is a typical example. Convert text to an object and then convert the object back, and the text should remain unchanged.
Figure 6 Symmetry
Property 2: Commutativity (Commutativity). This property refers to changing the order in which operations are performed, but the final result does not change. For example, as shown in the figure below, add an element to the head of a list, and then add an element to the tail. If the order of these two operations is exchanged, the final list should be the same.
Figure 7 Commutativity
Property 3: Invariants (Invariants). This property means that certain properties of the test object will not be changed after certain operations are performed. For example, as shown in the figure below, after sorting an array, the length of the array should remain the same.
Figure 8 Invariance
Property 4: Idempotence). This property generally means that the effect of performing an operation multiple times should be the same as performing an operation once. For example, as shown in the figure below, for a list, sorting it once and sorting it multiple times, the end result should be the same.
Figure 9 Idempotence
Property 5: Induction (induction). This property usually means that if a large object can be divided into smaller objects, and certain properties hold for these small objects, then it can be proven that these properties also hold for the larger object. For example, as shown below. If the black set contains an element, then the red, blue, green, and black sets should also contain an element.
Figure 10 Derivation
04
Summarize
Property-based testing has become a very popular testing technology in recent years, and developers have developed a variety of property-based testing frameworks. This technology has many advantages. For example, in theory it can cover all possible test inputs, more fully test the functions that users care about, and reduce testing costs. However, due to its characteristics, there are also some challenges. For example, because it only cares about the properties it wants to test, it results in a certain property being fully tested, while other properties are not tested at all. In addition, testers need to understand the software under test well enough to abstract the properties that should be maintained, which places high demands on testers. In addition, developing an efficient property-based testing framework is also a big challenge for developers. Developers need to think about how to make it easier for testers to write properties, how to develop better data generators, etc. Most importantly, property-based automated testing techniques are not meant to replace other testing techniques (e.g., unit testing) but can coexist with other testing techniques. Testers should choose appropriate testing technology based on testing needs so that it can better serve software testing.
references:
[1]https://medium.com/criteo-engineering/introduction-to-property-based-testing-f5236229d237
[2] Claessen, K., Hughes, J.: QuickCheck: a lightweight tool for random testing of Haskell programs. In: Proceedings of the Fifth ACM SIGPLAN International Conference on Functional Programming (ICFP’00), ACM, pp. 268–279 (2000). https://doi.org/10.1145/ 351240.351266
[3] QuickTheories. https://github.com/quicktheories/QuickTheories
[4] Hypothesis. https://hypothesis.works/
[5] RapidCheck. https://github.com/emil-e/rapidcheck
[6] ScalaCheck. https://github.com/typelevel/scalacheck
[7] fast-check. https://github.com/dubzzz/fast-check
[8] Rantly. https://github.com/rantly-rb/rantly
[9] SwiftCheck. https://github.com/typelift/SwiftCheck
[10] https://fsharpforfunandprofit.com/posts/property-based-testing-2/