Article directory
foreword
Software testing has a history of 50 years, and automated testing has a history of more than 20 years. Still, end-to-end, thorough automated testing remains a very elusive dream.
Existing problems: It is easy to automate test execution (understand: the process of testing work can be automated), difficult to automate test generation (understand: it is difficult to automatically generate test cases), and it is difficult to automate test judgment (oracle). Difficult (Understanding: Automatically realize the judgment, whether there is a problem, it is difficult to realize.). It can be said that we are still basically stuck in the automation stage of test execution. Without the automation of test generation and test verdict, there is no end-to-end, thorough automated testing. To achieve end-to-end and thorough automated testing, we need to break through the two bottlenecks of test generation and test judgment automation.
1. Software testing
Software testing: It is the process of constructing input (input) and acting on the system under test (Software Under Test, SUT), then observing its actual output (actual output), and comparing it with the expected output (expected output).
2. Random testing
Random test: It is to generate test .
To give a few examples:
- When the object under test is a function, the test input is the input parameter of the function, and the random test is to randomly generate the input parameter ;
- When the test object is an API interface, the test input is a request message, and the random test is to randomly generate a request message ;
- When the test object is the user interface, the test input is the operation sequence, and the random test is to randomly generate the operation sequence .
Pros and cons of random testing:
advantage:
- Easy to implement and use. Random testing does not require knowledge of program details (black box testing), and inputs are also generated randomly.
- There is no bias against the program. Since the input of the random test is randomly generated, there is no influence of human factors, and potential vulnerabilities will not be ignored because of trust in a certain part of the program.
- Can quickly find vulnerabilities. The testing speed of random testing is fast, and a large number of candidate vulnerabilities can be found in a short period of time through rapid and large-scale testing (note that the confirmation of vulnerabilities still requires manual participation).
shortcoming:
- The precision of finding bugs is not high. Due to the completely random nature of random testing, the bugs found are likely to be innocuous bugs.
- Too random leads to poor code coverage of the program. Most people think that the test of the program is too dependent on randomness, it is better to test the program more accurately through manual white box testing.
3. Fuzz testing
Background generated by fuzz testing:
On a stormy night in 1988, Professor Barton Miller of the University of Wisconsin was in his apartment connected to his school computer through a telephone line. The storm caused such scrambles on the phone lines that the connected Unix terminals were constantly receiving bad command input, eventually crashing the system. The frequent crashes surprised the professor who taught the operating system course, so an idea came to his mind to test the robustness of the Unix system. So he wrote in the course assignments to students:
The goal of this project is to evaluate the robustness of various UNIX utility programs, given an unpredictable input stream. This project has two parts. First, you will build a fuzz generator… Second , you will take the fuzz generator and use it to attack as many UNIX utilities as possible, with the goal of trying to break them... [2] The
professor asked the students to develop a fuzz generator in the assignment, this generator It is possible to generate unpredictable input streams, then feed these random inputs to Unix system facilities, and then try to attack these facilities and find and analyze random inputs and causes of errors. This was the birth of fuzzing, and the word fuzz that the professor used in his assignments was used to name the technique.
Fuzz testing is very similar to random testing at the beginning of its birth, and it uses random input to test the functional behavior of computer programs. So primitive fuzz testing has the same pros and cons as random testing.
Therefore, a special random testing technique used to find software vulnerabilities is called fuzz testing .
Fuzz testing has two characteristics:
- Mainly used in software security testing
- Provide some random, illegal input to attack the software, causing the software to crash.
3.1. Coverage-Based Fuzzing
For the exploration of how to improve the accuracy and depth of fuzz testing, one of them is to use the concept of code coverage (code coverage).
Code coverage includes: function coverage (function coverage) , statement coverage (statement coverage) , edge coverage (edge coverage) , condition coverage (condition coverage) and other coverage criteria, these coverage rates are to a certain extent It reflects the testing degree of the test case for the program code, that is, the higher the code coverage, the higher the testing degree of the test case for the code .
The use of coverage in fuzz testing is mainly in the process of generating test cases . The fuzzer first generates a certain number of random inputs, which are called seeds. The fuzzer then feeds these seeds into the program and retrieves the results of program execution and pre-selected coverage metrics. Next, the fuzzer will discard the seeds with low coverage according to the coverage, keep the seeds with high coverage and operate on these seeds to generate a batch of new seeds and then run the program as input. Repeat the above process until the coverage rate cannot be further improved, then terminate the generation of test cases.
That is: Randomly generate seeds—input the program—recycle program execution results, calculate coverage—retain high coverage, discard low ones—use the seeds with high coverage to perform certain operations to regenerate seeds—repeat ( When the coverage rate cannot be further improved, the generation of test cases is terminated).
Understanding: Fuzz testing based on coverage actually wants to generate some seeds with higher coverage. Because the seed is generated by "random generation" at the beginning, the seed needs to interact with the software system under test. When the seed meets the requirements and reaches the predetermined test coverage, the system also completes the test.
3.2. Mutation-Based Fuzzing
Since the initial fuzzer completely relies on randomly generated program input, it is difficult for fuzz testing to generate legal input that meets the program requirements, so that it is difficult to test the core functions of the program, and this is also one of the reasons for the extremely low code . So mutation-based fuzzing is proposed to solve this problem.
The core idea of mutation : operate on existing seeds (seeds can be legal or illegal, but in most cases, legal seeds are used, so that the quality of seed offspring obtained through mutation is higher) and seed offspring obtained through mutation Generate test input. Note that specific mutation operations need to be formulated by testers .
For example: for a legal program input "3+0" of a computer program to perform a mutation operation, the specified mutation operation (artificially specified by the programmer) is:
- Randomly add a character
- delete a character at random
- replace a character at random
Then the input after the legal input "3+0" mutation is:
- 3+ - 0 (additional character "-")
- 3+ (with one character "0" removed)
- 3 / 0 (replaced a character, turning "+" into "/")
The above is mutation-based fuzzing.
The mutation operation can be designed as a more complex and refined process, such as combining other seed quality evaluation indicators for mutation or performing some targeted mutation operation . Combining the mutation operation and the coverage index described at the end of the previous section to repeatedly test the program can make good use of the advantages of these two methods, thereby enhancing the effectiveness of fuzzing. The well-known fuzz testing tool AFL is developed based on this idea.
Understanding: Fuzz testing based on mutation is actually to generate seeds. It's just that the seeds generated in this way are more targeted (seeds generated by random testing are too blind).
3.3. Search-Based Fuzzing
Search-based fuzzing mainly relies on search algorithms. Search algorithms are also called heuristic algorithms, and their core idea is to inspire and guide algorithm execution through certain program information. They are called search algorithms because they can be performed more efficiently than random or traversal algorithms over larger search spaces .
Understanding: The essence of fuzz testing is to constantly explore the input space. To dig out the hidden defects of the program through testing is to search for a specific input in the input space, which makes the program behave abnormally at runtime; to meet a wider program coverage, it is to search for a specific input in the input space Causes more program statements to be executed while the program is running.
Summarize
At present, although the defects and problems at the beginning of random testing and fuzz testing have been overcome, the current fuzz testing still needs to be improved, such as: identifying, sorting and classifying the types of program errors triggered during fuzz testing, and Analysis of the root causes of errors , etc. The academia and industry are also constantly exploring the way of combining traditional static analysis tools such as symbolic execution technology and fuzzing technology.