Author | Li Wei Director of Security Evaluation Department of Shanghai Control Security
Source | Jianyuan Lab
Community | Add WeChat ID " TICPShanghai " and join "Shanghai Kongan 51fusa Security Community"
Introduction: We have been talking about functional, performance, and special tests before. These tests mainly focus on integration testing, system verification and other stages. From this article, we will go deep into the code level and explain an important unit test. Work - software code testing.
01
The significance of code testing for functional safety
Functional safety is being mentioned more and more in the industry, and the workload of testers tends to increase when the project is implemented. The execution of unit test is an indispensable link in the overall design requirements of functional safety, but usually the task design of the test engineer for the unit test phase is relatively small and the coverage is not comprehensive.
The functional safety standard ISO 26262-6 provides the recommended verification methods for component unit testing of different Asil levels, and also provides different testing methods for structured coverage testing of codes required by different safety levels. As shown below:
ISO 26262-6:2018(E) Table 9
Starting from this article, we will introduce three methods of structured coverage measurement testing of software functional safety code. This article will introduce statement coverage.
02
What can statement coverage test test
The three coverage methods (statement coverage, branch coverage, MC/DC) mentioned in the previous chapter are designed to test the code under test from different dimensions. We know that the code is executed from top to bottom from the first line. Some lines will be skipped when some loops or logical judgments are encountered during the execution process. A simple understanding of statement coverage is to require all codes to be executed once. From this dimension Do test design and test.
Let's use the following simple piece of code as an example:
This code has a total of 19 lines. We can see that the input variables a and b determine the output value y. The statement coverage calculation for this code is to see which line our code is executed, and then divide it by 19. If we design a=9, b=10, then we get y=y+10=20, the statement executes to the 9th line, skips to the 19th line, and runs a total of 10 lines. The statement coverage rate of this test case is 50 %.
If we design a=15, b=10, it will be executed to the 10th to 14th lines. So when we design the third test case to set a=30 and b=10, these three test executions will cover all the statements, and the coverage rate of the statements will reach 100%.
In actual work testing, the code we encounter cannot be so simple. Of course, we usually use tools to test the code, and we will not perform test design and manual testing on the code purely manually. At present, most testing tools will automatically generate some use cases to test the code, but the coverage rate will not reach 100%. At this time, we need to manually add the remaining coverage by analyzing the code and the generated coverage use cases.
03
Use tools for statement coverage testing
In this chapter, we use SmartRocket TestGrid, a tool for statement coverage test analysis of the code, to introduce how the tool generates test cases to complete the test task. The operation and use instructions of the tool are too long and do not belong to the purpose of our article sharing, so we will not introduce too much here.
3.1 Examples of tool testing
For the following code:
In this code, we can see that the function has two formal parameters, namely lua_State *L and init op, and the code logic is relatively simple. When op!= LUA_OPUNM and op!= LUA_OPBNOT is true, execute api_checknelems(L, 2) , perform other operations when it is judged to be false.
After automatic analysis, the tool will generate a control flow graph as follows:
The control flow graph can help us analyze the execution path under the complex logic of the code, and is very helpful for coverage statistics. In this example, we can design two test cases to cover the cases where the judgment logic is true and false respectively, so that 100% coverage of the statement can be completed. The test cases for these two days are 1. op!= LUA_OPUNM is true, op!= LUA_OPBNOT is true, 2. The above two judgment conditions are one true and one false, or both are false.
By looking at the project header file, we can get the values of the two constants LUA_OPUNM and LUA_OPBNOT to be 12 and 13 respectively. The following figure shows the test case 1 automatically generated by the tool:
We mainly look at the assignment of the input parameter and the expected assignment of the output pointer target. This function does not write the corresponding return value for the judgment, but only performs some operations for different judgment results, so the tool does not separate the actual value from the expected value. Make an assignment. In this example, the variable op is assigned a value of 14, so this test case covers the true branch statement in the code.
The following figure shows test case 2:
In this example, the variable op is assigned a value of 12, and the judgment condition is one true and one false, so what is covered is the branch code of the false statement in the code. For the assignment of the formal parameter L, you need to check the definition of lua_State in the code. Although this code does not return a value, both branches perform certain operations, and these executions involve the formal parameter L.
We can see that the tool covers the two judgment branches through these two test cases, and executes all the codes, so the test coverage rate of this code is 100%.
04
Test summary
In the actual code coverage test, the test object is the software code. At the level of code testing, we have some conclusions to share with you, hoping to bring you some help.
1. The test cases automatically generated by the tool cannot automatically reach 100% in some cases. At this time, we need to analyze the code and the coverage of existing use cases, and manually supplement the uncovered parts.
2. Some parts of the code are naturally not 100% covered. Some of this part of the code is deliberately designed to protect the system at the code level. In this case, manual supplementary explanation is required, such as the design of the abort function to automatically Exit here code execution etc.
3. Different test tools will encounter differences in code test statistics. For example, the total number of code lines will be different due to different statistical methods such as blank lines, comments, header files, etc., and the final total number of code lines will be different, as well as coverage calculation , Some tools have different statistical methods for functions such as "{}" in the code, and even if the same test statement covers the use case design, the statistical results given by the final tool are different.
4. Various errors are usually prompted during the automatic testing process using tools. These errors are usually caused by problems such as header files, macro definitions, and array out-of-bounds. After adjustment, the automatic testing coverage of the tool will be greatly improved.
5. Since the test object is pure code, strictly speaking, there is no big industry barrier for testers to test code coverage. That is to say, our test standards can be divided into the automotive industry software functional safety standards, the rail transit industry functional safety standards, the aviation industry software functional safety standards, etc., but these software may be written based on C and C++ languages. For testers, as long as they are For software written in the same language, we will not encounter different testing technology problems due to different industries. Testers are particularly beneficial in the cross-industry versatility of code testing.
6. The testing industry has developed rapidly in recent years while majors such as the Internet, computer, and software engineering have exploded. Testing technologies such as functional testing, performance testing, automated testing, compatibility testing, and robustness testing are also becoming more and more perfect. Looking forward to new technology popularization points. Code testing used to omit this verification process in many industries, or it was self-tested by developers. Now, with the development of functional safety, it is also of great help to improve the personal capabilities of test engineers. We also hope that through this course, we will bring you more Come more upgrades.
References:
[1] ISO 26262-6-2018 Road vehicles - Functional safety - Part 6 Product development at the software