Introduction
Code coverage is an indicator to measure the tested source code of a project . Some people think it is a very useful standard. Higher code coverage may represent higher and safer quality assurance; Be skeptical, although acknowledging that coverage is a criterion of test quality, but not necessarily believing that the code that has been covered is well-tested; and some people think that code coverage is useless in the project or even useful to the project Harmful, possibly with high code coverage and poor actual project quality, giving project members a false sense of security.
Although there are many types of code coverage (function coverage, statement coverage, judgment coverage, etc.), after a period of practice, the editor decided to focus on statement coverage because it is relatively simple, rules are easy to formulate, and results are easy to visualize. This article will do some exploratory discussion on statement coverage.
measure of code coverage
As far as most of the projects I have come into contact with, most of them use the form of continuous integration, and use continuous integration tools to perform automated construction (including compilation, release, automated testing ) to verify, so as to find integration errors as soon as possible. Many teams find that this process greatly reduces integration problems, allowing teams to develop cohesive software faster. Xiaobian's current project also uses Jenkins plus coverage tools such as Gcov (C++), Emma/JaCoCo (Java) and Coverage.py ( Python ) to calculate the test coverage of the source code.
The execution timing of code coverage can be divided into two types: daily execution and each build execution. The two execution modes are independent and exist simultaneously.
For daily tasks, a scheduled task can be configured, and the scope of each execution is all source code of the entire project. Through the data generated by the tool execution, a dashboard can be built to display the code coverage data for daily/weekly/quarterly/yearly/each language. At the same time, the dashboard can also be compared between projects to evaluate and compare the quality of each project.
For the task executed by each build, the task can be hooked into the code review process. Every time a build is triggered, the code review process and code coverage scan will be executed, and the results will be visualized and fed back to the submission engineer and reviewer. classmates. As shown in the figure below, you can design certain colors for code review. For example, the green part of the code is the newly added code, and the orange and pink lines of the code are the code that has been covered by the test, which is more convenient for reviewers to watch.
According to Xiaobian's experience, it is an important time to perform and feedback code coverage scans before submitting, because this is the time when the submitted development engineers are most interested in this data, and at other times, R&D will take the initiative to improve coverage. The probability of the rate will be greatly reduced.
code coverage results
The editor collected data on the code coverage of 3 projects, 5 languages, and N commits. After statistics, we found that the coverage of most projects is between 40% and 70%, and the projects written in Python language The coverage is the highest, followed by Javascript, and the lowest is C++.
After analysis, the possibility of huge differences in the coverage of various languages is related to the structure, paradigm and difficulty of writing between languages, and different measurement tools may also affect the accuracy of code coverage data. However, please note that the above data is not enough to become a general project code coverage indicator, and these data only provide a certain data reference for improving code coverage in the future.
By deploying the continuous construction + code coverage scanning system, we continue to receive positive feedback from the cooperating R&D team. Among them, the most praised is the function of adding coverage statistics in the code review process, which has beneficially improved the code review process. It also indirectly improves the code coverage rate under test (review is also a method for improving code coverage rate).
next job
After a period of practice, we found that the statistical problem of code coverage mainly exists in code coverage tools. Because the algorithms of each tool are different, it is difficult to formulate a unified standard. Our further work may lie in the source code of these tools, and continuously improve the accuracy of coverage statistics through the practice-feedback-correction cycle.
Another problem is the coverage statistics mentioned above. Most of them exist in unit and interface level tests. For large-scale integration tests, end-to-end tests, and UI tests, such coverage is difficult to measure. Therefore, this type of testing still relies on the control of use case coverage.
I hope this article can play a certain role in your project. Although people from all walks of life have different views on code coverage, the editor firmly believes that the continuous improvement of code coverage has a positive effect on the overall quality of the project. of.
If you have any questions, welcome to add QQ group test entry God 755431660 to learn together ~