Chapter 6 Test Report and Test Evaluation

---

---

foreword

Software testing is an important part of software engineering and an important means to ensure software quality. With the increasing complexity of software and the continuous development of the software industry, software testing has received more and more attention. This chapter mainly introduces how to report found software defects and related knowledge about software testing and evaluation.

---

1. Software defects and types of software defects

1. Definition and description of software defects

According to the general definition, as long as one of the following five rules is met, it is called a software defect.
• The software does not fulfill the functions specified in the software specification sheet.
• The software exceeds the scope specified in the software specification sheet.
• The software does not achieve what is stated in the software specification as it should.
• An error occurred in the software operation.
• Software testers think the software is difficult to understand, difficult to use, slow to run, or end users think the software is not effective to use.

The following are valid description rules for software defects.
• Single Accuracy : Each report is only for one software defect.
• Reproducible : Provide the precise steps of this defect, so that developers can understand, reproduce and fix the defect.
• Complete and unified : Provide complete and unified software defect repair steps and information, such as picture information, log files, etc.
• Short and concise : By using keywords, the description of the title of the software defect is short and concise, and can accurately explain the phenomenon of the defect.
• Specific conditions : Software defect descriptions should not ignore those seemingly detailed but necessary specific conditions (such as specific operating systems, browsers), which can provide clues to help developers find the cause of defects.
• Supplementation and improvement : From the discovery of software defects, the tester's responsibility is to ensure that he is correctly reported and given due attention, and continue to monitor the entire process of its repair.
• No evaluation : Software defect reports are aimed at software products, so software defect descriptions should not contain personal opinions, and should not be evaluated by developers.

2. Types of software defects

(1) The function is not normal
(2) The software is inconvenient to use
(3) The structure of the software is well planned
(4) The functions provided are not sufficient
(5) The interaction with the software operator is poor
(6) The usability is not good Good
(7) Not doing a good job of error handling
(8) Boundary errors
(9) Calculation errors
(10) Errors caused by using it for a period of time
(11) Errors in the control process
(12) Errors generated under the pressure of large data volumes
(13) Errors generated under different hardware environments
(14) Errors generated by poor version control
(15) Errors in software documentation

3. Properties of software defects

(1) Defect ID : It is a unique ID to mark a defect, which can be represented by a digital serial number, and is generally automatically generated by the software defect tracking management system.

(2) Defect description and defect annotation : refer to the detailed description of the defect discovery process and some auxiliary information about the defect.

(3) Defect type : refers to the defect types classified according to the natural attributes of defects, generally including functional defects, user interface defects, document defects, software configuration defects, performance defects, system/module interface defects, etc.
• Functional defects : defects that affect the functions and logic of various software systems.
• Defects in the user interface : affect the user interface and human-computer interaction characteristics, including defects in screen format, user input flexibility, and result input format.
• Documentation defect : Refers to the documentation that affects the release and maintenance of the software, including notes, user manuals, design documents, etc.
• Software Configuration Defects : Errors due to software configuration repositories, change management, or version control.
• Performance defects : do not meet the measurable property values ​​of the system, such as execution time, transaction processing rate, etc.
• System/module interface defects : mismatches, conflicts, etc. with other components, modules or device drivers, call parameters, control blocks or parameter lists, etc.

(4) Defect severity : Severity indicates the damage degree of software defects to software quality, reflecting its impact on products and users, that is, how the existence of software defects will affect software functions and performance. The judgment of the severity of a software defect should be made from the point of view of the end user of the software, that is, the severity of the defect should be judged for the user, and the seriousness of the adverse consequences caused by the defect to the user should be considered.
Generally divided into: fatal (Fatal), serious (Critical), general (Major), minor (Minor).
• Fatal : Any major function of the system is completely lost, user data is damaged, the system crashes, hangs, freezes, or endangers personal safety.
• Serious : The main functions of the system are partially lost, the data cannot be saved, the secondary functions of the system are completely lost, and the functions or services provided by the system are obviously affected.
• General : The secondary functions of the system are not fully realized, but it does not affect the normal use of users. For example: some problems such as inaccurate prompt information, poor user interface, and long operation time.
• Small : It makes the operator inconvenient or troublesome, but it does not affect the operation and execution of functions, such as some minor problems such as typos and irregular text arrangement that do not affect product understanding.

(5) Possibility of defect occurrence : refers to the frequency of occurrence of a defect, generally divided into: always, usually, sometimes, rarely, etc.
• Always : always generate this bug, the frequency of which is 100%.
• Usually : According to the test cases, this software defect will usually occur, and its frequency is about 80% to 90%.
• Sometimes : According to the test cases, this software defect sometimes occurs, and its frequency is about 30% to 50%.
• Rarely : According to the test cases, this software defect rarely occurs, and its frequency is about 1% to 5%.

(6) Defect priority : Priority indicates the importance of repairing defects and when they should be repaired. It is an index indicating the order of processing and correcting software defects, that is, which defects need to be corrected first and which defects can be corrected later. Determining the priority of software defects is more about considering the problem from the perspective of software development engineers, because the order of defect correction is a complicated process, some of which are not purely technical issues, and developers are more familiar with software codes and can be more clear than test engineers. Difficulty and risk of correcting defects. Generally divided into: high priority, high priority, normal queuing, low priority and so on.
• Highest priority : Refers to some critical errors, the defect makes the system almost unusable or the test cannot continue, and needs to be repaired immediately.
• High Priority : The defect is serious, affects testing, and needs to be repaired first.
• Normal Queue : Defects are normally queued for fixes and must be fixed before product release.
• Low Priority : Defects can be corrected when the developer has time.
The priority of software defects will change during the project.

(7) Defect Status : Used to describe the progress of a defect through a tracked repair process. Generally divided into: active or open, corrected or repaired, closed or inactive, reopened, postponed, reserved, cannot be reproduced, needs more information, etc.
• Activated or Opened : The issue has not been resolved, it exists in the source code, a "submitted defect" is confirmed, and it is pending, such as a newly reported defect.
• Fixed or Repaired : Defects that have been checked and fixed by developers, passed unit tests, and considered resolved but have not been verified by testers.
• Closed or Inactive : After verification by the tester, the state after confirming that the defect does not exist.
• Reopened : The state after verification by the tester, after confirming that the defect does not exist.
• Deferred : This software defect can be resolved in the next release.
• Reserved : Defects that cannot be fixed by the developer due to technical reasons or defects in third-party software.
• Unreproducible : The software defect cannot be reproduced by development, and testers are required to check the steps for defect reproduction.
• Need more information : The development can reproduce the software defect, but the developer needs some information, such as the log file of the defect, pictures, etc.

(8) Origin of software defect : refers to the stage when the fault or event caused by the defect is detected for the first time. Divided into: requirements, architecture, design, coding, testing, users, etc.
• Requirements : Software defects discovered during the requirements phase.
• Design : Software defects discovered during the detailed design phase.
• Coding : Software defects found during the coding phase.
• Testing : Software defects found during the testing phase.
• User : Software defects found during the user use phase.

(9) The source of the software defect : refers to the place where the software defect occurs. Divided into: requirements specification, design document, system integration interface, data flow (library), program code, etc.
• Requirements specification : problems caused by errors or unclear requirements specifications.
• Design Documentation : The design documentation description is inaccurate. Inconsistencies with the requirements specification.
• System integration interface : defects caused by the mismatch of system module parameters and lack of coordination among development groups.
• Data flow (library) : Defects due to errors in the data dictionary, database.
• Program Code : Defects caused purely by problems in coding.

(10) Defect root cause : refers to the root cause of software defects, so as to further seek to improve the software development process and improve the management level. Generally: test strategy, process, tools and methods, team/people, lack of organization and communication, hardware, software, work environment, etc.
• Testing strategy : Wrong testing scope, misinterpreting testing objectives, exceeding testing capabilities, etc.
• Processes, tools and methods : Ineffective requirements gathering process, fruitful risk management process, unused project management methods, no estimating procedures, ineffective change control process, etc.
• Team/People : Project teams have overlapping responsibilities and lack of training. Inexperienced project team, lack of morale, impure motivation, etc.
• Lack of organization and communication : Lack of user involvement, unclear responsibilities, management failures, etc.
• Hardware : Incorrect hardware configuration, lack of hardware, or processor defects lead to loss of arithmetic precision, memory overflow, etc.
• Software : Incorrect or lack of software settings, or errors in the operating system leading to failure to release resources, errors in tool software, errors in compilers, etc.
Working environment: organization adjustment, budget change, poor working environment, etc.

2. The life cycle of software defects

The following is the simplest software defect life cycle situation, which systematically represents the various stages of software defects from being discovered: (1)
Discovery -opening : testers find software defects and submit software defects to developers.
(2) Open-Fix : The developer reproduces and fixes the defect, and then submits it to the tester for verification.
(3) Repair - Close : The tester verifies that the repaired software closes the defects that no longer exist.

In some cases, the life cycle becomes a bit more complex, as shown in the following figure: As

you can see, software defects may undergo several revisions and restatements during the life cycle, sometimes repeatedly. The situation shown in the figure above is quite common in actual testing work. Usually, there are two additional states in the software defect life cycle: ① review state and ② postponed state .

3. Isolate and reproduce software defects

If the found software defect can only be reproduced by taking complicated steps, or cannot be reproduced at all, in this case, the following methods can be adopted to separate and reproduce the software defect. Practice has proved that these methods are helpful to testers.
(1) Ensure that all steps are recorded
(2) Pay attention to time and operating conditions
(3) Pay attention to software boundary conditions, memory capacity and data overflow problems
(4) Pay attention to software defects caused by the sequence of events
(5) ) Consider resource dependencies and the interaction of memory, network, and hardware sharing
(6) Don't ignore hardware

4. Software testers need to face software defects correctly

(1) Not every software defect found by testers must be repaired.
The reasons for not repairing software defects are:
① Not enough time
② It is not considered a real software defect
③ The risk of repairing is too high
④ It is not worth repairing

(2) The number of defects found does not explain the quality of the software
(3) Do not expect to find all the defects in the software

5. Report software bugs

1. Basic Principles of Reporting Software Defects

(1) Report software defects as soon as possible
(2) Describe software defects effectively

2. IEEE Software Defect Report Template

The ANS/IEEE829-1998 standard defines a document called a software defect report, which is used to report "any abnormal event that occurs during the testing process".
(1) Software defect report identifier
Specifies the unique ID of the software defect report for location and reference.

(2) Summary of software defects
Concisely state the facts and summarize software defects. Give the version reference information of the tested software, related test cases and test instructions and other information. For any identified software defect, relevant test cases should be given. If a certain software defect is discovered by accident, a test case that can discover this unexpected software defect should also be written.

(3) Description of software defect
The writer of the software defect report should provide enough information in the report, so that the general repair personnel can understand and reproduce the occurrence process of the event. The following are the various contents in the software defect description.
• Input
Describes the input (eg, file, keystroke, etc.) used during the actual test.
• Expected result
This result comes from the design result of the test case being run when the event occurs.
• Actual results
Record the actual running results here.
• Anomalies
refer to how much actual results differ from expected results. Also record some other data (if this data is very important), for example, the amount of data about the system is too small or too large, the last day of the month, etc.
• Date and Time
The date and time the software defect occurred.
• Procedural step
The step at which a software defect occurs. This is especially important if you are using long, complex test procedures.
• Test environment
The environment used. For example, system test environment, acceptance test environment, customer's test environment, test site, etc.
• Reproduction Attempts
How many attempts were made to reproduce the test.
• Tester
The details of the person who conducted the test.
• Witnesses
Know about other people who were in the test.

(4) Impact
The "impact" of a software defect report refers to the potential impact of a software defect on users. When reporting a software defect, the tester classifies the software defect and points out its impact in a concise manner. A frequently used approach is to assign severity and priority to software defects. Of course, the specific methods vary from company to company, but the general principles are the same. Practical experience with testing has shown that while it may never be possible to fully overcome the imprecision in determining severity and prioritization, it is possible to describe major characteristics such as minor, major and severe in defining ratings. Reduce this imprecision to a certain extent.

6. Tracking and management of software defects

1. Software defect tracking management system

(1) The role of the software defect management system
The application of the software defect management system in the testing work has the following advantages:
① Maintain a high-efficiency testing process
② Improve the quality of software defect reports
③ Implement real-time management and security control
④ Utilizing the system is also beneficial Collaborative work among project team members

(2) Implementation principle of defect tracking management
The software defect tracking management system can manage software defects through adding, modifying, sorting, searching and storing operations.

2. Manual reporting and tracking of software defects

7. Evaluation of software testing

1. Coverage evaluation

Coverage evaluation indicators are used to measure the completeness of software testing, so coverage can be used as a measure of test effectiveness. The most commonly used coverage evaluations are requirements-based test coverage and code-based test coverage, which respectively refer to the completeness evaluation of requirements (requirements-based) or code design / implementation standards (code-based).

2. Quality evaluation

The evaluation of test coverage provides an evaluation of the completeness of the test, while the evaluation of the defects found during the test provides the best software quality indicators.

Defect analysis typically provides defect assessment in the form of four types of metrics:
• Defect discovery rate .
• Defect latency .
• Defect density .
• Overall software defect removal rate .

8. Test summary report

---

Summarize

The above is what I will talk about today. This article only briefly introduces the basic concepts of software testing. I hope it can be helpful to you who are reading. If you are also interested in software testing, just follow me to learn.

If you think my writing is not bad, please give me more likes and encouragement. Your support is also the biggest motivation for me to keep moving forward. At the same time, you are welcome to share this article with your friends and learn together. Finally, everyone is welcome to discuss with me the problems encountered in the learning process in private messages and comments, so that everyone can make progress together.