Providing accurate, complete, concise, and consistent defect reports is the main evaluation indicator that reflects the professionalism and high quality of software testing. Unfortunately, some defect reports often contain too little or too much information and are disorganized and difficult to understand. This results in defects being returned, thereby delaying timely corrections, or in the worst case, because the impact of the defects is not clearly stated, developers ignore these defects, causing them to be released with the software version.
Therefore, software testing engineers must realize that writing software defect reports is an important task in the test execution process. They must first understand the expectations of readers of defect reports, follow the writing guidelines for defect reports, and write complete software defect reports. This article will explain the readers of software test defect reports, describe the main components of software defect reports and the writing requirements for each part, point out some common mistakes and practical improvement methods, and finally summarize the key points of writing defect reports.
1. Audience of defect report
Before writing a software defect report, you need to understand who the readers of the defect report are and what information the readers most want to get from the defect report. Usually, the direct readers of defect reports are software developers and quality managers. In addition, people from departments such as marketing and technical support may also need to review the defect situation. Everyone reading a defect report needs to understand the product the defect targets and the technology used. In addition, they are not software testers and may not know much about the details of specific software testing.
In summary, the information readers of defect reports most want to receive include:
- Easily search for defects in software test reports;
- The reported software defects have been isolated as necessary, and the reported defect information is more specific and accurate;
- Software developers hope to obtain the essential characteristics of the defect and the steps to reproduce it;
- Departments such as marketing and technical support hope to obtain the distribution of defect types and the degree of impact on the market and users.
One of the tasks of software testers is to write good software defect reports based on the above requirements of readers.
2. Guidelines for writing defect reports
Writing clear and complete defect reports is the best way to ensure that defects are handled correctly. It also reduces follow-up work for engineers and other quality assurance personnel.
In order to write better defect reports, you need to follow the "5C" guidelines:
- Correct: The description of each component is accurate and will not cause misunderstanding;
- Clear: The description of each component is clear and easy to understand;
- Concise: Contains only essential information and does not include any redundant content;
- Complete: Contains complete steps to reproduce the defect and other essential information;
- Consistent: Write all defect reports in a consistent format.
3. Organizational structure of defect reports
Although different software testing projects have different specific components of defect reports, the basic organizational structure is similar. A complete software defect report usually consists of the following parts:
- The title of the defect;
- Basic information about the defect;
- Test software and hardware environment;
- Tested software version;
- type of defect;
- the severity of the defect;
- Defect processing priority.
- Operation steps to reproduce the defect;
- Description of the actual results of the defect;
- Description of the correct outcome expected;
- Annotation text and captured defect image.
For specific test projects, the basic information of defects is usually relatively fixed and easy to describe. The areas where problems tend to arise when actually writing a software defect report are the title, operation steps, actual results, expected results, and comments. The following specifically discusses how to provide complete information for these "accident-prone areas". Since English is the main language of software development, the following software defect report information is all written in English.
4. Defect report writing techniques
4.1 Title
Titles should be kept short, precise, provide essential information about the defect, and be easily searchable by readers.
A good defect title should be written as follows:
- Try to write the cause and result of the defect ("After A is executed, B occurs," or "B occurs, after A is executed");
- Avoid using vague words such as "feature breaks, functionality is incorrect, behavior doesn't work," etc. Specific text should be used to explain how the feature breaks, is incorrect, or does not work;
- To facilitate searches and queries, please use keywords;
- To make it easier for others to understand, avoid making your test details jargon, slang, or overly specific.
Please review the table below, which lists problematic titles and gives examples of how to improve them.
Original Description Error Cause Improved Title
|
original description
|
wrong reason
|
Improved title
|
| Hyphenation does not work | The description is too general. When doesn't it work? | Text breaks at line's end, but no hyphen appears |
| Incorrect behavior with paragraph alignment | The description is too general. What is the incorrect behavior? | Justified alignment leaves gaps in text composition when tracking is also applied |
| Assert:CmdAssertHereInsertSomethingBad-Happens | Cause and effect information is not included. Assert is too long. | Assert, "SomethingBad" when attempting to update linked bitmap stored on server |
| After each launch then clicking edit and then copy/paste, there is too much delay | Causes and effects are not specified and excessive details are included. | Performance slows noticeably after ?rst launch and copy/paste |
| Quotes appear as symbols when they are imported | Information is not sufficiently segregated. Are all quotation marks like this? What type of symbol? | Imported smart quotes from Word appear as unrecognized characters |
hint
Use conjunctions such as "after", "when" or "during" to help describe the cause and effect of a defect, for example:
- Application crashes after input any letters in numeric field.
- Internal error occurs when closing application.
- Application suspended during email transmission."
4.2 Reproducible Steps
Reproduction steps include complete steps on how to enable others to easily reproduce the defect. In order to meet this requirement, the information on the replication steps must be complete, accurate, concise, and reproducible.
However, in the actual software testing process, there are always some bad defect reports. The main problems lie in the following three aspects:
- The steps to reproduce contain too many redundant steps, and the sentence structure is confusing, making it poorly readable and difficult to understand;
- The steps to reproduce contain too little information and are missing the necessary steps for the operation. Because the steps provided are incomplete, developers often need to make guesses and try hard to reproduce the steps, wasting a lot of time. Due to missing critical steps, these defects are often re-sent to testers by engineers as "unreproducible";
- The testers did not isolate the conditions under which the software defect occurred and the affected area. The software developers were unable to determine the software part affected by the defect and could not make complete corrections.
To avoid these problems, good replication steps should contain essential information and be written in the following manner:
- Provides preparatory steps and information for testing;
- environment variables. For example, if there is a problem with defaults or presets, debug or release builds, etc., indicate the operating system and application environment variables used;
- Set variables: indicate which printer, font or driver is the information necessary to reproduce the bug;
- Recurrence path. If there are multiple ways to trigger the defect, include those methods in the steps. Likewise, include certain paths if they do not trigger the defect.
- A simple step-by-step guide to reproducing the defect;
- Try to record only one operation for each step;
- Use numbers before each step to number the steps;
- Try to use phrases and short sentences and avoid complex sentence patterns and patterns;
- The steps to reproduce must be complete, accurate, and brief;
- No steps are missing;
- Every step is precise;
- There are no extra steps;
- Consolidate common steps into fewer steps, such as:
1. Create text frame.
2. Add text.
Can be simply combined into one step:
1. Create a new text frame and add text.
- Only record what each operation step is, do not include the results after each operation step is executed.
It is important to note again that readers of defect reports have a basic understanding of the technology, but may have limited knowledge of the details of software testing. So, on the one hand, there is no need to tell simple things like launching the product or how to open a file in a bug report. On the other hand, do not include overly detailed technical details of software testing unless this is information critical to the defect.
4.3 Actual result
The actual results are the behavior of the software and the resulting behavior after performing the reproduction steps.
The description of the actual results is much like the title of the defect. It is a re-emphasis of the title information and lists the specific performance behavior rather than simply stating "incorrect" or "not working".
If an action produces multiple defect results that are different from each other. For ease of reading, these results should be separated using lists of numbers. For example:
Actual result:
1. Assert “CmdLineofCodeHere…”
2. Assert “AlsoBrokenHere…”
Sometimes, an action will produce a result, which in turn produces another result. This situation can be difficult to summarize clearly and concisely. For example:
Actual Result：
1. Assert:“CmdLineofCodeBlahBlah…”
2. When this assert is dismissed, app becomes active but all text is unrecognizable.
3. After selecting the text by dragging the text tool, the text appears normally once again.
For these more difficult cases, there are various workarounds to make them easier to read:
- Whenever possible, break defects into multiple defect reports and use cross-references to illustrate how they are related to each other. The results of these actions are usually similar but the flaws are different. First, conduct more isolation tests to narrow the scope of defects and see if multiple defects occur;
- In the actual results section, list only one or two manifestations of the defect. Use the Notes section to list other issues with the defect;
- After the first manifestation of the defect, move subsequent steps and characteristics of the defect to the comments section. Important information is almost always contained in the first assertion or error, and other errors are variations of the first error.
4.4 Expected result
Desired results should be described in the same way that actual results are described. It is usually necessary to list what the expected results should be, and give the reasons for the expected results, which may be quoted specifications, the performance behavior of the previous version, general customer needs, the need to eliminate cluttered information, etc.
To understand more clearly what information a good desired result should contain, consider the following example:
Expected result：
The text that appears should be fully highlighted so that if the user wishes to make changes, they don't have to manually highlight and then type (as in Mac OS 10.x and Windows behavior.)
Why is this example good? Because it contains the following content:
- The correct behavior should be produced: The text that appears should be fully highlighted
- 为什么应该产生：…so that if the user wishes to make changes, they don't have to manually highlight and then type.
- Specific reference objects are given: As in OS 10.x and Windows behavior.
4.5 Notes
Comments should include supplementary information that may cause confusion in the reproduction steps, which is a further description of the operation steps. These supplementary information are more detailed content for reproducing the defect or isolating the defect.
The comments section can contain the following aspects:
- Intercept defect feature image files (Screenshots);
- Test files needed for the testing process;
- Test additional printer drivers;
- Describe the key points again to avoid developers from returning defects to testers to add more information;
- Indicate again whether the defect already existed in the previous version;
- Whether there are different performances between multiple platforms;
- Annotations contain isolation information for the defect, indicating the specific scope of the defect.
For example, a defect comment might contain the following:
Notes：
- Text displays outside frame in Win2000 and WinXP, but not Win98.
- Does not happen after screen has redrawn.
- Does not occur when two documents are open.
- Refer to attached screenshots and testing file
5. Things to note when writing defect reports
Improving the writing level of defect reports is a step-by-step process of continuously accumulating experience. Before formally submitting a defect report, please self-check the content and format of the defect report to avoid many unnecessary errors.
5.1 Self-check and ask questions
- Does the defect report contain complete, accurate, and necessary information to the reader?
- Does a defect report report a defect?
- Can readers easily search for the defect?
- Are the steps fully reproducible and clearly expressed?
- Do you include the environment variables or data files used for testing that are needed to reproduce the defect?
- Is the title of the defect written in a cause vs. effect format?
- Are actual results and expected results not described clearly enough to cause ambiguity?
5.2 Avoid common mistakes
- Use personal pronouns such as "I" and "You". You can use the verb directly or use "User" instead if necessary;
- Use emotive language and emphasis such as boldface, all caps, italics, exclamation points, question marks, etc. As long as the phenomenon of defects and complete information are objectively reflected, do not make any highly subjective criticism or ridicule on the quality of the software;
- Use vague words such as "Seems", "Appears to be", etc., but need to report confirmed defect results;
- Contains content that is considered humorous. Because different readers have different cultures and concepts, many humorous content is often difficult for others to understand accurately and may even cause misunderstandings. Information that objectively describes the defect is sufficient;
- Place uncertain test issues (Issues) in the defect management database. If you are not sure whether a certain phenomenon in the test software is a software defect, you can communicate through email or verbally to confirm that it is a defect and then report it to the database. Avoid inaccuracies in query and statistical results.
Essential tutorial for advanced Python interface automation testing (the most detailed on the entire network in 2023)