After the integration test is passed, each module has been assembled into a complete software package, and system testing needs to be performed. Traditional system testing refers to the software system that has passed the integration test. As an important part of the computer system, it will be tested in combination with other system elements such as computer hardware, peripheral equipment, and supporting software. Compare the definitions to find out where the software does not meet or conflict with the requirement specifications, so as to propose a more complete solution. Here, the particularity of virtual reality (Virtual Reality, VR) project testing that requires software and hardware support is particularly pointed out.
This section takes the system test summary report of "an educational administration platform system" as an example to introduce how the system test activities of the software project are organized.
01. Preparation before the test
1. Purpose of the test
System testing mainly has the following four purposes.
(1) Obtain an evaluation of software quality by analyzing test results.
(2) Analyze the test process, products, resources, and information to provide reference for formulating test plans in the future.
(3) Evaluate whether test execution and test plans meet product requirements.
(4) Analyze system defects and provide suggestions for repairing and preventing bugs.
2. Definition of terms
If the following defects occur, the test will define the problem as a serious bug.
(1) The system is unresponsive and is in a crashed state, requiring manual repair before recovery.
(2) After selecting a menu, "page cannot be displayed" appears or an abnormal error is returned.
(3) After an operation (add, modify, delete, etc.), "the page cannot be displayed" appears or an abnormal error is returned.
(4) When verifying the required fields, the required fields are not entered, but "the page cannot be displayed" appears or an abnormal error is returned.
(5) The system defines a field that cannot be repeated. After entering repeated data, "the page cannot be displayed" appears or an abnormal error is returned.
02. Test summary
The software system test lasted 22 days in total, tested 124 function points, executed 1780 test cases, and executed an average of 14.3 test cases for each function point, and found 244 bugs in total, including 42 serious bugs and 35 invalid bugs. On average, there are 1.8 bugs per tested function point.
A total of 9 test versions have been released for this software, among which, V1~V4 is the planned iterative development version (for the baseline identification of the project plan), and V5~V9 is the regression test version. The planned test version V1~V4 test progress was completed 2 days late according to the project plan and the report was submitted. V5~V9 are unplanned regression test versions, and the overall test was completed 5 days later than planned.
This software test implements defect tracking management through the defect management in the project management tool. There are detailed bug analysis tables and stage test reports in the V1~V4 test stages.
1. Functional test cases
(1) The main functions realized by the system, including the query, addition, modification, and deletion of the content of each entity.
(2) Secondary functions implemented by the system, including automatic semester switching, assigning permissions to users, binding course teachers to classes, binding syllabus to courses, permission control menu buttons, etc.
(3) The input and output fields specified by the requirements, and the input restrictions specified by the requirements.
2. Usability test cases
(1) The correctness, consistency, and comprehensibility of the prompt information of the operation button.
(2) Limit the correctness, consistency and comprehensibility of prompt information.
(3) Identification of required fields.
(4) Comprehensibility of the input method.
(5) The consistency between the data language and the interface language under the Chinese interface.
03. Test environment
1. Software and hardware environment
The software and hardware environment for this system test is shown in Table 1.
■ Table 1 System test software and hardware environment configuration
2. Network environment
The network topology of this system test is shown in Figure 1.
■ Figure 1 System test network topology
04. Test results
1. Bug trend chart
A total of 9 test versions were released for this system test, among which, V1~V4 are planned iterative development versions, and V5~V9 are regression test versions. The bug version trend diagram is shown in Figure 2.
■ Figure 2 Bug version trend chart
(1) The first stage (V1~V4): Incremental confirmation test.
V1: From Figure 7-3, we can see that V1 has a total of 43 bugs. Because the V1 version has a functional module that was tested only in the V2 version, there are relatively few test modules in V1, and there are relatively few bugs in this version.
V2: Since a functional module in V1 is added to V2 for testing, this version not only verifies the bugs in V1, but also performs a regression test on V1, so the number of bugs in V2 has an obvious growth trend compared with V1 .
V3: Because of the V2 version Bug acceptance test and the V1 and V2 regression tests, a total of 23 bugs were found in the V3 version, which has decreased significantly, indicating that the efficiency and quality of the previous test work and the corrections made by the program developers are relatively high high.
V4: The number of bugs in the V4 version has a slight upward trend, because a new development function module is proposed, and the definition of requirements for this version has changed.
(2) The second stage (V5~V9): Bug verification and function regression confirmation test.
V5 and V6 have carried out regression testing, and V7 has verified the previous bugs.
V5: The first round of regression testing was carried out, and the number of bugs found was 21.
V6: Carry out the second round of regression testing. The first regression testing did not involve the test of the permission control menu button. This regression test focused on this aspect of the test, and found a lot of bugs related to permissions.
V7: No comprehensive regression testing was conducted, and only bugs that failed to pass verification in V1~V6 were verified, so the number of bugs is obviously relatively small.
V8: The V8 version has carried out a comprehensive regression test, and at the same time focused on testing the authority control, business process, and front-to-back relationship mapping, so 7 of the bugs found this time are of a serious level, indicating the function and business control of the system test in the final sprint stage There are still certain problems.
V9: The V9 version has been modified and verified by the program developers more precisely. It is a version submitted after a 4-day delay. This time, a comprehensive regression test has been carried out on the re-released version, especially for serious-level bugs. No bugs were found. Problems, only some functional issues have been corrected for operational convenience. Overall, the system function has been stable.
2. Bug severity
As shown in Figure 3, the bugs found in the test are mainly concentrated in the "general" and "minor" levels, which belong to general defects, but there are 42 serious bugs in the test, and the serious bugs are mainly manifested in the following 4 aspect.
(1) The main functions of the system are not realized.
(2) After adding data code duplication, an error that the page cannot be found occurs.
(3) The function of automatic school semester switching control has not been effectively constrained, and some course scheduling data has an abnormal query problem.
(4) The designed database role management and control are chaotic, some roles cannot find the page, or the page does not have the operation authority and so on.
■ Figure 3 Bug severity chart
3. Bug introduction stage
As shown in Figure 4, the bugs found in this system test are mainly bugs in the background coding stage and the foreground coding stage, even close to 80% of the total number of bugs.
■ Figure 4 Bug introduction stage analysis
4. Reasons for bug introduction
As shown in Figure 5, the bugs found in this system test are mainly due to the unsatisfactory requirements of front-end coding, background coding and usability, which account for 78% of all bugs.
■ Figure 5 Analysis of causes of bug introduction
5. Bug status distribution
As shown in the bug state diagram in Figure 6, it can be seen that there are 3 bugs that have not been effectively resolved. This is because the requirements of the graduation design process in the later stage have changed and need to be redesigned, so they have not been processed for the time being, and other parts have been resolved.
■ Figure 6 Bug state diagram
05. Test conclusion
1. Functionality
The system correctly realizes the academic function based on semester management, realizes the automatic acquisition and initialization of training programs, courses and syllabus, realizes the functions of semester teaching arrangement, experiment arrangement, graduation design management, etc., and also realizes roles and permissions Management query, add, modify, delete and other operations, the system also realizes the function of subdividing authority control to some menu buttons.
While the system realizes the graduation project management function, there are problems such as graduate students' voluntary selection and teacher review process authority control is not strict, and the authority design can be further supplemented and perfected.
2. Ease of use
The existing system achieves the following ease of use.
(1) Consistency and comprehensibility of prompt information related to query, add, delete, and modify operations.
(2) Correctness of input restrictions.
(3) The correctness, comprehensibility and consistency of the input restriction prompt information.
The existing system has the following usability deficiencies.
(1) The layout of the interface is not beautiful, and the operation of some page function buttons is not in line with public habits.
(2) The input and output fields have poor understandability.
(3) The input lacks explanatory notes.
(4) The corresponding information in Chinese and English is not completely correct.
3. Reliability
The reliability control of the existing system is not strict enough, and many controls are realized through page control. If the page control fails, users may directly insert data into the database, causing errors.
The fault tolerance of the existing system is not high. If an error occurs in the system, the error type returned is a page not found error, and it cannot be restored to the state before the error.
4. Compatibility
The existing system is mainly tested under Windows, and is compatible with IE11 browser, Chrome84 browser and Firefox88 browser, but compatibility tests for other platform browsers have not been conducted.
5. Security
Existing systems control the following security issues.
(1) After saving a logged-in page, you cannot open it locally and operate it independently without logging in.
(2) If you directly enter the URL of a certain page, you cannot open the page and perform operations, and you will be redirected to the login page.
The following security features are not implemented in the existing system.
(1) Username and password are case sensitive.
(2) There is no limit on the number of login errors.