Investigation Report on the Status Quo of Software Testing in China (3)

Original Zhu Shaomin Software Quality Report Software Quality Report

Two reports have the status quo of automated testing .

1. Half of the domestic software testing teams do not use exploratory testing , although Exploratory Testing (ET) was relatively mature as early as 2006, with clear ideas, methods, operation processes and management. ET first appeared in 1983, and the relatively clear concept was in 1995. If you are not sure about ET, you can refer to: History of Definitions of ET and What exactly is agile testing and exploratory testing? Compared with ET, ST (Scripted Testing, script-based testing) is relatively traditional. Teams developing key software products can use ST as the main method, supplemented by ET. At present, this part accounts for one-third, and the expected results should account for more than 60%, which is far from the expected results . For most non-critical systems, ET can be fully implemented, but ST is also indispensable, because automated testing also belongs to ST. The ideal situation is to combine ET and ST ( hopefully not to regard ET and ST as "test methods") , for example, new functional testing can use ET, regression testing can use ST, if you want to better understand its operation, you can Reference: Reflections on a new formula for software testing .

2. The overall model of the R&D team's automated functional testing in the past year has shown a "rugby model", with small ends (UT, UI) and large middle (API), which is also considered benign . API testing ranks first, followed by UI (user interface) testing, and unit testing (UT) is the lowest. It would be better if the results of UI and UT were interchanged. From an ideal pyramid pattern, UT should be the highest, followed by API, and finally UI. There are no barriers to automated testing of the code, and the code is changed every day. The automation of interface testing requires less investment and greater benefits, and should be vigorously promoted. On the one hand, API stability is required, because its changes will have a greater impact.

From the survey results, there is no obvious difference between the front-end and the back-end , but in theory, the back-end is also the key area of automated testing, which can reach more than 90%.

3. Nearly half of the people do not understand the automatic input (or RoI) model, which is normal. Engineers focus on technology and work when they want to. They may not pay attention to input and output, but focus on RoI. Most of them are managers. But when you enter the field of software research and development, you also need to understand, and you should pay more attention to whether the work you do is worthy of a larger investment. In addition to the pyramid and rugby models that have been introduced in item 2 above, the ice cream model is the opposite of the pyramid model and should be discarded, and its proportion is close to that of the rugby model . The cupcake mode has too much UI. Although it is understandable, it is not encouraged. The survey results were significantly higher, with the cupcake model significantly higher than the football model and close to the pyramid model.

4. The implementation of functional automation testing is not ideal . Before, some classmates told me that I am more optimistic and the reality is cruel. The proportion of no automation is not low, 18.53%, and the very few automated tests are as high as 30%. And the real automation test done well (>80%), only 10%. The proportion of less than 50% in China is as high as 75.33% (27.07+29.73+18.53), which is 10% lower than the international level (39+26=65%). Both domestic and foreign, the level of automated testing is still low, although everyone thinks that Automated testing is important.

Below are the results of an international survey

5. For software projects in the past year, there is no requirement for the unit test code line coverage, which is close to 60%, and the unit test is very unsatisfactory. Less than 5% of the total coverage is required, and more than 80% is a reasonable requirement, and only 13%.

6. API automated testing tools mainly use JMeter , followed by self-developed tools, Postman and SoupUI also have a high usage rate.

7. Web function automation testing tools mainly use Selenium+WebDriver , more than half , followed by self-developed tools, most web testing tools have no market. "Other" accounts for one-third, and further investigation is required, or everyone leaves a message for recommendation.

8. Mobile application automated testing tools mainly use Appium , which is close to half (44%) , and "other" accounts for a large proportion, higher than 40%. The official Espresso application is very low, and Calabash is lower, but I still hope that you will pay more attention to these two tools.

9. The performance testing tools are relatively concentrated. The open source one belongs to JMeter, and the commercial tool belongs to LoadRunner. There are almost no rivals, and the proportion of "other" is relatively low (18.4%) . For special applications, you need to develop it yourself (21%). Gatling is good for web testing, but not many engineers are familiar with the Scala scripting language, but it is not difficult to master Scala.

10. There are too many security testing tools, which are widely distributed, and the proportion of "other" is very high (66%) , which also includes many people who do not use security testing tools, and the number of people who do security testing is lower than performance testing. Not sure which one to choose . In order to find out the situation of the security testing tools, a new survey (https://zh.surveymonkey.com/r/TT8S3N2) was conducted. Since it is a foreign website, the speed is very slow. It seems that the survey will be moved to a domestic website. , we need everyone's support .

11. Test management and defect management tools are relatively concentrated - JIRA and Zendao , self-developed and "other" account for a similar proportion, about 23% .

12. The testing work that my team will strengthen in the next 1-2 years will mainly focus on the improvement of " automated testing " and " development technology " . Automated testing is indeed unsatisfactory, and almost half of the teams are focusing on their work in 2018. They hope that automated testing will improve when the survey is conducted next year.

How do you feel after reading this report? Welcome to leave a message to discuss!

In the fourth report, we again conducted a comparative analysis of traditional IT companies and new Internet companies on these 12 items to see what their differences are.

refer to:

Investigation Report on the Status Quo of Software Testing in China (3)

Investigation Report on the Status Quo of Software Testing in China (3)

Guess you like