Interpretation of the Software Testing Manifesto

I have been engaged in software testing since 2000, and gradually formed my own software testing ideas. The first time I presented my own testing ideas clearly was "Full Software Testing" published in 2007. As stated in the preface: "From the first stage of project initiation From the review stage of requirements and design in one day, from late defect correction to product maintenance-in the entire software life cycle, developers and testers happily cooperate and work together to push the development efficiency and quality of software products to the highest level. A new height." The embodiment of these ideas in test management is to allow testers to integrate into the project earlier, to collaborate more actively and closely with developers, and to cooperate with project stakeholders to ensure that the project is completed on time and with high quality. ,Right now:

• The sooner testers find bugs, the sooner the project can be closed;
• Testers find as many bugs as possible, the fewer bugs left in the product, and the higher the quality of the product; 
• The work of testers and themselves (developers) is for the same goal - to release the product on time and with high quality; 

Since then, I communicated more with testers and developers in the industry, thought more, and had a clearer thinking on testing. So on September 5, 2012, I posted on Sina Weibo ( http://weibo.com/1652927771/ yArIqoCKW ) published his personal software testing manifesto:

After the declaration was released, it was affirmed by many netizens. The most powerful one belongs to @北根胶饼食蛋堡: I feel that the summary is too good, can ' t agree more , I was watching "The Soul of Soft Test" recently, and many of the opinions are the same. Although it is simple four sentences, but Every sentence can be expanded to write a book . And @ Let the Test Fly Up : See also the highlights of the eight diagrams on the lower right. Whenever it is difficult to distinguish between left and right, think of it, and use it as a starting point, try to think about it and go further, think more and strive for stability. The dividing line is not straight. It seems to reflect that there is testing ( debugging) in the development and development in the testing, but it does not completely conform to the unity of opposites.

Of course, there are also different voices, for example: @胡正辉: First, product engineering should be emphasized, then requirements engineering should be emphasized in product engineering, and quality assurance engineering should be emphasized on the basis of requirements engineering. The quality assurance project of separated product engineering is a tree without roots and water without a source. This actually has nothing to do with my declaration. There is no negation of product engineering or requirements engineering. In fact, the first sentence " encourages the determination of verification standards in advance and uses them to drive development " is to emphasize the importance of requirements. Requirements are the source of software development, and the focus here is "software testing", which is relatively limited to the connotation of testing itself and the relationship between testing and development. If it expands, I need to publish my "Software Engineering Manifesto".

There are other meaningful additions:

•  @programmer Zou Xin : It is worth thinking about for development, testing, and project management personnel. Recognize the importance of internal testing, but pay more attention to the long-term impact of products on users;
•  @王立杰-WangLijie : It is recognized that testing improves quality, but it is more advocated that quality is built-in;
•  After discussing with @Testin-Daiyibin , add an article, namely: recognize the value of automated testing, but advocate the creativity and systematicness of test analysis and design.

Then return to the theme below, explain my own software testing manifesto one by one, and briefly explain how to apply them to actual software development work.

 

1. Recognize the value of testing, but encourage the determination of verification standards in advance and use them to drive development 

First recognize the value of testing, testing is one of the important means of quality assurance, as discussed in my blog " What role does software testing play? ":

• Complete a comprehensive assessment of product quality, provide software product release (such as acceptance testing), software system deployment (such as performance planning testing), software product identification (third-party independent testing), and arbitration of disputes between the entrusting party and the entrusted party (third-party independent testing) ) and other decision-making;
• Continuous testing (including requirements review, design review, code review, etc.) can provide continuous and rapid feedback on product quality, so that product quality can be continuously and timely improved throughout the development process, and various rework can be reduced. Reduce the cost of software development;
• Through testing, find out the defects of the products to be delivered, especially to find all kinds of serious defects as much as possible, reduce or eliminate product quality risks, improve customer satisfaction, expand market share, and increase customer loyalty.
• By analyzing the defect, find out the root cause of the defect (problems in the software process, including wrong behavior) or summarize the defect mode of the software product, so as to avoid making the same mistake or similar product problems in the future, and achieve defect purpose of prevention

Therefore, the value of software testing cannot be ignored, but we have always advocated "Quality is built in". The quality of software products is gradually formed in the process of requirement analysis, function design, system design, programming, etc. Know the needs of customers in advance, clarify the acceptance criteria of software products, and develop based on these requirements and acceptance criteria. Developers know the goals they want to achieve, the requirements and limitations of the system to be realized, and can do things for the first time in their work. Yes, or in other words, the likelihood of getting things right the first time will be significantly improved, and the number of defects introduced in requirements, design, and code will be greatly reduced. It is not difficult to understand this point. If you still can't understand it well, just look at how brick walls are built? Do you pull the level line first and then build the wall, or do you pull the line after the wall is built?

Moreover, for testers, the test basis for the whole life cycle is also clear, and clear quality feedback can be provided in a timely manner, disputes between test and development are not likely to arise, and test efficiency will be significantly improved. This declaration and Acceptance Test Driven Development (ATDD) share the same ideas and connotations, and you can try to do this in both the traditional R&D process and the agile process. Although the requirements for some projects are not clear enough, or the requirements change greatly, it cannot be used as an excuse for "I am too lazy to thoroughly analyze the requirements". If you can clarify 60% of the requirements, you can't stop at 30%. Otherwise, both "iteration" and "refactoring" are synonymous with "rework" beautification. Do companies expect their teams to make mistakes and make constant corrections?

 

2. Recognize the irreplaceable value of professional testers, but encourage developers to do a good job of testing

The previous sentence has already answered the value of testing, and here is the discussion of who will do the testing work? Testing is valuable, but it is not necessarily done by full-time testers. Just as some companies (such as facebook) do not have full-time testers, the development of software products can also be carried out normally. Perhaps in some start-up companies, software services with special business models, and some mobile terminal and free products, full-time testers may not be required, but for most software products and software companies, professional testers are still needed, because the system The reason for the complexity and business is more likely to be that the test itself requires more methods and technologies. When we cannot simply master the testing methods and techniques of software products (systems), we need professional testers. Even in a relatively low-end manufacturing industry, it is not difficult to master their work skills, but the work efficiency of skilled workers is several times that of junior operators. The methods, technologies and tools involved in software testing, from functional testing to performance testing, security testing, compatibility testing, accessibility testing to reliability testing, etc., from test planning, test analysis and design to test result analysis, from Equivalence class division, decision tables, causal diagrams, model-based testing, automated testing, etc. have formed a huge system. If you are not focused, it is difficult to do well. If you are not proficient in testing, how can you have good testing efficiency? Without professional testing skills, the risk of testing is also great. These contents are fully discussed in my other blog " Will Professional Testing Teams Die or Reborn ". Just as a netizen also commented on this blog: " If there is no professional testing team , then the plane in the sky will definitely fall down for no reason, and the strike of the ECG monitor in the ICU will not be news. The nuclear bomb has not received real information. It is not impossible to launch a command to activate itself, and the world will no longer be a safe world . ”

Why are developers more encouraged to do a good job of testing? This is because:

• Unit testing is the foundation. Without unit quality, how can there be system quality? Unit testing is mainly carried out at the code level, and it is intertwined with programming. Programming and unit testing are difficult to handle separately, so unit testing is best done by developers to ensure good work efficiency and work quality;
• If a test tool is developed, should it be used by developers first, or only by testers? Whether it is a functional testing tool, a performance testing tool, or a security testing tool, developers can use it first, and the testing efficiency will be higher. Building and verifying at the same time can detect problems in time, debug and correct problems faster, and eliminate problems in the bud. This is why we have always advocated continuous construction and continuous testing;
• If developers do more tests, they will be more able to recognize their own problems and understand the causes of the problems. In the future, they will be better able to avoid the same problems in design and programming, and the effect of defect prevention will be better.

The tests that developers are encouraged to do here mainly focus on unit tests (functional and performance tests), integration tests, etc. System testing, further verification and confirmation of user needs, large-scale performance testing, compatibility testing, security testing, etc. are completed by professional testers.

 

3. Recognize the value of the test plan, but emphasize that the plan is a process of continuous adjustment based on risk

This point is easier to understand. If you do something without a plan, you will be blind, and the possibility of failure is very high. The purpose of the test plan is to clarify the test goals, test requirements (including test scope, test task priority, etc.), test risks, test resources and schedule, etc., but at the same time, the requirements will change, and the design and code quality of the development will exceed our expectations. Influenced by various factors such as anticipation, insufficient estimation of test workload and other new test risks, we may need to constantly adjust the test plan to adapt to new test requirements. The plan is important, but it is not static, that is, we emphasize:

• The test plan cannot stay on the document, it is the planning and guidance of the test process, so that the test work can be carried out more smoothly and effectively;
• The test plan is not a document, but a planning process, timely adjustment to meet the new needs of the project;
• Adjusting the test plan is also a learning process, which will help to develop a more reliable test plan in the future (for the next project).

 

4. Recognize the value of exploratory testing, but prefer to have a systematic approach to testing and a relatively standardized process

我们都知道，测试不能穷尽，测试不能做到百分之百，总是有不能测到的地方，总是有缺陷遗留下来，这就给我们留下了足够的探索空间。探索式测试（Exploratory Testing，ET）的出现正是因为在软件系统中存在许多未知的东西难以得到快速、简单的验证，需要我们转变思路，不要以固定的模式来完成测试，而是要换一种新的模式来进行测试，以提高测试效率。因为需求不清楚、时间紧等各种原因，探索式测试才更有效，在一定程度上是因为软件开发本身的问题，所以，我也戏称“敏捷开发”为“探索式开发”。从这个意义上讲，探索式测试方法是不得已而为之的一种方式。在传统行业，没有看到一种“探索式检验”（除了食品安全检验，在我国还不够成熟，可能会采用探索式检验，哈哈），而是有明确的技术规格，有相应的检测仪器或方法进行检验，可以明确地给出检查结果。

探索式测试作为明确的术语或概念，最早是由测试专家CemKaner博士在1983年提出的，距今天差不多有30年，但绝大多数测试人只是最近几年才听到或熟知这个概念。说明其价值是有限的，如果价值很高，也不至于我们现在才比较关注的。但最近几年探索式测试很热，为什么？

一方面要感谢James A. Whittaker撰写的《ExploratorySoftware Testing》一书，比较全面地介绍了探索式软件测试（国内是2010年引进本书的，但也有不足，我在为史亮和高翔写的《探索式测试实践之路》的序中谈到这一点），对推广探索式测试有很大的促进作用。另方面，在互联网时代，需求衍变越来越快；软件已经成为一种服务（SaaS），迭代周期越来越频繁。敏捷方法开始流行，被软件企业广泛采用，敏捷测试随之而生，正是探索式测试用武之时。而且探索式测试的确给人一些新鲜的感觉，将测试工作变成更有趣的探索式活动，在享受工作的同时完成测试，容易受到测试工程师的欢迎。

探索式测试也在不断发展，人们试图帮助它建立一套方法体系，例如James Bach提出的基于会话的测试管理(Session Based Test Management,简称 SBTM)。该管理方法将测试任务分解成一系列会话(Sessions，发生在特定时间盒内的会话活动，对软件系统的测试就是看成不断地问系统的过程，从系统那里获得答案，探索式测试的会话特征更为明显),测试人员在会话过程中完成一个特定测试任务的设计、执行和记录。

但从探索式测试的“探索”概念本身来看，还是强调“设计与执行”同时发生的特点来看，探索式测试更多强调人的创造性，强调随软件功能的使用对其理解不断深入来发现问题，更强调这种上下文驱动的思维模式，而对验收的标准、验证的具体指标缺乏关注，更谈不上测试需求的分析、测试的系统设计，在系统性和规范性方面有很大的欠缺，所以难以得到国际标准的支持，在多数软件产品的测试工作中探索式测试只能起着辅助、补充的作用。

任何严重的缺陷的遗漏可能给公司带来不可估量的损失，软件测试更注重对软件质量的全面评估，最大程度地减少软件产品的质量风险，从这个意义看，测试目标、测试需求、测试风险等都是非常重要的，需要认真分析，然后在此基础上进行系统的测试设计。测试的结果需要严格的覆盖率衡量，而要确保高覆盖率，需要事先进行精心的设计，从业务流程、数据流程、用户场景等各个方面进行细致分析，采用合适的测试方法设计出相应的测试用例来覆盖流程路径、数据输入空间以及各种产品使用的场景。事先能从需求覆盖出发来设计测试用例，事后还可以从代码覆盖来检验测试的效果。当然，更理想的方式是用基于模型的测试或形式化方法来验证系统的需求，给出更客观的、更准确的质量评估，我们对产品的发布就更有信心，客户就能得到高质量的产品或服务。

 

5. 认可发现缺陷的价值，但更重视对软件产品质量的全面评估与持续反馈

If a defect is found and corrected, the product quality will reduce a risk; the more defects are found in the current product, the more product quality risk can be eliminated, which is one aspect of the value of software testing. But we cannot have the idea that the more defects a tester finds, the greater the value of the tester. For example, we can't wait for the developers to complete the design and code work before we discover problems to reflect the value of testing. We can't clearly watch developers make mistakes, or clearly know that developers may make mistakes in some places, and we don't give reminders or help, but wait until they finish their work, and then we find out the problem. To reflect our value.

The correct approach is to provide feedback on quality in a timely manner, which may be a reminder of quality risks, or a complaint about quality concerns, or a positive suggestion for quality improvement. For example:

• During requirements analysis, if you find that the requirements are not clear enough, and you find any places that may cause confusion to development and testing, you must point them out in time and help to correct them.
• If you find that the newly added function in the document does not make much sense, or if you find it difficult to see the value of its function to customers, you should actively communicate with the product manager and suggest removing this function, or ask the product manager to explain clearly and convince yourself.
• If you feel that the development task arrangement is unreasonable, or that the discussion and training before development are not enough, and your understanding of the business is still superficial, you must communicate with the development in time to help eliminate this potential quality risk;
• If you find that the development does not pay attention to unit testing, or does little unit testing, it is necessary to assist the development to do a good job in unit testing, provide unit testing guidance, provide unit testing framework, and provide all services that can help development improve unit testing.
• If it is found that individual development engineers do not abide by the programming specifications, the quality feedback mechanism must be activated...
•  … …

this means:

• Testing work is not a certain environment or staged work in the software development life cycle, but runs through the entire software development life cycle, and testers pay attention to quality all the time;
• Testers should not only pay attention to existing product defects, but also pay attention to problems that may lead to defects, and try to help product requirements personnel, designers, and programmers to prevent quality problems from occurring.
• Testing is not only the work of testers, but also has a relationship with other teams (personnel) in software development; testing is not a matter within the test team, but a matter of the entire development team.