Article finishing in Zhushao Min ( "full software testing" the authors) speech at the General Assembly TiD2019 quality competitiveness of "AI software testing technology help achieve" unity of quality and efficiency ""
TiD2019 quality competitive Assembly has invited domestic well-known software testing expert, Soft Alliance standard evaluation team leader, "the entire software testing (3rd Edition)" by Zhushao Min teacher for the participants to bring "AI software testing technology help achieve" quality way integration "" wonderful speech. Zhushao Min teacher from the current test and automation of the situation, AI and AI technology to enhance the efficiency of the test helps test coverage about the three areas.
Currently testing and automation of the situation
Currently the software quality engineering and software development environment faster and faster pace, the development team is subject to competition in the market, need to continue to build, continuous integration, continuous testing in order to achieve continuous delivery. Continuous testing is necessary in today's software development, the ultimate goal of testing needed to achieve this ultimate state.
At Google, a test case execution can not be more than 60 seconds, the shorter the time the more valuable. Google also stressed that while automated testing. Before Google has a well-known GTAC, it is the automated testing session. Now the meeting has been changed to project performance. At present, the performance gradually pay attention to it. Google before the company's testing department and quality department called the department of engineering efficiency, now including Ali, Baidu and other large companies have begun to focus on performance. Performance is reflected not only from the speed, but also reflect the quality up.
So why do the test? The final test for the business service. Whether we are developing or testing to do, ultimately in order to solve business problems, to solve customer problems. Develop more to build quality, and the quality of the test is the guardian, can help developers make fewer mistakes, reminding developed areas for improvement. This will help teams build better quality. As the guardian of quality is not entirely to be the last to do system testing, acceptance testing, quality can guard to guard the quality of the whole process.
Efficiency test is the focus of attention in today's society. White-box testing, black box testing, manual testing, automated testing, which a more efficient it? Zhushao Min introduced the efficiency of manual testing can sometimes be high, but now less static tests done, more to the dynamic test. Static testing tools to do automatic detection of some security and vulnerability patterns, not need to invest a lot of manpower, manpower only need to analyze the results. Some tools false alarm rate may be high, basically reduced to 10% higher, then it may also have a 20% -30%.
Automated regression testing range will bring proven efficient, agile development is growing, certainly not completely rely on manual testing. In general, software testing is more suitable for automated regression testing, although this view is relatively old, but still applies to today's environment.
However, the development tool capacity is relatively weak, overly concerned tools, focus on the script itself, often overlooked business, product and test the quality and coverage. Face a lot of code and components, experienced engineers exploratory tests on systems and business level to ensure quality. But the machine is very difficult to understand this business of artificial intelligence tools today are mostly out of business, combined with real tools more closely with the business is still very rare. AI testing tools can understand the code, but the data really combine business with very little. Like machine learning sometimes requires a lot of data, but based on real business data is relatively small, such as open-source code for 4TB there are, the more these codes can be done Java, C, Python language learning in this area, and even do automatic positioning to Bug, Bug auto repair. But if the real business combination, and found that the business combination of this tool is still very weak, and needs to be tested on people.
From the test in terms of quality, code coverage is just a more objective results, but be 100% code coverage does not guarantee high quality . Development can be done easily 100% code coverage, but does not measure illegal or enter some special value. So this perspective, the lines of code to achieve 100% coverage of more defects will be missed. Today most companies even cover 100% of the lines of code are not required, only a part of the business lines of code required to achieve 80% coverage. Code is the control flow and data flow. Line coverage, the branch control flow is covered. The main data flow is defined and referenced variables in general will not do the software code analysis, only in the aerospace, nuclear industry, or some other life-related system will do the definition of variables and reference analysis.
Testing is to meet the requirements of the business. Before going to a financial technology company or some other exchange companies and found that they did not really do the business test, and even a business process diagrams are not , can not guarantee that the business no problems. Business process diagram is a business-level control flow, and data flow services. Functional testing done before the time, to do more to test data, but the data may be complicated business, and do not fully validated based on data flow diagrams, just to verify the data module or function of the data does not really validate the entire business data flow. So business decision-making, business processes, business rules, business operations, business data, or even the future business of sustainable development, including its security, we need to verify.
In addition to the business user role, user behavior, events, conditions, scenarios, data, operating environment, today there are more micro cloud platform services architecture, as well as a combination of configuration parameters need to consider testing.
Uniform quality and efficiency
当公司是做敏捷开发，拥有专职的测试工程师时，上述“质效合一模型”是值得大家思考和引用的。当开发做新功能的时候，针对新功能做设计、写代码，开发自动化测试比较困难，无法调试脚本，被测对象也不稳定，即使能调效率也是比较低的。所以从这个角度来讲，新功能最好用探索式测试去做，因为不用写测试用例，把省下了写测试用例的时间，来写上一个迭代的、已经通过评审/验证的、稳定的功能，并针对上一个版本的功能做自动化测试。在这样的情况下你才能更好地保证质量，又能提高效率，因为所有的回归测试是自动化的，新功能相对是未知的，用探索式测试发挥人的潜力与智慧，就更好地做到质量和效率的统一。
虽然现在都在提倡自动化测试，上面这张图的数据反映了自动化测试的国际水平，但是工信部五所（赛宝）做的国内自动化测试现状调查结果比这个要差很多。这里，自动化测试做得好的只有32%左右，真正做到90%以上的只有4%。还有19%做得还可以，自动化比重超过了50%。可以看出，大多数的自动化测试实际做得不够好。
AI提高测试效率
早期是手工测试，然后到自动化测试，到后面用健壮的TA工具与开源框架集成，得到大家的欢迎和广泛的应用。
现在行业在推云计算、大数据，希望公司可以构建一个代码库，把所有的代码都放在一起，这样可以更好地复用或者学习。公司也在云化自己的开发环境和测试环境，这样才能有更多的数据积累下来，可以实现更好的机器学习。当前基于云计算平台和大数据，基于生态链工具，可以更好地去完成自动化测试，更具有弹性。未来测试基于新的微服务架构或者其他架构，真正做到智慧、自主。在架构方面甚至在需求工程方面，正在研究这种自主的系统，算法不变可以自主扩展、动态成长。
未来人更多地是做分析、做设计、建模、训练模型。所有的执行应该交给工具去做的。而且在今天是有这个条件，有大量的数据，许多算法也都是成熟的。Google提供了TensorFlow（TF）或其他的一些平台，但这些是脱离业务的。针对自己的业务，需要大量的分析来构建对应的模型。另外人机交互智能更有价值，不能完全靠机器， Google搞AlphaGo，先是AlphaGo做出来才了有AlphaZero，还是要有一个过程。人工智能或者机器人，或者未来的测试机器人是需要人去训练的。就相当于把测试工程师的经验和知识，对业务流程和业务场景的理解赋能给机器人。让工具与用户见面，跟业务有更强的绑定。现在人类可以给工具做按钮、菜单、文字、图标的识别训练，但它还不能真正的认知业务流程。东南大学、复旦大学正在研究知识图谱，在特定的业务领域可以帮助计算机去提高认知。
一般来讲，人工智能有计算智能、感知智能、认知智能，现在计算智能和感知智能已经相对比较成熟，但认知智能比较弱，要让人工智能真正去理解人类业务还是比较困难的。需要人类构建业务流程或者知识图谱，让机器去理解、学习。
人工智能需要靠很大的人力，没有大量的数据标识，机器是不会去识别或者做预测的。所以有时候讲在某些落后地区，非一、二线城市会有大量的人相当于民工一样在做数据标识，这个过程就会比较慢。
人工智能的算法不一定要有数据标识，监督学习需要标识数据，还有无监督学习、强化学习。目前无监督的学习更受欢迎，有更好地应用和发展。
从单元测试扩展到系统测试，文字识别变成图像识别来处理。不管是汽车电子还有其他的领域，可以把有限状态机或者是基于模型测试跟人工智能测试结合起来，测试效果会更好一些。在移动应用这方面的计算能力比较弱一些，可以用多个机器人去模拟用户的操作。在游戏行业，腾讯、华为、网易等公司在测试上应用AI能力方面有了良好应用成果，与高清摄像头结合，可以做到更多的场景识别、人工智能辅助决策，实时游戏控制等。（具体见演讲PPT）
AI技术助力于测试覆盖率
AI技术助力于测试覆盖率，可以借助AI提高功能测试覆盖率、源码测试覆盖率、回归测试覆盖率、系统非功能性测试覆盖率等。同时，也可以借助关键字引导的搜索策略以提高测试覆盖率。
回归测试利用AI，选择40%测试用例集可以达到80%的测试覆盖率。通过针对关键字索引提高测试覆盖率。例如：Eggplant AI导入已有测试资产创建模型、使用智能算法选择最佳测试集运行测试，基于模型算法能最大程度减少构建与维护的成本。Eggplant AI的测试覆盖率与以前相比，有很大提高，达到80%-90%。
通过遗传算法和对源码“编译时插桩”的方式自动生成测试用例，以探索二进制程序内部新的执行路径，而且能够不断优化执行路径，从而获得高效的模糊测试策略。这样可以大大提高源码测试覆盖率。
使用NLP进行E2E Web测试依赖项检测验证所有依赖关系，并使用一个新的恢复算法来确保在最终的测试依赖关系，提高回归测试覆盖率。
Peach Fuzzer模糊测试工具通过代理和状态机感知状态，实时调整变异策略，从而发现应用程序中更多的安全性漏洞，可以提高系统非功能性测试覆盖率。
朱少民老师认为未来的AI理想状态有两种，第一种是结合MBT与AI合力相助测试。第二种是探索式测试（ET）与AI结合， AI智能生成代码，ET给AI喂数据、加速AI模型的训练。（具体见演讲PPT）
如要下载PPT，关注《软件质量报道》公众号，输入“质效合一”获取。
其它参考：
