CTS whole process Automation Plan
Vision function
cts whole process automation. Generally includes the following modules:
- Version construction
- Forced OTA upgrade
- Automatic retry
- + Log backup and analytical reports
- Send e-mail
Scheme pain points
| aims | Program | Issue rating & pain points |
|---|---|---|
| Version construction | Current program: python library paramiko remote ssh login details: win end paramiko python script library ssh remote login Linux compiling server, a file is written to compile a python command, and then execute the "Compile a key command" py file |
Problem: Low version sometimes not been downloaded, the need for manual intervention. Automated build is not a problem, something daily has been achieved. |
| Forced OTA upgrade | Current program: the program needs to study and discuss on how to change the configuration In addition multiple set-top box upgrade is not realized in prior program: 1. Compile the server OTA package copy to the Apache server path win side (ready escalation rules file and configuration, upgrade configuration is to avoid the upgrade cycle) 2.adb the install apk change the configuration modifications versionserver, mandatory upgrade restart (upgrade configuration simultaneously) disposed aidl after separation, is achieved jni client, the completion of a form apk accepts parameters-Service, so can am satrtservice write configuration. But now is no longer available, specific reasons to be discussed (Android9 prohibit the background to start service) |
Problem: High can not adb command to modify the configuration, the module is approximately equal to abolish the program guess: am satrt activity and -es mass participation? In addition, OTA upgrade data filled in selinux: How do I clear (involving cts copy media files) under enable? Guide key (no need to burn once again burn?) OTA upgrade and USB burn, version difference is? |
| Automatic retry | Current program: shell script calls gnome terminal to achieve pseudo child process to start a new retry comes from cts-tf> lr command the result listener why the use of the current program? : It is due to try: 1.python child process: start the cts-tradefed script, start the test, you can not add --shard-count parameter 2.shell direct ./cts-tradefed xxx calling script open test, you can run the test, but you can not quit. If you run too cts test you will find this script has been in a state of waiting for input TF-cts> 3. In addition the more important point, more than two standard output can not be redirected, so the result listener currently used ./cts-tradefed lr means, guess there are other means, please discuss |
Problem: High in general I set retry5 times, so open seventy-eight terminal testing machines began to appear obvious Caton. In addition, this retry program can not (run the script does not generate new gnome terminal), for an unknown reason by ssh remote call. I believe this is the most low of retry means, please discuss additional rule out human intervention if there is a significant problem (how to determine which is what we want and bring it to report robust developer analysis), focusing on the next module. |
| + Log backup and analytical reports | 目前方案： 1.将Linux测试机的结果copy到备份服务器 2.requests和bs4解析报告html，并进行失败项过滤分类，生成Markdown格式HTML（为了同步ICenter以及比文本好看一丢丢？） |
问题等级：低 虽然全自动化的理想很美好，但是不得不面对的现实是： 版本早期，失败项少则几十多则五六百，脚本如何确定什么状态的报告是能发出来的，这个需要重点讨论。根据经验，开发人员需要的用来分析的不应是环境引起的失败（例如网络断了，没导key，没插电视，机顶盒adb掉线，系统挂了不响应了导致大批量失败），为什么呢？因为这种状况应该由测试人员（工具）做保证，调整正确物理环境排查过滤掉，而不是反复反复报出来由开发人员再过滤一遍，综上一句话：测试工具如何保证测试结果的稳健性，纯净性！！！尤其是早期系统不稳定故障多的时候比较难不依赖人工干预，还是需要测试人员去监测运行状况的 |
| 邮件发送 | 目前方案： 目前没做，将‘简单分类’复制粘贴ICenter，写测试总结，失败项分析任务安排,然后转发邮件 为什么不自动发邮件？ A:还是上面的问题，我们无法确认报告是我们想要的报告，自动发出的垃圾邮件垃圾数据无分析必要的话，反而会增加沟通成本。 测试人员会频繁问，这结果能证明是系统问题？我怎么发现好多环境引起的问题，你确定这是正常稳健的测试结果吗？于是测试人员不得不再次回头确认测试结果，再次retry，获得与经验相符的稳健结果 |
问题等级：低 个人建议，邮件自动转发是不必要的，因为测试完需要测试人员根据经验来过滤并指定责任人来安排分析任务。因为失败项是全领域的，而开发人员是分领域的，不安排的话，各领域面对一堆数据，会比较懵，不知道自己该分析哪个 |
备注
一些补充
目前的架构是
win端为控制台，win端需要将Linux编译服务器，Ubuntu测试机挂载为网络驱动器
Ubuntu测试机需要双网口，同时接10与192外网。
机顶盒有线接10，无线接192外网
另一种考虑：
能不能完全屏蔽掉win端，所有模块都搬到Ubuntu测试机上去，
Ubuntu上远程编译服务器编译版本
Ubuntu上配置好Apache服务器给机顶盒强制升级
Ubuntu上直接开启测试，retry
Ubuntu上解析结果，服务器备份结果
然后人工获取服务器备份发邮件进行分析任务安排。
以上相较于目前方案优点在于，免了win端远程Ubuntu执行命令，执行结果标准输出&日志打印回显win端是件麻烦事不回显德华作为控制台的win端就显得鸡肋。