A little experience of CTS test
Introduction to CTS --------------------------------------------------------- 3
What is CTS------------------------------------------ 3
Test environment setup --------------------------------------------- 3
Test environment -------------------------------------------- 3
Preparation before the test ------------------------------------------------------ 3
Test phone configuration ---------------------------------------- 4
Execute the test ------------------------------------------------ 4
Overall test below 6.0 ------------------------------------ 4
Test related instructions ------------------------------------------------ 5
Mainly for 6.0 and below version test use------------------- 7
Tested for Android7.0 and later versions ------------------ 8
Test result output -------------------------------------------- 10
FAQ---------------------------------------------------10
Introduction to CTS
What is CTS
Introduction to CTS: Compatibility Test suite series compatibility test
google defines a compatibility specification (Compatible Definition), and CTS is to ensure that a test conforms to the specification. Thus Android -based applications can run on devices based on the same api version. Devices that pass the CTS test can get the android trademark and enjoy the permissions of the android market
There are three general CTS tests: Cts, Gts, Cts verify , the CTS test tool can be downloaded from the official website ( http://source.android.com/compatibility/downloads.html ), select the CDD document corresponding to the Android version (introducing the CTS test ), download the CTS test package (the test package will be updated continuously), unzip the downloaded CTS test package to the corresponding folder, which should contain: android-cts folder contains Docs, Repository, Tools three folders,
The test package of Android 6.0 is somewhat different from the CTS test package of Android 7.0, and the latest Media Files is CTS Media 1.2.
Test environment setup
test environment
· Linux system (preferably 64-bit ubuntu10.04)
· The Java JDK (version 1.7 and above), Android SDK and ADB need to be configured correctly to ensure that the system and mobile phone can be connected normally
Preparing for the test
· Download the software version that needs to be tested
· Connect to IPV6 over-the-wall network
· Insert an available SD card, preferably larger than 1G
· Insert an available Sim card, if it is a dual-card version, make sure that there is an available Sim card in the card slot of card 1.
· Write the legal IMEI number, you need to write the number
· Check whether the phone has set the CTS flag
Test phone configuration
· Reset the phone to factory settings (Settings→Backup andreset→Factory data reset)
· Make sure that there is enough space in the SD (more than 1G), you can format the SD to empty it.
· Make sure there is no Google account or other account in the phone
· Make sure the date and time are correct (the time will be automatically updated after the Wifi or data connection is turned on, the default time zone is selected), and set the Pacific time gtm: -8
· Set the screen to be always on (Settings→Display→Sleep→Never)
· Set up screenlock (Settings→Security→Set up screenlock→None)
· Reset the phone to factory settings (Settings→Backup andreset→Factory data reset)
· Before 2.1_R2 to 4.2_R4, install CtsDelegatingAccessibilityService.apk,
(under android-cts/repository/testcases, it can be installed with adb install), and go to Settings→Accessibility to enable Delegating Accessibility Service.
·
· Install CtsDeviceAdmin.apk, (available adbinstall CtsDeviceAdmin.apk under android-cts/repository/testcases), and go to Settings→Security→Deviceadministrators to open the two packages inside
·
· Copy the media file (the file decompressed from android-cts-media-1.2.zip) and the file named test to the external SD card of the mobile phone, which can be copied by script (execute in the android-cts-media-1.0 directory) bash copy_media.shall)
·
· Make sure that the three options of USB debugging, Stay awake, and Allowmock locations in the developer options are checked.
· Set the system language to American English (Settings→Language&input→Language)
· Set the system input method to Google's default input method
(Settings→Language&input→Default→English(US))
· Turn on Wifi, turn on data connection, turn on GPS
· Make sure the USB connection is stable
execute the test
Overall test
Before testing, the phone needs to install CtsDeviceAdmin.apk and enter in the terminal:
cts/android-cts/repository/testcases$adbinstall CtsDeviceAdmin.apk Check Android Devices Manage android.deviceadmin.cts.CtsDevicesAdminReceiver android.deviceadmin
under Settings-Security-Deviceadministrators . cts.CtsDevicesAdminReceiver2
l Use adb devices to check whether the phone is connected
n enter the test environment
l Enter the CTS test
Test related instructions
xxx@CTS-ceshi:~/cts/6.0_r14/android-cts$cd tools/
xxx@CTS-ceshi:~/cts/6.0_r14/android-cts/tools$./cts-tradefed
Android CTS 6.0_r14 build:3623550
03-17 16:46:58 I/DeviceManager: Detectednew device c23392e
l cts-tf > runcts --plan CTS --disable-reboot //Start to execute the entire test package of CTS
l cts-tf> help use help documentation
Run:
runcts --plan test_plan_name: run a test plan
runcts --package/-p : run a CTS test package
runcts --class/-c [--method/-m] : run a specific test class and/ormethod
run cts --continue-session session_ID: run allnot executed tests from a previous CTS session
runcts [options] --serial/-s device_ID: run CTS on specified device
runcts [options] --shards number_of_shards: shard a CTS run into given number ofindependent chunks, to run on multiple devices inparallel
runcts --help/--help-all: get more help on running CTS
List:
l/list d/devices: list connected devices and their state
l/list packages: list CTS test packages
l/list p/plans: list CTS test plans
l/list i/invocations: list invocations aka CTStest runs currentlyin progress
l/list c/commands: list commands: aka CTS test run commands currently inthe queue waiting to be allocated devices
l/list r/results: list CTS results currently present in the repository
Add:
addderivedplan --plan plane_name --session/-s session_id -r[pass/fail/notExecuted/timeout]: derive a plan from the given session
Dump:
d/dump l/logs: dump the tradefed logs for all running invocations
Options:
--disable-reboot: Do not reboot device after running some amount of tests.
l cts-tf > lr View failed projects Successful projects, unexecuted projects
Session Pass Fail Not Executed Start time Plan name Device serial(s)
0 2743 130 3555 2017.01.18_16.46.14 CTS1 24ade972
l cts-tf> run cts --continue-session 0 --disable-reboot
Continue unexecuted projects
cts-tf > l r
Session Pass Fail Not Executed Start time Plan name Device serial(s)
0 2743 130 3555 2017.01.18_16.46.14 CTS1 24ade972
l cts-tf> add derivedplan --plan testname -s 0 -r fail
l cts-tf> run cts --plan testname failed to execute the project
l Multiple machines execute at the same time
Android6.0 test requires IPV6 environment, you can use the command run cts –plan CTS –skip-preconditions
Android6.0 test takes a long time, google provides the function of testing multiple mobile phones together:
r un cts –plan CTS –shards mobile phone Quantity –skip-preconditions
For example, if I need 3 machines to test together, I can enter the command: run cts –plan CTS –shards 3 –skip-preconditions
In addition: enter help in the terminal to see the related commands of
cts run cts –plan test_plan_name Test a plan
run cts –packages /-p packages_name Test a package alone
run cts –class/-c class_name –method/-m method_name Test a class and method
such as: run cts -candroid.hardware.camera2.cts.BurstCaptureTest -m testYuvBurst
li command to view the running time cts-tf > li
lr to view the test result cts-tf > lr
It is mainly used for testing of 6.0 and below versions
CTS v1 console command reference
Table1. This table summarizes the CTS v1 console commands for varioususes.
Host |
Description |
help |
Display a summary of the most commonly used commands |
help all |
Display the complete list of available commands |
exit |
Gracefully exit the CTS console. Console will close when all currently running tests are finished |
Run |
Description |
run cts |
Run the specified tests and displays progress information. One of --plan, --package, --class or --continue-session needs to be specified The CTS console can accept other commands while tests are in progress If no devices are connected, the CTS desktop machine (or host) will wait for a device to be connected before starting tests If more than one device is connected, the CTS host will choose a device automatically |
--plan <test_plan_name> |
Run the specified test plan |
--package/-p <test_package_name> [--package/-p <test_package2>...] |
Run the specified test packages |
--class/-c <class_name> [--method/-m <test_method_name> |
Run the specified test class and/or method |
--continue-session |
Run all not executed tests from previous CTS session; the sessions testResult.xml will be updated with the new results |
--shards <number_of_shards> |
Shard a CTS run into given number of independent chunks, to run on multiple devices in parallel |
--serial/-s <deviceID> |
Run CTS on the specific device |
-t <class_name>#<test_method_name> |
Run a specific test method |
--force-abi 32|64 |
On 64-bit devices, run the test against only the 32-bit or 64-bit ABI |
List |
Description |
list packages |
List all available test packages in the repository |
list plans |
List all available test plans in the repository |
list invocations |
List 'run' commands currently being executed on devices |
list commands |
List all 'run' commands currently in the queue waiting to be assigned to devices |
list results |
List CTS results currently stored in repository |
list devices |
List currently connected devices and their state 'Available' devices are functioning, idle devices, available for running tests 'Unavailable' devices are devices visible via adb, but are not responding to adb commands and won't be allocated for tests 'Allocated' devices are devices currently running tests |
Add |
Description |
add derivedplan --plan <plan_name> |
Create a plan derived from given result session; use this option to rerun reports and validate test issues |
针对Android7.0 以及以后版本测试
For Android 7.0 or later, you'll use CTS v2.
CTS v2 console command reference
Table1. This table summarizes the CTS V2 console commands for varioususes.
Host |
Description |
help |
Display a summary of the most commonly used commands |
help all |
Display the complete list of available commands |
version |
Show the version. |
exit |
Gracefully exit the CTS console. Console will close when all currently running tests are finished. |
Run |
Description |
run cts |
Run the default CTS plan (that is, the full CTS invocation). The CTS console can accept other commands while tests are in progress. If no devices are connected, the CTS desktop machine (or host) will wait for a device to be connected before starting tests. If more than one device is connected, the CTS host will choose a device automatically. |
--plan <test_plan_name> |
Run the specified test plan. |
--module/-m <test_module_name> [--module/-m <test_module2>...] |
Run the specified test module or modules. For example, run cts --module CtsGestureTestCases executes the gesture test module (this can be shortened to run cts -m Gesture). |
--module/-m <test_module_name> -- test <test_name> |
Run the specified module and test. For example, run cts -m Gesture --test android.gesture.cts. GestureTest#testGetStrokes runs the specific package, class, or test. |
--retry |
Retry all tests that failed or were not executed from the previous sessions. Use list results to get the session id. |
--shards <number_of_shards> |
Shard a CTS run into given number of independent chunks, to run on multiple devices in parallel. |
--serial/-s <deviceID> |
Run CTS on the specific device. |
--include-filter <module_name> [--include-filter <module2>...] |
Run only with the specified modules. |
--exclude-filter <module_name> [--exclude-filter <module2>...] |
Exclude the specified modules from the run. |
--log-level-display/-l <log_level> |
Run with the minimum specified log level displayed to STDOUT. Valid values: [VERBOSE, DEBUG, INFO, WARN, ERROR, ASSERT]. |
--abi <abi_name> |
Force the test to run on the given ABI, 32 or 64. By default CTS runs a test once for each ABI the device supports. |
--logcat, --bugreport, and --screenshoot-on-failure |
Give more visibility into failures and can help with diagnostics. |
--device-token |
Specifies a given device has the given token eg. --device-token 1a2b3c4d:sim-card.. |
--skip-device-info |
Skips collection of information about the device. Note: do not use this option when running CTS for approval. |
--skip-preconditions |
Bypasses verification and setup of the device's configuration, such as pushing media files or checking for Wi-Fi connection. |
List |
Description |
list modules |
List all available test modules in the repository. |
list plansor list configs |
List all available test plans (configs) in the repository. |
list invocations |
List 'run' commands currently being executed on devices. |
list commands |
List all 'run' commands currently in the queue waiting to be assigned to devices. |
list results |
List CTS results currently stored in repository. |
list devices |
List currently connected devices and their state. 'Available' devices are functioning, idle devices, available for running tests. 'Unavailable' devices are devices visible via adb, but are not responding to adb commands and won't be allocated for tests. 'Allocated' devices are devices currently running tests. |
Dump |
Description |
dump logs |
Dump the tradefed logs for all running invocations. |
ADD |
Description ps: |
a/add s/subplan |
Create a subplan used to create a subplan |
--session <session_id> |
The session used to create a subplan |
--result-type <status> |
Which result to include in the subplan |
--name/-n <subplan_name> |
The name of the new subplan |
Cts-tf> run cts --retry sessionId --retry-type FAILED/NOT_EXECUTED 执行未完成的任务 ，输出结果
|
|
测试结果输出
l 测试结束后 /repository/results/2016.05.27_21.29.17/testResult.xml使用浏览器打开，查看测试的结果
可以进入android-cts/repository/results/2016.05.27_21.29.17输入命令： sudo gedit testResult.xml 在这里必须使用命令打开，否则不能保存，可以看到测试结果：
最近因为测试 的需要,接触到了Android cts测试 ,以上是最近整理的一点东西 ,如果有不对的地方,也请看到的大神指点一二,谢谢 !
FAQ
l 修改了如下内容 ，正常测试没有问题，可不需要修改
增加了环境变量 ，使用的是6.0的aapt ，这是要测试的相应的版本的aapt
1 。root@CTS-ceshi:~# gedit .bashrc
exportANDROID_HOME=/home/liangcheng/android-sdk-linux #/*SDK所在的路径*/
exportPATH=$PATH:/home/liangcheng/android-sdk-linux/build-tools/23.0.3
l 手机在linux上可以通过adb devices 看到 ，但是不弹出连接时间的对话框
需要查看 develop options 中 Select usb Configuration 是否是在MTP模式上