Android Security Test Suite (STS) is a test suite from Google about the installation of Android security patches. STS is related to security patches. It is a new security test suite added to the CTS test. STS only started testing in 201808. It is a case of GTS that the security patch date is within 3 months. If it fails, you cannot obtain Google certification. Before May 2018, Google's security patch attribute was updated with aosp under the build library. Now it needs to be updated by the vendor after passing STS. STS needs to generate a user version of the result through the userdebug version of the targetfile.
STS test prerequisites
- The device must be flashed with userdebug rom
- USB debugging has been turned on, USB installation and click functions need to be normal
- Root can be obtained, and adb root must be executed successfully.
STS tool download by yourself
STS test command
sts-tradefed run sts-dynamic-incremental -m <module name> -t <single item name> MR version test command
sts-tradefed run sts-dynamic-full -m <module name> -t <single item name> IR version test command
Default execution:
sts-tradefed run sts-dynamic-full
Specify target device:
sts-tradefed run sts-dynamic-full -s [Specify your android device]
------------------
Differences between CTS, GSI, GTS, VTS and STS
the difference:
1. Burning img versions are different
CTS,GSI,GTS,VTS burn user version img
STS burn user-debug version img
2. Do you need to burn additional Google img?
VTS needs to push Google img and boot-debug.img additionally
GSI needs to push additional Google img
Not required for other tests
3. Is it necessary to push the media package before testing?
CTS, GSI, GTS need to push media package
Not required for the rest of the tests
//Method to push media package: execute script + device SN number in Media
4. The test instructions are slightly different, you only need to modify the corresponding test name.
//Note: CTS test and GSI test share a test script, so the GSI test can be performed under the CTS test, but not at the same time (that is, the two tests cannot share the same test tool at the same time)
4.1.CTS test (estimated 170 hours for a single test)
Full test command: run cts --shard-count 3 -s CTS00000001 -s CTS00000002 -s CTS00000003 // -s [SN number]
Retest command: run retry -r 1 --shard-count 3 -s CTS00000001 -s CTS00000002 -s CTS00000003 / -r [Number of test reports] Single-
sided module: run cts -m CtsPermission2TestCases -s CTS00000001 // -m [Test module]
Single-sided case: run cts -m CtsPermission2TestCases -t android.permission2.cts.PermissionPolicyTest#platformPermissionPolicyIsUnaltered -s CTS00000001 // -t [Test item]
4.2.GTS test (single test is expected to take 20 hours)
full test command: run gts --shard-count 3 -s GTS00000001 -s GTS00000002 -s GTS00000003
Retest command: run retry -r 1 --shard-count 3 -s GTS00000001 -s GTS00000002 -s GTS00000003
4.3.VTS test (single test is expected to take 45 hours)
Note: burning is required Google System.img
Full test command: run vts --shard-count 3 -s VTS00000001 -s VTS00000002 -s VTS00000003
Retest command: run retry -r 1 --shard-count 3 -s VTS00000001 -s VTS00000002 -s VTS00000003
4.4.GSI test (single test is expected to take 35 hours) --> Shared Tool with CTS
Note: Google System.img needs to be burned.
Full test command: run cts-on-gsi --shard-count 3 -s GSI00000001 -s GSI00000002 -s GSI00000003
retest command: run retry -r 1 --shard-count 3 -s GSI00000001 -s GSI00000002 -s GSI00000003
4.5.STS test (single test is expected to take 7 hours)
Note: You need to burn the Userdebug version to test
the full test command : run sts-dynamic-full --shard-count 3 -s STS00000001 -s STS00000002 -s STS00000003
Retest command: run retry -r 1 --shard-count 3 -s GSI00000001 -s GSI00000002 -s GSI00000003
4.6.CTS_V test-->DQA is responsible for testing
#Other test instructions
l i
ld
l r
lc
--------------------------------
1. Instructions
basic:
| kit | instruction | test version | update frequency | download link |
|---|---|---|---|---|
| cts | run cts | release | quarter | https://source.android.com/docs/compatibility/cts/downloads |
| gts | run gts ; run gts-interactive | release | quarter | https://docs.partner.android.com/gms/testing/gtsRequires login |
| vts | run vts | release | quarter | https://docs.partner.android.com/gms/testing/vtsRequired to log in |
| cts-on-gsi | run cts-on-gsi | release | CTS suite: quarterly; system.img: monthly | Same as CTS |
| sts | run sts-dynamic-full | debug | per month | https://drive.google.com/drive/folders/1xqPTtC6MWiQizfFVdG7Ho0f2oGsmH0e- |
| tvts | run tvts | release | https://docs.partner.android.com/tv/test/tvts/release-notes |
worse:
| Function | instruction | Remark |
|---|---|---|
| Retry multiple machines only 64-bit and not including 3 packages | run retry --retry 1 --shard-count 2 --abi arm64-v8a --exclude-filter CtsLibcoreTestCases --exclude-filter CtsNetTestCases --exclude-filter CtsAppSecurityHostTestCases | These 3 packages involve IPV6 |
| Test CTS subplan | run cts --subplan | |
| help | help | |
| cts help | run cts --help; run cts --help -all |
monthly:
| month | CTS (month of use) | GTS (month of use) | VTS (month of use) | STS (security patch month) | img (security patch month) |
|---|---|---|---|---|---|
| Release month | 3 6 9 12 | 3 6 9 12 | 3 6 9 12 | last month | |
| 2023-10 | android-cts-13_r5-linux_x86-arm | android-gts-11-R1(11-14)-10495246 | android-vts-10477834_arm64(13_R5) | android-sts-13_sts-r19-linux-arm64 | android13-arm64-img-10694250.zip |
| 2023-11 | android-sts-13_sts-r20-linux-arm64 | signed-gsi_arm64-img-10828306.zip | |||
| 2023-12 | android-sts-13_sts-r21-linux-arm64 | signed-gsi_arm64-img-10953849.zip | |||
| 2024-01 | android-cts-13_r6-linux_x86-arm | android-gts-11-R2(11-14)-11137706.zip | android-vts-11183460_arm64(13_R6) | android-sts-13_sts-r22-linux-arm64 | signed-gsi_arm64-img-11114083.zip |
Websites used often:
| Function | URL | Remark |
|---|---|---|
| https://source.android.com/docs/ | Intranet | |
| gms of partner | https://docs.partner.android.com/gms/testing/overview | External network + authorized account login |
| partner's security | https://docs.partner.android.com/security | External network + authorized account login |
| GTS/VTS/system.img latest version download | https://drive.google.com/drive/folders/0ByUA0TZpIQ_UYll6QnFxaVVNVTA?resourcekey=0-OaeHWcxibqcFpnZw_OzYiw | External network + authorized account login |
| TV system.img latest version download | https://docs.partner.android.com/tv/test/android/gsi?hl=en | External network + authorized account login |
| SPL | https://drive.google.com/drive/folders/0B85mEDAGzAbsckRrZFhhV3YwTEk?resourcekey=0-6zNsbY0nWUmYsJO2TF-RZQ | External network + authorized account login |
| what | https://partner.android.com/approvals | External network + authorized account login |
3. Image file
adb shell getprop | grep patchThe information obtained through the command looks at the date of the security patch.
[ro.build.version.security_patch]: [2023-10-05] # 此为system.img的
[ro.vendor.build.security_patch]: [2023-10-05] # 此为公司版本的
- When using ; between commands, the next command will be run after the previous command is completed (regardless of whether it is successful or not).
- When && is used between commands, the next command will be run after the previous command is completed (when successful)
CTS-on-GSI one-click image replacement
Execute the following instructions in the directory where system.img is located:adb reboot bootloader&&fastboot flashing unlock&&fastboot flashing unlock_critical&&fastboot reboot fastboot&&fastboot flash system system.img&&fastboot -w&&fastboot reboot
VTS one-click image replacement
Execute the following instructions in the directory where system.img/vendor_boot.img is located:adb reboot bootloader&&fastboot flashing unlock&&fastboot flashing unlock_critical&&fastboot reboot fastboot&&fastboot flash system system.img&&fastboot flash vendor_boot vendor_boot-debug.img&&fastboot -w&&fastboot reboot
# CTS-on-GSI
adb reboot bootloader
fastboot flashing unlock
fastboot flashing unlock_critical
fastboot reboot fastboot
fastboot flash system system.img
fastboot -w
fastboot reboot
# VTS
adb reboot bootloader
fastboot flashing unlock
fastboot flashing unlock_critical
fastboot reboot fastboot
fastboot flash system system.img
fastboot flash vendor_boot vendor_boot-debug.img
fastboot -w
fastboot reboot
4. Daily build version
Daily build address android 13:
https://ci.android.com/builds/branches/aosp-android13-tests-release/grid?legacy=1
1) Download CTS, VTS, CTS-VERIFIER search fields, such as android12
aosp-android12-tests-dev
aosp-android12-tests-release
2) Download TVTS search field
git_master-tv-dev
3) Download the google system image search field
git_tm-gsi-release
git_sc-tv-gsi-release
git_rvc-tv-gsi-release
5. Use of sub-plans
Why use subplans?
Taking the CTS test as an example, there are 1058 test modules, and it will take 1 week to complete all tests. If you only need to verify certain modules, such as media-related modules. When there is no subplan, a module needs to enter an instruction. Using a subplan can put all media modules in a test plan, and running one instruction can implement all media module tests. help add You can view the specific usage of sub-plans.
Taking the 13_r4 suite of Android 13 as an example, its CTS has 1055 modules. When the prototype is sufficient, it can be divided into multiple units to improve testing efficiency.
| Test items | Number of modules | time consuming |
|---|---|---|
| CTS | 1055 | When there are enough prototypes, you can use a sub-plan and divide the test into three parts to improve the test efficiency. P1: DEQP1 unit: 30h; P2: Media1 unit: 24h, and then retry multiple times; P3: Other modules: 2 units: 36h, and then more times retry; |
| CTS-on-GSI | 347 | 1 unit: 36h, then retry multiple times |
| CTS-Verifier | Manual testing for 3 days | |
| GTS | 273 | 1 unit: 24h |
| STS | 20 | 1 unit: 6h |
| VTS | 643 | 1 unit: 6h |
6. Prototype side operation
Enter developer mode: settings->About->build num (tap 5 times) ->open Developer options
Set the device to stay on: Developer options->Stay awake->on
Turn on USB debugging: Developer options->USB debugging->on
Turn off the USB installation application prompt: Developer options->Verify apps over USB->off
7. Computer operation
1 Media related tests need to place the media files in the /tmp/android-cts-media path of the computer. There is no need to copy the Media files to the phone before testing. They will be copied automatically during the test.
If there is no media file in the /tmp/android-cts-media path, it will be downloaded from the Internet. Since the file is relatively large, it will be more time-consuming.
rkp attestattion key
Get rkp json file
In factory mode, the version information is on the left and right, and a prompt message pops up; in another menu, right-click, and it will be automatically saved to the USB flash drive.
way1:需要登陆网页输入账号
python3.9 ./device_info_uploader.py --credentials ./cred.json --json-csr csrs.json --cache-token --company-id 25958473112589
way2:可以直接使用
python3.9 ./device_info_uploader.py --credentials-keyfile ./-rkp-cd-service-account.json --json-csr csrs.json --company-id 25958473112589
8. Test analysis
Android13
test cycle (including verifier): 7 days
Pre-test cycle (excluding verifier): 4 days
Number of machines: 7
| Test items | time cost | Number of machines (units) |
|---|---|---|
| CTS-without-deqp-media | 3d | 2 units |
| CTS-without-deqp-media-64 | 2d | 2 units |
| CTS-deqp | 30h | 1 set |
| CTS-deqp-64 | 15h | 1 set |
| CTS-media | 3d | 1 set |
| CTS-media-64 | 2d | 1 set |
| gts | 2d | 1 set |
| cts-on-gsi | 3d | 1 set |
| vts | 6h | 1 set |
| sts | 6h | 1 set |
| cts-verifier | 4d | 1 set |
This article refers to:
https://www.cnblogs.com/qev211/p/17570911.html
https://www.cnblogs.com/a-n-yan/p/16043606.html
Getting started with Android S XTS testing_android xts-CSDN blog