When we write automated test scripts, Traditionally you must know the properties of the elements, such as id, name, class and so on. By way of AI then positioning elements may not need to know the properties of elements, the elements of the judgment of the evaluator to locate, for example, to see a search box directly ai:searchto locate, or I would like to locate a close button, the direct use ai:close, we do not need to know the properties of this search box and the Close button.
appium can support AI positioned by means of plug-ins. This positioning is a way to experience it took me two weeks.
System Requirements
First, we need some of the system dependencies to deal with images.
- macOS
brew install pkg-config cairo pango libpng jpeg giflib- Linux
sudo apt-get install pkg-config libcairo2-dev libpango* libpng-dev libjpeg-dev giflib*- Windows
Not supported.
If you experience problems, you may have to install each package individually.
I started on Windows test for a long time unsuccessful, because I have a ready-made appium environment, later found simply does not support, then, replaced macOS, so the back of the operation is completed at macOS, of course, if you have a Linux environment, I that is OK.
Android Studio installed
Because the mobile device I want to operate on an Android, you need to install the Android SDK, so Android Studio integrates the Android SDK.
Studio Download Android: https://developer.android.com/studio
During the installation of Android Studio in android SDK you need to set the path, my path is:
/Users/tech/Library/Android/sdkThen, you need to configure the environment variables:sudo vi ~/.bash_profile
ANDROID_HOME=/Users/tech/Library/Android/sdk
PATH=${PATH}:${ANDROID_HOME}/platform-tools
PATH=${PATH}:${ANDROID_HOME}/toolsFinally, to validate the configuration:source ~/.bash_profile
Installation appium
1, do not use appium-desktop, installation appium command mode.
> brew install node # get node.js
> npm install -g appium # get appium2, mounting appium-doctor
> npm install appium-doctor3, by appium-doctorchecking the command environment:
> appium-doctor
info AppiumDoctor Appium Doctor v.1.11.0
info AppiumDoctor ### Diagnostic for necessary dependencies starting ###
info AppiumDoctor ✔ The Node.js binary was found at: /usr/local/bin/node
info AppiumDoctor ✔ Node version is 10.15.1
WARN AppiumDoctor ✖ Xcode is NOT installed!
info AppiumDoctor ✔ Xcode Command Line Tools are installed in: /Library/Developer/CommandLineTools
info AppiumDoctor ✔ DevToolsSecurity is enabled.
info AppiumDoctor ✔ The Authorization DB is set up properly.
WARN AppiumDoctor ✖ Carthage was NOT found!
info AppiumDoctor ✔ HOME is set to: /Users/tech
info AppiumDoctor ✔ ANDROID_HOME is set to: /Users/tech/Library/Android/sdk
info AppiumDoctor ✔ JAVA_HOME is set to: /Library/Java/JavaVirtualMachines/jdk-9.0.1.jdk/Contents/Home
info AppiumDoctor ✔ adb exists at: /Users/tech/Library/Android/sdk/platform-tools/adb
info AppiumDoctor ✔ android exists at: /Users/tech/Library/Android/sdk/tools/android
info AppiumDoctor ✔ emulator exists at: /Users/tech/Library/Android/sdk/tools/emulator
info AppiumDoctor ✔ Bin directory of $JAVA_HOME is set
...appium AI plug-in
GtiHub Address: https://github.com/testdotai/appium-classifier-plugin
Use Appium 1.9.2-beta version of the above. Also, be sure to use XCUITest driver (for iOS) or UiAutomator2 or Espresso driver (for Android). IOS and Android older drivers do not support the required Appium in any case, it is not recommended.
Classifier settings
To make this plugin available for Appium, simply go to the installation directory of the main appium items below, and run:
> cd /usr/local/lib/node_modules/appium
> npm install test-ai-classifierThis plug mounted to Appium dependency tree and make it available.
I have all kinds of error when installing this plugin, you can try the following command.
> sudo npm --registry http://registry.npm.taobao.org install test-ai-classifier --unsafe-permuse
1, by way of command to start appium
> appium
[Appium] Welcome to Appium v1.14.0
[Appium] Appium REST http interface listener started on 0.0.0.0:4723
...2, writing automated test scripts:
from appium import webdriver
from time import sleep
CAPS = {
"deviceName": " MEIZU_E3",
"automationName": "UiAutomator2",
"platformName": "Android",
"platformVersion": "7.1.1",
"appPackage": " com.meizu.flyme.flymebbs",
"appActivity": ".ui.LoadingActivity",
"noReset": True,
"unicodeKeyboard": True,
"resetKeyboard": True,
"customFindModules": {"ai": "test-ai-classifier"},
"testaiConfidenceThreshold": 0.1,
"shouldUseCompactResponses": False,
}
driver = webdriver.Remote('http://localhost:4723/wd/hub', CAPS)
sleep(3)
# 用 AI 定位到搜索框
driver.find_element_by_custom("ai:search").click()
sleep(5)
driver.find_element_by_id("com.meizu.flyme.flymebbs:id/kf").send_keys("flyme")
driver.find_element_by_id("com.meizu.flyme.flymebbs:id/o7").click()
result = driver.find_elements_by_id("com.meizu.flyme.flymebbs:id/a2a")[0].text
print(result)
driver.quit()automationName
If you want to test the Android, it is necessary to refer toUiAutomator2orEspresso.customFindModules
must be specified as{"ai": "test-ai-classifier"}.testaiConfidenceThreshold
This function determines consider the elements of the lowest confidence level. By default, the value of 0.2. Parameter number between 0 and 1, where 1 represents the confidence to be perfect, 0 means do not need confidence.shouldUseCompactResponses
This indicates appium contains additional information about the elements found in the elements, which greatly accelerated the process to obtain this plug-in input.
Ultimately, this line of code that I want to experience:
driver.find_element_by_custom("ai:search").click()To locate the search box by ai.
Indeed to locate, but the positioning speed extremely slow, need about 10 to 20 seconds.
If you want to know appium-classifier-pluginsupport those types of positioning elements, see here: https://github.com/testdotai/appium-classifier-plugin/blob/master/lib/labels.js
Currently supports more than 100 kinds of types.