background
At present, the corresponding framework has been built on the Android side, and it has actually been produced. Since Android uses Unittest+HtmlTestRunner to generate reports, if new functions need to be added, it needs to be changed to the underlying framework. Therefore, the responsible iOS side is currently planning to use The method of Pytest+Allure has the advantage of better plug-in support, and the report will look better (pretentious)
PS: Android automation is indeed much more stable than iOS automation
An introduction to iOS automation framework and tools
1. UIAutomation
UIAutomation is a UI automation testing framework provided by Apple, written in JavaScript.
Based on UIAutomation, there are extended tool frameworks and driven frameworks. The extended framework provides many useful js tools in the way of JavaScript extension library, and the injection framework usually provides some Lib or Framework, requiring testers to import these contents in the code project of the application under test, and the framework can complete the comparison through them. app driver.
Driven UI Automation uses the UI Automation library at the bottom of automated testing, and drives UI Automation through TCP communication to complete automated testing. In this way, the language for editing scripts is no longer limited to JavaScript. This tool is widely used in iOS UI automation testing
2.XCTest
XCTest is a simple and powerful test framework introduced by Apple in iOS 7 and Xcode5, which is integrated in Xcode and used to write test code. It provides various levels of testing.
XCTest tests are very simple to write and follow the xUnit style.
When Xcode creates a project, it uses XCTest by default, and creates two Targets, Unit Test (unit test) and UI Test (interface test) by default. Unit Test is mainly used to test most of the basic functions of the code, such as most Model class and method testing, business logic testing, network interface calling testing, etc.
UI Test generally considers the user's interaction process, simulates the user's interactive operation, uses the UI recording feature of XCTest to obtain a series of view elements and operation events on the interface, and then triggers the event in the test method.
So this is a framework that can provide various levels of testing, such as unit testing, automated testing, performance testing, etc.
XCTest reference materials
3. KIF
KIF is the abbreviation of the Keep It Functional project. It is an iOS app functional testing framework from Square. The testing framework only supports iOS.
All tests are written in Objective-C language. For testers, it is necessary to master Objective-C language proficiently. It is very easy for Apple developers to use, and it is a testing tool widely recommended by developers.
KIF uses unpublished Apple API (private API), which is safe for testing purposes. Based on the third-party iOS UI unit testing framework, it can be used for unit testing of projects and UI integration testing. But the disadvantage is that it runs slower.
4.Frank
Frank is a tool for automatic testing in the iOS development environment. After the development is completed in the Xcode environment, structured test cases can be realized through Frank. The underlying language is Ruby. As an open source iOS testing tool, it has been widely used in foreign countries. Applications.
However, there are relatively few relevant domestic data. Its biggest advantage is that it allows us to implement the actual operation logic in familiar natural language.
It provides functional testing capabilities for the iOS platform, which can simulate user operations for black-box testing of applications, and use Cucumber to write test cases, so that the test uses, for example, the same as natural language to describe functional requirements, so that the test can be "executable document ” to become a bridge between business customers and delivery teams.
- advantage
Test scenarios are written in understandable English sentences with the help of Cucumber, with active community support and a growing library.
- shortcoming
There is limited support for gestures, so it's a bit hard to run tests on the device.
5. Calabash-iOS
Calabash is a cross-platform app testing framework for iOS and Android developers, which can be used to test screenshots, gestures and actual functional code.
Calabash is open source and free and supports the Cucumber language. Cucumber allows you to express the behavior of the app in natural English language and implement BDD (Behavior Driven Development, Behavior Driven Development).
Calabash-iOS is a functional and automated testing framework for iOS based on Calabash.
- advantage:
There is a large community support;
the list items are simple, and the English-like test sentences support all actions on the screen, such as sliding, zooming, rotating, tapping, etc.
- shortcoming:
After a test step fails, all subsequent steps are skipped, which can cause more serious product issues to be missed.
The test is time-consuming, because it always installs the app first by default, and requires the Calabash framework to be installed in the iOS ipa
file, so the tester must have the iOS app source code.
Not friendly to languages other than Ruby
6. Appium
Appium is an open source, cross-platform automated testing tool that supports iOS, Android and FirefoxOS platforms.
With Appium, developers can test mobile applications without recompiling the app or making any adjustments, allowing the test code to access back-end APIs and databases.
It is supported by the iOS platform by driving Apple's UIAutomation framework, and the bottom layer uses WEBDriverAgent to drive
Developers can write test scripts in any language compatible with WebDriver, such as Ruby, C#, Java, JS, Objective-C, PHP, Python, Perl and Clojure.
7. AirTest
Airtest is a UI automation testing solution launched by Netease Games. It is a cross-platform UI automation testing framework based on image recognition, which is suitable for games and apps. The supported platforms are Windows, Android and iOS. It also provides a Poco framework based on UI control recognition, and currently supports platforms such as Android native, iOS native, Unity3D, cocos2dx, UE4, and Egret. In order to make it easier for testers to get started, Netease also considerately provides the AirtestIDE tool, which has built-in related plug-in functions for Airtest and Poco, and can use it to quickly and easily write Airtest and Poco codes
Compared with Appium, the biggest highlight of POCOUI is that it integrates self-developed image recognition for clicking, and the grammar of code writing will be more refined.
PocoUi reference materials
Combining the above 7 solutions, if you can use Object-C to write XCTest use cases, it is actually the best choice, but because I am not familiar with Object-C, learning a language only for iOS automation is relatively costly, so comprehensive consideration Anyone who decides to make a comparison between Appium and Airtest
should be familiar with Appium for automated testing. As an open source project of Netease, Airtest is also very good, so let’s look at the difference between the two
Two Appium and Airtest's iOS automation test comparison
| Comparison item | Appius | Airtest |
|---|---|---|
| installation configuration | Medium: Need some pre-dependent environment and fill in the startup parameters | Simple IDE can be directly connected |
| language support | Covers most major languages | Python-based |
| Easy to learn | medium | easy |
| market share | TOP1 | gradually increase |
| support platform | Android，iOS，H5 | Android，iOS，H5 |
| Integration framework | UiAutomator, UiAutomation framework | Poco element recognition, Airtest image recognition |
| Multiple devices in parallel | support | support |
| Relevant information | many | less |
| Community Support and Maintenance | There is a dedicated community to support and maintain, and there are many exchanges and discussions | There are Q groups for communication |
| Subsequent upgrades | guaranteed | Depends on the follow-up maintenance of the Netease team |
| testing report | HtmlTestRunner、Allure | Comes with its own report template |
| CI continuous integration | support | support |
| element positioning | Some pages cannot get elements | I personally feel that element positioning is not easy to use |
| effectiveness | Text positioning is faster, rest positioning is slower | Text positioning is faster, rest positioning is slower |
| stability | Generally, mostly depends on WDA | Generally, mostly depends on WDA |
| writing efficiency | Depends on familiarity with Appium-Api | Depends on Api familiarity with Airtest and PocoUI |
| Applicable scene | Suitable for self-developed and native App applications | Mostly used to quickly build graphics scripts and game automation |
Comparison conclusion:
1. In terms of execution efficiency, there is not much difference between Appium and Airtest . The main method is to locate elements through text. It is slower to locate elements without text.
2. In terms of code writing efficiency, I use Appium better than Airtest (due to the Android project). Airtest’s Api is new and different from appium’s. Familiarity with Api requires a certain process. It may be slower in the early stage, and later It will be faster if you are proficient, Airtest will package more APIs, and initial users need to package Appium's APIs by themselves
3. Regarding stability, both Appium-wda and iOS-tagent are packaged and transformed based on facebook-wda (facebook has not been maintained, and the appium team is maintaining it, and airtest is currently undergoing transformation and maintenance). The stability is about the same, and there has been no disconnection for the time being. iOS-tagent is said to simplify some internal logic calls, but the overall experience is similar. For higher versions of iOS, Airtest seems to recommend Appium’s wda
4. Regarding applicable scenarios, Appium mainly uses element positioning to automate operations. It is suitable for native apps without special emphasis on a certain type of app. It is a scene with strong compatibility . Airtest is backed by Netease, so in terms of scene use, it is more automated by game companies, especially its image recognition click, which can better recognize the scene in game automation, but Airtest can also click on elements through PocoUI, involving many graphics, such as Games can use Airtest
5. For the positioning of elements in specific scenarios, Appium and Airtest cannot obtain the element information in some apps, which makes it impossible to verify the part of the content. Every time Appium switches pages, it needs to manually refresh the page. Airtest can display the interface in real
time , the connection method of Airtest will be slightly simpler than that of Appium
6. Regarding the maintenance of subsequent versions, Appium’s current community maintenance is stable, and it is updated every few days. Airtest currently sees that the latest maintenance time is 3 months ago. In comparison, Appium’s maintenance work will be better
7. Regarding multi-device parallelism, both Appium and Airtest can run multiple devices. At present, I start multiple wda services based on Tidevice and open multiple port numbers. The usage is similar.
8. Compared with other aspects, the comparison is basically the same
Since the company's App is related to native + H5, and I have a good understanding of the use of Appium, if the difference between the two is not very big, I still choose to use Appium for iOS automated testing
Three Appium driving principles
Driving principle
XCUITest is a framework for IOS automation testing developed by Apple. You need to know some iOS programming knowledge such as Swift.
WebDriverAgent is an iOS automation testing tool developed by Facebook. Let's take a look at the schematic diagram below:
WDA creates a Server on the Client, and installs an application called WebDriverAgentRunner on the mobile phone; this application will receive instructions from the Server, and connect to the underlying XCTest.framwork, so that XCTest.framwork calls the Apple API to operate the mobile phone for automation
And appium integrates WebDriverAgentRunner, so it realizes the cross-platform capability of appium
From the picture above, we know that Appium directly integrates the entire WebDriverAgent into its own project, and then uses WebDriverAgent as the communication mechanism. Appium actually provides a client function. So after iOS 9.3 system, the core of automated testing is WebDriverAgent, and Appium provides a client to write scripts and send commands.
The Appium automation architectural pattern can be represented by an abstract architecture, as shown in the following diagram:
It can be seen from the figure:
- The client side was previously provided by Appium itself;
- The server side is: WebDriverAgent and Instruments; (Appium directly integrates WebDriverAgent as a whole, and Instruments is to support the system before iOS 9.3)
- Before Server communicated with bootstrap.jar, here WebDriverAgent provides WebDriverAgentRunner (similar to jar function), and WebDriverAgent communicates with it;
- WebDriverAgentRunner is an application. After the Client and server are running, WebDriverAgentRunner will be installed on the mobile phone. This application will receive instructions from the Server, connect to the underlying framwork, and tell XCTest.framwork to operate the mobile phone for automation
About WebDriverAgent
Facebook listings:
- Implemented a server, through which the iOS device can be remotely controlled: start the application, close the application, click, scroll and other operations;
- Call Apple's API to perform actions by connecting the framework;
- Support multiple devices to automate at the same time;
- Appium, Macaca have been integrated.
- However, WebDriverAgent only provides a server (and inspect for element positioning), and does not provide a java or python client to write scripts like Appium. When the script is executed, it sends instructions to the server and then runs it. WebDriverAgent requires you to implement the client side by yourself, that is, to encapsulate the WebDriver library in Java/Python, and then send commands. So WebDriverAgent is actually similar to Appium server, just a server.
Introduction to Appium
Principles of Appium
- Appium is an open source, cross-platform testing framework that can be used to test native and hybrid mobile applications. Appium supports IOS, Android and FirefoxOS platforms. Appium uses the jsonwire protocol of WebDriver to drive the UIAutomation library of the Apple system and the UIAutomator framework of the Android system. Appium's support for the IOS system benefits from Dan Cuellar's research on IOS automation. Appium also integrates with Selendroid to support older android versions.
- Appium supports all languages supported by Selenium WebDriver, such as java, Object-C, JavaScript, Php Python, Ruby, C#, Clojure, or Perl language, and you can use Selenium WebDriver's Api. Appium supports any kind of testing framework. If we only use Apple's UIAutomation, we can only use javascript to write test cases, and we can only use Instruction to run test cases. Similarly, if we only use Google's UIAutomation, we can only write test cases in java. Appium enables true cross-platform automated testing.
- Appium chose the client-server design pattern. As long as the client can send http requests to the server, then the client can use any language to implement it. This is how appium and webdriver support multiple languages;
Advantages of Appium
- open source
- Cross-architecture: Native App, Hybrid App, Web App
- Cross-device: Android, iOS, Firefox OS
- Does not depend on source code
- Write test cases in any WebDriver-compatible language. Such as Java, Objective-C, JavaScript with
Node.js (in both callback and yield-based flavours), PHP, Python,
Ruby, C#, Clojure, or Perl. - No need to recompile APP
- Support IOS mobile phone recording video
Appium Philosophy
- You don't need to recompile or modify your application for automation.
- You don't have to be limited to a certain language or framework to write and run test scripts.
- A mobile automation framework should not reinvent the wheel on interfaces. (The interface of mobile automation should be unified)
- Whether in spirit or in name, it must be open source.
Changes in Appium's tools under iOS
- Before iOS 9, UIAutomation under instruments has been used as the driving underlying technology (the disadvantage is due to
the limitation of instruments, a single mac can only correspond to a single device); - The XCUITest tool was introduced in the iOS 9.3 era to replace UIAutomation;
- In the iOS 10 era, Apple directly abandoned UIAutomation, and Facebook launched WebDriverAgent (the implemented server
can support a single mac to correspond to multiple devices); - Appium fully adopts the WebDriverAgent solution after iOS 9.3.