Automated testing mentioned, very common misconception is to equate to some kind of automated testing automated testing tools, such as the most widely known Selenium, but this is only understood in the narrow sense. In real project practice, the content contains a lot of automated testing, not just some sort of automated testing tools, I understand a sophisticated automated test platform, these should include the following aspects:
- Automated tool frame or operating system under test: for example, white Selenium web programs, Appium, windows application program of a mobile terminal, autoit, QTP like. Please note that they are only automation framework provides a function capable of automatic operation of the system under test, the test frame is not because they do not have the test tissue framework of the basic use cases, the results of the check and other functions, it is basically required and in practice a testing framework to integrate use.
- Testing framework: the framework of integration and automation, providing use cases sorting, classification, scheduling, result verification, report output and other functions, such as Java's Junit, TestNG, Python's Pytest, Ruby's Rspec, and the ability to adapt to a variety of Cucumber and other languages.
- Construction of the test environment: Using Vmware, AWS and other virtualization and cloud technologies to build and manage the operating system level server, build and manage application-level services such as the use of docker container technology. When multiple containers can use docker compose, docker swarm, etc., may be used when kubernetes more complex to manage and maintain the whole environment. Test environment ideally should be automated one-button deployment, with full deployment scripts under version management and system control. Traceability can be rolled back, can be easily built from scratch, you can quickly build up in a local test environment, test personnel have their own exclusive test environment for control over the environment, along with the ability to safeguard the environment, can at any time according to need to build or destroy.
- Test data preparation: automated testing requires a complete, controlled test data as support, or automated testing is furnished, but can not predict the results of the implementation will be a burden to the team. The main obstacle fact that the human capacity and rights management, need to test staff for operations and systems have a deep understanding, to data (such as business operations and data which fields which API corresponding accurate preparation required, and the corresponding database which table). Preparing data from a bottom-up can be built directly into the database (typically more difficult to achieve, which has the right to the database, and familiar with the table structure, but it is the most efficient method), create a data call the API (API message needs to be familiar with the system under test structure, and have permission) to create (the most visible through UI automation script, but the least efficient, least stable), after executing automated tests also need to have the appropriate way to clean up the data. Another way is through mock server, so that both rely on the system to solve the problem, but also ready to complete data-dependent, but the development of mock server requires a certain amount of work, and need to be very familiar with the system and services, and the use of mock server will lead to the authenticity of the tests are unable to detect problems between the real system integration. Java's spring boot, wiremock, python's flask, Ruby's sinatra and so can be used to make mock server.
- Concurrent execution: concurrent execution is an effective way to test execution speed improvement, there are different ways to implement concurrent execution, such as the use of the parallel test function building tools, such as the maven surefire and failsafe, gradle multiple threads of execution functions, cucumber-jvm of concurrent execution. You can also use other tools or libraries to achieve, such as ruby of parallel-test, and then more complex, you can write a special program or service to test distributed to execute on different docker container, in BAT and other top Internet companies Share in the test had such a case. Another challenge is the need to address the concurrent execution of good test cases, test data, problem isolation test environment, to avoid interference occurs concurrent with each other. This requires good at decoupling the design test, for each test or test scenarios can all be performed alone, the case with embodiment among interdependent absent. In the preparation of the test data, to do the uniqueness and independence of data, data prepared in advance before execution, clearing unnecessary data after the implementation, but also consider some scenarios synchronize shared data when running multi-threaded deadlock and other issues, but the current mainstream programming languages such as Java, Python when handling multi-threaded programming has a very mature approach. About preparing test environment, according to the project scenario there are different methods can be performed on multiple test environments, can also be performed on only one environment, but some systems may not exist accounts while landing and other issues need to be considered in advance, or to circumvent . Here I would like to make a docker for advertising, lightweight docker container technology, low-overhead start destruction, between containers isolated from each other and so on, well suited to rapidly build and test environments to perform concurrent testing.
- Integration with CI: automated testing only and largest integrated CI to play, or if can only be performed on the machine testers, there will be no visible results of the team, the lack of standardized scheduling policy, can not lead to the team's attention to other issues, resulting in automated test useless. And a complete line of CI also need to include automated testing, agile philosophy in a word, there is no automated testing of the CI can not be called CI. Automatic Test and CI integration is not technically difficult to achieve, in essence, is to test up executed by the execution machine is performed locally into the CI, the current mainstream CI tools, such as Jenkins, Bamboo, CircleCI etc. can easily complete the configuration . It should be noted is still a test environment, where the test environment consists of two parts, the environment and the implementation of automated testing system under test environment. Environment tested system needs to be able to be accessed CI execution machine, perform automated testing environment in general in the implementation of CI machine, you need to configure the machine to perform a CI environment, such as Java or Python environment, Webdriver, browser and so on. This will involve a lot of details, such as CI enforcement mechanisms are generally Linux environment, you can not run the graphical interface, you need to use your browser's headless mode, still strongly recommended docker to encapsulate the entire test execution environment, so that the implementation of the CI the machine requires only docker pull, docker run on it, because to avoid differences in the environment, such as dependencies brought many problems. It is described in another post my article how to perform automated testing docker container .
- And Integrated Use Case Management System: The automated test execution will generate test reports, but further, if we can integrate and use case management system, automatic annotation results of the implementation, it is more easy, and in many projects is common practice. Use case management systems vary widely, some of which are developed in-house system, we need to analyze and solve different companies and projects used in accordance with the actual situation. But generally require the use case management system can provide the results of the API label, there is a special program after test execution to collected test report, then call the API with the case management system to label the results. With the industry's best known JIRA, for example, provides an API to query and mark the results of test case, so only need to tidy up the results of message formats, it can call, not difficult to achieve in accordance with the provisions of API.
Above it describes a common automated test platform which features need to be included and how to implement these functions. If the mobile end of the test platform, there are more problems, such as the compatibility of different devices, how concurrency and scheduling multiple different devices, BAT has proven practices in these top Internet company, my own experience in this area and not much, still in the learning stage of exploration. In essence, to build a more complete test automation platform, you need to have in-depth understanding of the system under test, there is a solid basis for software testing and programming, familiar with at least one programming language, familiar with the CI and Devops, have broad knowledge and surface technology stack, know how to choose the right tools and framework to solve the problem.