Breaking the landscape of software automation testing

Mistakes in automated testing

Automated testing is only considered to replace manual labor, so we see many companies implementing automated testing just to convert existing Test Cases into automated scripts.

Doing so neither improved the overall level of the test, nor did it improve the test results. The result is that problems that can be tested manually can be tested by automated testing, and problems that cannot be tested manually are not tested by automated testing.

Because the concept of testing is still in the stage of existing Test Case, and Test Case is in the stage of business process testing.

The final automated test is only to go through the business process according to the test case and complete the inspection of the business process.

Problems with layering and deployment

With the development of technology and the diversity of software, testing is not limited to GUI testing based on CS structure, but WEB UI testing based on BS browser. For example, the current Android system, Apple IOS system, Microsoft's Windows Mobile system, etc. have also joined the field of automated testing.

Application software is also becoming more complex, such as:

Hierarchical changes: interface layer, interface layer, business logic layer, entity model layer
Changes in deployment: from single-machine operation to dual-machine hot backup to load balancing, and more recently to distributed systems.
Changes stored: relational databases, non-relational databases, cache databases, search engine databases

From the pyramid structure below, it can be seen that only the UI interface layer is displayed to the user by the software

/\
           /  \
          / UI \
         /------\
        /   API  \
       /----------\
      /   Service  \     
     /--------------\
    /    Component   \
   /------------------\  
  /      Database      \
 /______________________\

The above is the layering of software. After a software is deployed, the structure will be more complex.

/\
           /  \
          /CDN \
         /------\
        / WEBSER\
       /----------\
      / APP Server \     
     /--------------\
    / Message Queue  \
   /------------------\  
  / Cache|SearchEngine \
 /   Database| NoSQL    \ 
/________________________\

As far as WEB application testing is concerned, the content involved is too extensive, from browser->WEB server->APP server->cache->database, through various proxies, load balancing, distributed file system and so on.

Our tests cover:

CDN test, domain name resolution test,
WEB UI testing, including HTML, Ajax
API server test, api is a non-human-computer interaction interface, which communicates with the API server through a specific protocol.
code unit testing
Configuration testing, testing after configuration changes during configuration management, including systems and applications
Security testing, interface security, authentication, permissions
Injection testing, JS injection, SQL injection, Shell injection
Cache test, hit rate test, including CDN, WEB server, cache server, search engine
Stress testing, robustness testing
Expansion test, horizontal expansion test, vertical expansion test
High availability test, cluster test

Problems with stress testing

Please refer to my other article "Problems in Stress Testing"

Here I want to emphasize stress testing separately. Many people's testing methods are problematic.

Stress testing is not about preparing a machine to install stress testing software to start testing. The environment for stress testing is very important, and many testers who have worked for many years are unaware of this problem.

The stress test has two focuses, one is the construction of the stress test environment, and the other is the stress test sequence.

Stress Test Environment

Whether it is a single computer or a network, a stress test requires a good stress test environment. For example, the network is like a highway. If the highway becomes a bottleneck, can you test accurate data?

First prepare the test environment. For example, the CPU speed, disk IO speed, RAID card speed, RAID card cache size, memory speed, PCI-E bus speed, and even multi-symmetric CPU-related configuration, memory and CPU channels are involved in stand-alone testing. Questions... wait

If you are testing a distributed system, in addition to the above considerations for a single node, you should also consider the packet forwarding and connection limit of the router/firewall, the backplane bandwidth and throughput of the switch, and the forwarding capability of the load balancer.

test order

The entry point of the stress test sequence is very important. Most people start the test sequence from the UI (human-machine interface), that is, the UI drives the business logic. This test sequence is wrong, such as user->browser->WEB server- >APP server->cache->database and so on, which brings a lot of problems.

\------------------/
 \    Web server  /
  \   App Server /
   \ Cache / MQ /
    \ Database /
     \ Disk IO /
      \      /

The performance bottleneck of software is usually an hourglass type. The biggest bottleneck is the database. We can solve the performance problem from the perspective of architecture for other server bottlenecks.

All we should test from the database, first confirm whether the configuration optimization of the database can meet our expectations. Then there are caches, message queues, search engines, etc...

So far we know that databases, caches, message queues, and search engines will not be the bottlenecks in our stress test. The next step is to test the application server and application software.

If your testing landscape can be scaled up a bit, there is much more to consider than the above. You also need to consider hardware, network, operating kernel parameter optimization, TCP/IP stack optimization, verifying whether the operation and maintenance configuration can meet our needs, etc...

Bottleneck Analysis

We need a set of monitoring solutions that can monitor the performance of hardware and software.

The purpose of the test is not to get a result and tell the developer that your software can support XXX concurrency, but to monitor each operation in our test, calculate the time used by each function, analyze the performance bottleneck, and guide the developer to improve software.

Monitoring is divided into external monitoring and internal monitoring.

External monitoring is the easiest to implement, with mature tools and solutions, CPU, memory, disk IO, network traffic, and more.

Internal monitoring refers to the state of changes after the software is loaded into the memory, such as memory addresses, variables, function calls, dynamic link library loading, open file handles, socket addresses, and data packets.

guide development

Quickly locate the problem points of the software through data and charts, and guide the development to complete the improvement of the software

Continuous integration is nothing

Continuous integration and automated construction are almost always implemented by a test team, but the actual situation is not ideal, and it only stays in the stage of tool configuration. Few people use automated builds in production.

Why can't continuous integration be applied to production environment?

(To be continued, please pay attention to the author's WeChat public account, it is now 6 o'clock in the morning, and I am going to bed)

The ultimate goal of testing

I think that testing is not only to complete the software acceptance according to the test cases, if only testing the user-visible UI (human-machine interface) can not meet the testing needs of modern software.

Testers should look at problems from a higher perspective, and testers are capable of guiding developers, improving software performance, robustness, security, and influencing the design of software architecture. Testers need to have extensive cross-border knowledge support, continuous learning and improvement, and breaking the existing pattern.

2016-12-03 06:30 AM