Code coverage and precision testing that can be learned only by black-box testing

Common coverage statistics tools

emma
roof
jacoco

emma and cobertura are coverage statistics designed for unit testing, jacoco and emma belong to the same company, but are tools designed for broader coverage statistics.

jacoco

There is a mission chapter in jacoco's documentation, which describes the positioning of jacoco very well. The general idea of the original text is that other tools are not actively and effectively maintained, and other tools are designed for a single task, they are not born for "integration". From this point we can see the design concept of jacoco.

Thanks to jacoco's design concept and good api design, it can be easily integrated with existing tools and even platformized. It can also be used for unit testing and integration testing at the same time, so it is a very good coverage statistics tool. Many companies rely heavily on jacoco for precise testing.

Principles of Coverage Analysis

To understand the statistical principle of code coverage, we need to understand the mechanism of jvm in depth. This knowledge is high-end advanced knowledge in the field of java. Due to space limitations, we only explain the general principles. For the complete content, please refer to the books of the VM virtual machine series and some code insertion materials released by newrelic in the early years.

Briefly speaking, the java source code will be compiled into a class file by javac, and the class file saves the basic information of the class and the instruction set of the jvm. The underlying runtime of java, that is, when the jvm parses the class, it will read the class in the file format into the memory and run it. Android also draws on this whole set of design concepts. The runtime on Android is actually dalvik and art.

When we want to count code coverage, we need to add probe analysis to the code execution path. Usually, when reading a class, add tags at the exit and entry of key instruction blocks. When the instruction block is executed, the probe will be hit and the recording will be completed.

It is often troublesome to modify the bottom-level jvm bytecode, and it is necessary to be proficient in various jvm instructions and java class structures. At present, there are very mature open source projects that can be expanded to deal with this aspect. The following are some well-known bytecode modification tools.

ASM
JavaAssist
ByteBuddy, BTrace, JVM-Sandbox Among them, ASM is the underlying foundation of all bytecode operations and the bottom bytecode modification tool. Other tools are some high-level wrappers on top of it. With the help of these tools and some debugging features of the JVM itself, we can conveniently manipulate the jvm code or process.

Insertion method

There are many ways to insert piles, and the common ways are as follows

Source code instrumentation: offline instrumentation, support for android
Bytecode instrumentation: offline instrumentation, support for android
javaagent mode: run-time instrumentation without code, on the fly mode jacoco supports bytecode instrumentation and javaagent two instrumentation methods. That is to say, even if there is no source code, the coverage data can be counted, but in the final analysis, it is still necessary to combine the source code to obtain more detailed coverage data. After all, coverage statistics are not simply the level of coverage data itself.

How jacoco works

jacoco supports four working modes

file: Generate a file locally when the process ends
tcpserver: open the port and wait for the client to obtain coverage
tcpclient: actively send coverage data
none: Do not generate coverage Many people will use the file mode, but the tcp server mode is the easiest to use. Coverage data can be controlled because there is no need to apply for file access permissions on the server. You can choose the appropriate working mode according to your company's deployment situation.

The on the fly instrumentation mode is the most used. First, you need to add some javaagent parameters of jvm when your tested java program starts.

-javaagent:[yourpath/]jacocoagent.jar=[option1]=[value1],[option2]=[value2]
destfile
output：file、tcpserver、tcpclient、none
address
port

You can set the suitable working mode by yourself. The offline instrumentation mode is suitable for the coverage statistics of android, and requires the instrument instruction of construction tools such as maven and gradle.

jacoco-cli is a component of jacoco, which can complete the code analysis without relying on maven and gradle build tools. It is mainly used in tcpserver working mode. The usage is as follows

java -jar jacococli.jar dump [--address <address>] --destfile <path> [--help] \
[--port <port>] [--quiet] [--reset] [--retry <count>]

This is a small scene of a hands-on exercise in the academy, counting the startup coverage of the jmeter tool

project_root=/Users/seveniruby/temp/java_2/jacoco/apache-jmeter-5.2.1
jacoco_cli_jar=org.jacoco.cli-0.8.6-20200329.124045-45-nodeps.jar

java -javaagent:org.jacoco.agent-0.8.6-20200329.124039-45-runtime.jar \
  -jar $project_root/bin/ApacheJMeter.jar

#退出jmeter

#生成覆盖率报告
java -jar $jacoco_cli_jar report jacoco.exec \
--classfiles "$project_root/bin/ApacheJMeter.jar" \
--classfiles $project_root/lib/ext/ApacheJMeter_http.jar \
--html jmeter_coverage/

#生成带有源代码的覆盖率报告
 java -jar $jacoco_cli_jar report  jacoco.exec  \
 --classfiles "$project_root/bin/ApacheJMeter.jar" \
 --classfiles $project_root/lib/ext/ApacheJMeter_http.jar \
 --html jmeter_coverage/ --sourcefiles ~/projects/jmeter/src/

#生成xml报告
java -jar $jacoco_cli_jar report jacoco_tcpserver2.exec  \
--classfiles "$project_root/bin/ApacheJMeter.jar" \
--classfiles $project_root/lib/ext/ApacheJMeter_http.jar \
--xml  jmeter_coverage_tcpserver2/jacoco.xml

Sonarqube can analyze jacoco's exec files and xml files, and automatically import coverage. The analysis of exec files will be abandoned later, and the analysis of xml files is mainly supported.

sonar-scanner   \
  -Dsonar.host.url=http://sonarqube.testing-studio.com:9000   \
  -Dsonar.login=$SONARQUBE_TOKEN   \
  -Dsonar.projectKey=jmeter   \
  -Dsonar.projectVersion=1.0  \
  -Dsonar.coverage.jacoco.xmlReportPaths=$PWD/jmeter_coverage_tcpserver2/jacoco.xml \
  -Dsonar.projectBaseDir=/Users/seveniruby/projects/jmeter/ \
  -Dsonar.java.binaries=/Users/seveniruby/projects/jmeter/

import coverage

The scope of the code to be analyzed can be limited by the following parameters, usually specifying the scope of the package to be covered

sonar.sources
sonar.inclusions

import coverage

Sonarqube can intelligently analyze the coverage of new code between old and new versions, which is very good, and some previous work of analyzing code diff is saved. His code analysis is also smarter, and a simple line break will not interfere with the analysis scope of code diff.

by coverage data

sonarqube supports two general test data imports

Generic coverage data: sonar.coverageReportPaths
General test execution data: sonar.testExecutionReportPaths This facilitates integration with various frameworks and facilitates secondary customization by test engineers. Generic test data template.

Generic Coverage Data Template

In the scanner analysis of sonarqube, general test data can be imported by adding corresponding configuration parameters. Common Uses of Generic Test Data Import

Convert the test reports of various test tools into formats supported by sonarqube and import them into the platform
Convert the coverage reports of various coverage tools into standard formats and import them into the platform
Customize the differential diff coverage according to requirements. For example, in addition to covering the new code, it is also necessary to analyze the dependent code that uses the new code

Code diff analysis is a relatively large topic. Diff is only the simplest analysis strategy for code. To understand the code in depth, we need to further analyze the call chain of the code. Let's look at the simplest code diff analysis tool first. Code diff analysis tool

JGit: git analysis tool
JavaParser: syntax analysis
ASM: read bytecode
javap: jdk comes with bytecode analysis tools

Finally: The following are the supporting learning materials. For friends who do [software testing], it should be the most comprehensive and complete preparation warehouse. This warehouse also accompanied me through the most difficult journey. I hope it can help you too!

Software testing interview applet

The software test question bank maxed out by millions of people! ! ! Who is who knows! ! ! The most comprehensive quiz mini program on the whole network, you can use your mobile phone to do the quizzes, on the subway or on the bus, roll it up!

The following interview question sections are covered:

1. Basic theory of software testing, 2. web, app, interface function testing, 3. network, 4. database, 5. linux

6. web, app, interface automation, 7. performance testing, 8. programming basics, 9. hr interview questions, 10. open test questions, 11. security testing, 12. computer basics