Software Testing-TestNG must know and know

Share a big cow's artificial intelligence tutorial. Zero-based! Easy to understand! Funny and humorous! Hope you join the artificial intelligence team too! Please click http://www.captainbed.net

TestNG has several notable and easy-to-use features.

Famous annotation function
Concept of test group
Multi-threaded test
Support dependent test method, parallel test, load test, partial failure

Then TestNG has the above characteristics, how to integrate it into your framework and use it?

Step 1: You need to confirm that the Java environment has been configured.

The second step:

1. If maven is configured in the project, just introduce the following dependencies

<dependency>
    <groupId>org.testng</groupId>
    <artifactId>testng</artifactId>
    <version>6.14.3</version>
</dependency>

2. If there is no maven, you can directly import TestNG.jar

The third step: write test business logic, insert TestNG annotations in the code

Step 4: Configure test information in the configuration file testng.xml or build.xml corresponding to TestNG

Step 5: Just run TestNG

The benefits of annotations have been mentioned before, so let's start with the most widely used @Test annotation. The method or class marked by @Test is our corresponding test case. As shown in the code below, runTest1 and runTest2 are two test cases written by ourselves.


package com.tester;

import org.testng.annotations.Test;

public class TestInterface {

    @Test
    public void runTest1() {
        System.out.println("@Test - runTest1");
    }

    @Test
    public void runTest2() {
        System.out.println("@Test - runTest2");
    }
}

@Test is that simple? No, its essence lies in its following parameters.

groups: group test, which group a test belongs to, can run all tests of a specific group
dependsOnMethods, dependsOnGroups: dependent test, one test depends on the execution result of another test
expectedExceptions: exception test
dataProvider: parameterized test, pass parameters to the test
enabled: Ignore the test, do not execute the test
timeOut, threadPoolSize, invocationCount, successPercentage: concurrent testing, setting various parameters of concurrent testing
alwaysRun: If true, it will run anyway

This time I first introduce an innovation of TestNG, group testing. It does not exist in the JUnit framework, it allows you to execute all test cases in a test group. Not only can you declare which group a test case belongs to, you can also specify a group to include other groups. Then, TestNG can be called and required to include a specific set of groups (or regular expressions), while excluding another set. This gives us the greatest flexibility in how to partition tests. If you want to run two different sets of tests back to back, you don't need to recompile anything, just run different groups.


package com.tester;

import org.testng.annotations.AfterGroups;
import org.testng.annotations.BeforeGroups;
import org.testng.annotations.Test;

public class TestGroup {

    @Test(groups = "appium-test")
    public void runAppium() {
        System.out.println("runAppium()");
    }

    @Test(groups = "appium-test")
    public void runAppium1() {
        System.out.println("runAppium1()");
    }

    @Test(groups = "interface-test")
    public void testConnectOracle() {
        System.out.println("testConnectOracle()");
    }

    @Test(groups = "interface-test")
    public void testInterface() {
        System.out.println("testInterface()");
    }
}

In the above code, we can see that two test groups appium-test and interface-test are declared, and there are two test cases under each test group. There is also a case where the entire class is directly classified as a test group as shown in the following code.


package com.tester;

import org.testng.annotations.Test;

@Test(groups = "selenium-test")
public class TestSelenium {

    public void runSelenium() {
        System.out.println("runSelenium()");
    }

    public void runSelenium1() {
        System.out.println("runSelenium()1");
    }
}

How does it work? As mentioned above, the corresponding testng.xml needs to be configured. We write the corresponding testng.xml as follows.


<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >

<suite name="TestAll">
    <!-- 只跑selenium-test这个测试组 -->
    <test name="selenium">
        <groups>
            <run>
                <include name="selenium-test" />
            </run>
        </groups>

        <classes>
            <class name="com.tester.TestSelenium" />
            <class name="com.tester.TestGroup" />
        </classes>
    </test>
</suite>

Of course, a test method or a class can also belong to multiple groups at the same time.

@Test(groups = {"appium","uiautomator"})
public void testUI() {
    System.out.println("testUI");
}

Next, let's talk about the parameters of this annotation of @Test.

In addition to the groups parameter, @Test has two more important parameters dependsOnMethods and dependsOnGroups. With these two parameters, we can do dependency testing.

What is dependency testing? Sometimes it is necessary to call test cases in a specific order, or share some data and state between methods. At this time, we can use dependsOnMethods or dependsOnGroups to declare that the annotated method will only be executed after the dependent method is successfully executed.

package com.tester;

import org.testng.annotations.Test;

public class App {

    @Test(dependsOnMethods = { "method1" })
    public void method2() {
        System.out.println("This is method 2");
    }
    
    @Test
    public void method1() {
        System.out.println("This is method 1");
    }

}

The annotation of method2 declares dependsOnMethods = {"method1" }, that is, method2 must be executed after method1 is successfully executed and passed. If method1 fails to execute, method2 will not be executed. In our interface automation framework, for example, some of our interfaces rely on the login state, then our dependency test can play a big role. We can declare the methods of these interfaces to depend on the login interface. After the login interface is successful, we get the login state before executing the following interface.

The group test is mentioned above. Combining the group test and the dependency test is dependsOnGroups.

package com.tester;

import org.testng.annotations.Test;

// 声明这个类中的方法都是deploy组的
@Test(groups = "deploy")
public class TestServer {
    @Test
    public void deployServer() {
        System.out.println("Deploying Server...");
    }

    // 只有deployServer通过后才跑
    @Test(dependsOnMethods = "deployServer")
    public void deployBackUpServer() {
        System.out.println("Deploying Backup Server...");
    }
}

package com.tester;

import org.testng.annotations.Test;

public class TestDatabase {

    //属于db组
    //只有deploy组全部通过后才跑
    @Test(groups="db", dependsOnGroups="deploy")
    public void initDB() {
        System.out.println("This is initDB()");
    }

    //属于db组
    //只有initDB方法通过后才会跑
    @Test(dependsOnMethods = { "initDB" }, groups="db")
    public void testConnection() {
        System.out.println("This is testConnection()");
    }
}

In the above code, it is a mixed use of dependsOnMethods and dependsOnGroups. All methods in the TestServer class belong to the deploy group. The initDB method will run only after the deploy group is passed, and testConnection will run only after the initDB has passed.

After talking about dependency testing, let's talk about expectedExceptions for exception testing. If you expect an exception will occur during the test, you can add expectedExceptions to do the expected exception test.

package com.tester;

import org.testng.annotations.Test;

public class TestRuntime {

    @Test(expectedExceptions = ArithmeticException.class)
    public void divisionWithException() {
        int i = 1 / 0;
        System.out.println("After division the value of i is :"+ i);
    }
}

In the above code, dividing by 0 will definitely run out of ArithmeticException, we can declare the exception to do the expected exception test.

Sometimes, the code we write is not ready, and the test case needs to test whether the method/code fails (or succeeds). We can use to (enabled = false)disable this test case.


package com.tester;

import org.testng.Assert;
import org.testng.annotations.Test;

public class TestIgnore {

    @Test // 默认 enable=true
    public void test1() {
        Assert.assertEquals(true, true);
    }

    @Test(enabled = true)
    public void test2() {
        Assert.assertEquals(true, true);
    }

    @Test(enabled = false)
    public void test3() {
        System.out.println("俺没有执行");
        Assert.assertEquals(true, true);
    }
}

The test3 method that declares enabled = false will not be executed.

We also have a parameter opposite to this one, which is alwaysRun. The case where alwaysRun = true is declared will be executed no matter what.


package com.tester;

import org.testng.Assert;
import org.testng.annotations.Test;

public class TestAlwaysRun {

    @Test // 默认 enable=true
    public void test1() {
        Assert.assertEquals(true, false);
    }

    @Test(dependsOnMethods = { "test1" },alwaysRun = true)
    public void test2() {
        Assert.assertEquals(true, true);
    }
}

In the above code, test2 depends on test1, but test1 asserts that true and false are equal and will inevitably fail. If it is a dependent test, test2 will definitely not be executed. But test2 declares alwaysRun = true, so test2 will still be executed.

A very interesting feature in TestNG is parameter testing. In most cases, you will encounter such a test scenario where the business logic requires a huge and varying amount of testing. Parametric testing allows developers to run the same test, using different values over and over again. For example, in our interface automation framework, everyone knows that our interface testing needs to assign different values to the input parameters to test whether the business logic of the interface is correct. At this time, it is very convenient for you to use TestNG.

TestNG allows you to pass parameter test methods directly in two different ways: using testng.xml and data provider.

Use xml to pass parameters

Passing parameters through xml needs to be applied to the parameter @Parameter in the @Test annotation.

Create a name: TestParameterXML.java , and its code is as follows.

package com.tester;

import org.testng.annotations.Parameters;
import org.testng.annotations.Test;

public class TestParameterXML {

    @Test
    @Parameters({ "dbconfig", "poolsize" })
    public void createConnection(String dbconfig, int poolsize) {

        System.out.println("dbconfig : " + dbconfig);
        System.out.println("poolsize : " + poolsize);
}

Create a file named: testng.xml , the code is as follows


<?xml version="1.0" encoding="UTF-8"?>
<suite name="test-parameter">

    <test name="example1">

        <parameter name="dbconfig" value="db.properties" />
        <parameter name="poolsize" value="10" />

        <classes>
            <class name="com.yiibai.TestParameterXML" />
        </classes>

    </test>
    
</suite>

First, we see testng.xml, we declare two <parameter>, respectively dbconfig and poolsize, and this TestParameterXML.java in @Parameters and createConnection two parameters corresponding named, so that we can xml The value defined in is passed into the method for printing.

Pass parameters through @DataProvider

Just pass parameters through xml, which is relatively cumbersome. We can use @DataProvider to pass parameters more flexibly.

Create a file named: TestParameterDataProvider.java , the code is as follows

package com.tester;

import org.testng.Assert;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;

public class TestParameterDataProvider {

    @Test(dataProvider = "provideNumbers")
    public void test(int number, int expected) {
        Assert.assertEquals(number + 10, expected);
    }

    @DataProvider(name = "provideNumbers")
    public Object[][] provideData() {

        return new Object[][] { { 10, 20 }, { 100, 110 }, { 200, 210 } };
    }
    
}

@Test(dataProvider = "provideNumbers"和@DataProvider(name = "provideNumbers")

Corresponding to the name in. Use @DataProvider to declare the data set, then the corresponding matching method can get the parameters in the data set and correspond in order. The results of the code run are as follows:


[TestNG] Running:
  \Users\abc\AppData\Local\Temp\testng-eclipse--1925148879\testng-customsuite.xml

PASSED: test(10, 20)
PASSED: test(100, 110)
PASSED: test(200, 210)

===============================================
    Default test
    Tests run: 3, Failures: 0, Skips: 0
===============================================


===============================================
Default suite
Total tests run: 3, Failures: 0, Skips: 0
===============================================

[TestNG] Time taken by org.testng.reporters.XMLReporter@1b40d5f0: 13 ms
[TestNG] Time taken by org.testng.reporters.SuiteHTMLReporter@6ea6d14e: 34 ms
[TestNG] Time taken by org.testng.reporters.EmailableReporter2@4563e9ab: 7 ms
[TestNG] Time taken by [FailedReporter passed=0 failed=0 skipped=0]: 0 ms
[TestNG] Time taken by org.testng.reporters.jq.Main@2aaf7cc2: 69 ms
[TestNG] Time taken by org.testng.reporters.JUnitReportReporter@45c8e616: 4 ms

Of course @DataProvider not only supports basic types, but also implements object parameters.

package com.tester;

import java.io.IOException;
import java.io.InputStream;
import java.util.HashMap;
import java.util.Map;
import java.util.Properties;

import org.testng.Assert;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;

public class TestParameterDataProvider2 {

    @Test(dataProvider = "dbconfig")
    public void testConnection(Map<String, String> map) {

        for (Map.Entry<String, String> entry : map.entrySet()) {
            System.out.println("[Key] : " + entry.getKey() + " [Value] : " + entry.getValue());
        }
    }

    @DataProvider(name = "dbconfig")
    public Object[][] provideDbConfig() {
        Map<String, String> map = new HashMap<String, String>();
        map.put("jdbc.driver", "com.mysql.jdbc.Driver");
        map.put("jdbc.url", "jdbc:mysql://localhost:3306/test");
        map.put("jdbc.username", "root");
        map.put("jdbc.password", "123456");
        return new Object[][] { { map } };
    }
}

Then our final running results are as follows:


[TestNG] Running:
  \Users\abc\worksp\testng\ParameterTest\src\main\java\com\yiibai\testng.xml

path => F:\worksp\testng\ParameterTest\db.properties
drivers : com.mysql.jdbc.Driver
connectionURL : jdbc:mysql://localhost:3306/test
username : root
password : 123456

Tue May 02 23:15:52 CST 2019 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.

===============================================
test-parameter
Total tests run: 1, Failures: 0, Skips: 0
===============================================

Finally, there is another usage that is used in conjunction with TestNG's ITestContext, which can open new doors.


package com.tester;

import org.testng.Assert;
import org.testng.ITestContext;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;

public class TestParameterDataProvider4 {

    @Test(dataProvider = "dataProvider", groups = {"groupA"})
    public void test1(int number) {
        Assert.assertEquals(number, 1);
    }

    @Test(dataProvider = "dataProvider", groups = "groupB")
    public void test2(int number) {
        Assert.assertEquals(number, 2);
    }

    @DataProvider(name = "dataProvider")
    public Object[][] provideData(ITestContext context) {
        Object[][] result = null;

        for (String group : context.getIncludedGroups()) {

            System.out.println("group : " + group);

            if ("groupA".equals(group)) {
                result = new Object[][] { { 1 } };
                break;
            }

        }

        if (result == null) {
            result = new Object[][] { { 2 } };
        }
        return result;

    }
}

Create a file named: testng.xml, the code is as follows


<?xml version="1.0" encoding="UTF-8"?>
<suite name="test-parameter">

    <test name="example1">

        <groups>
            <run>
                <include name="groupA" />
            </run>
        </groups>

        <classes>
            <class name="com.tester.TestParameterDataProvider4" />
        </classes>

    </test>

</suite>

If we declare to run groupA in our xml, we enter the judgment of if ("groupA".equals(group)). The final running results are as follows:


[TestNG] Running:
  \Users\abc\worksp\testng\ParameterTest\src\main\java\com\yiibai\testng4.xml

group : groupA

===============================================
test-parameter
Total tests run: 1, Failures: 0, Skips: 0
===============================================

Concurrent testing

Concurrent testing is actually running all tests in a multi-threaded mode, so that the maximum running speed can be obtained and the execution time can be saved to the greatest extent. Of course, concurrent operation also has a price, that is, our code is required to be thread-safe, and the running scene is not exclusive. The recommendation is to avoid using shared variables as much as possible in the test code. If you really use it, you should use the synchronized keyword with caution to lock and synchronize shared variables. Otherwise, it is inevitable that your use case may be unstable when it is executed. Of course, we can also use this combined with selenium for stress testing, combined with interface automation framework for stress testing, and so on. Let us come one by one.

1. Specify directly in the test method

Sometimes, we need to perform concurrent tests on a test case, such as an http interface, that is, repeated calls of an interface. An elegant support method is also provided in TestNG, specifying threadPoolSize and invocationCount in the @Test tag. This is an implementation directly specified in the test method.

package com.tester;

import java.text.SimpleDateFormat;
import java.util.Date;

import org.testng.annotations.AfterClass;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;


public class TestClass1 {
    private SimpleDateFormat df = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");//设置日期格式
    @BeforeClass
    public void beforeClass(){
        System.out.println("Start Time: " + df.format(new Date()));
    }

    @Test(enabled=true, dataProvider="testdp", threadPoolSize=2, invocationCount=5, successPercentage=98)
    public void test(String dpNumber) throws InterruptedException{
        System.out.println("Current Thread Id: " + Thread.currentThread().getId() + ". Dataprovider number: "+ dpNumber);
        Thread.sleep(5000);
    }

    @DataProvider(name = "testdp", parallel = true)
    public static Object[][]testdp(){
        return new Object[][]{
            {"1 "},
            {"2 "}
        };
    }

    @AfterClass
    public void afterClass(){
        System.out.println("End Time: " + df.format(new Date()));
    }
}

Among them, invocationCount=5 means that this test method needs to be executed 5 times, threadPoolSize=2 means that 2 threads are opened to execute, and successPercentage=98 means that the success rate exceeds 98% and it is considered successful.

@DataProvider(name = "testdp", parallel = true), the parallel attribute defaults to false, indicating that the test method using this data source cannot be executed concurrently. The parallel attribute is set to true, indicating that the test method using this data source can be executed concurrently.

Let's first look at the results of the operation, as follows:

Start Time: 2019-10-10 14:10:43
[ThreadUtil] Starting executor timeOut:0ms workers:5 threadPoolSize:2
Current Thread Id: 14. Dataprovider number: 2
Current Thread Id: 15. Dataprovider number: 2
Current Thread Id: 12. Dataprovider number: 1
Current Thread Id: 13. Dataprovider number: 1
Current Thread Id: 16. Dataprovider number: 1
Current Thread Id: 18. Dataprovider number: 1
Current Thread Id: 17. Dataprovider number: 2
Current Thread Id: 19. Dataprovider number: 2
Current Thread Id: 21. Dataprovider number: 2
Current Thread Id: 20. Dataprovider number: 1
End Time: 2019-10-10 14:10:58

The thread pool and the concurrent thread pool of dp are two independent thread pools. The thread pool here is used to set up multiple methods, and the test data for each method is provided by dp. If there are 5 sets of data in dp, then in fact 10 executions, each time the interface will be adjusted 5 times, this interface The total number of calls is 10*5=50 times. Among the 5 threads specified by threadPoolSize, when each thread adjusts the method individually, if the dp used also supports concurrent execution, a new thread pool (dpThreadPool) will be created to execute the test data concurrently.

2. Set through testng.xml


package com.tester;

import org.testng.annotations.AfterClass;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;

public class ParallelClassesTestOne
{
    @BeforeClass
    public void beforeClass() {
        long id = Thread.currentThread().getId();
        System.out.println("Before test-class. Thread id is: " + id);
    }
    @Test
    public void testMethodOne() {
        long id = Thread.currentThread().getId();
        System.out.println("Sample test-method One. Thread id is: " + id);
    }
    @Test
    public void testMethodTwo() {
        long id = Thread.currentThread().getId();
        System.out.println("Sample test-method Two. Thread id is: " + id);
    }
    @AfterClass
    public void afterClass() {
        long id = Thread.currentThread().getId();
        System.out.println("After test-class. Thread id is: " + id);
    }
}


package com.tester;

import org.testng.annotations.AfterClass;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;

public class ParallelClassesTestTwo
{
    @BeforeClass
    public void beforeClass() {
        long id = Thread.currentThread().getId();
        System.out.println("Before test-class. Thread id is: " + id);
    }
    @Test
    public void testMethodOne() {
        long id = Thread.currentThread().getId();
        System.out.println("Sample test-method One. Thread id is: " + id);
    }
    @Test
    public void testMethodTwo() {
        long id = Thread.currentThread().getId();
        System.out.println("Sample test-method Two. Thread id is: " + id);
    }
    @AfterClass
    public void afterClass() {
        long id = Thread.currentThread().getId();
        System.out.println("After test-class. Thread id is: " + id);
    }
}

We create a new testng.xml

<suite name="Test-class Suite" parallel="classes" thread-count="2" >
  <test name="Test-class test" >
    <classes>
      <class name="com.tester.ParallelClassesTestOne" />
      <class name="com.tester.ParallelClassesTestTwo" />
    </classes>
  </test>
</suite>

Select the file in Eclipse and run it as a TestNG test suite. You will see the following output in the console:


Before test-class. Thread id is: 10
Before test-class. Thread id is: 9
Sample test-method One. Thread id is: 9
Sample test-method One. Thread id is: 10
Sample test-method Two. Thread id is: 10
After test-class. Thread id is: 10
Sample test-method Two. Thread id is: 9
After test-class. Thread id is: 9

Because our testng has test, class, method level concurrency, you can set the following under the suite tag in testng.xml:

<suite name="Testng Parallel Test" parallel="tests" thread-count="5">

Description: During the execution of the current test plan, a separate thread is used for the execution of each test case (the test method in the test case shares one thread), and a maximum of 4 threads are concurrently used.

<suite name="Testng Parallel Test" parallel="classes" thread-count="5">

Note: During the execution of the current test plan, a separate thread is used for the execution of each test class (the test methods in the test class share one thread), and up to 4 threads are concurrently used.

<suite name="Testng Parallel Test" parallel="methods" thread-count="5">

Note: During the execution of the current test plan, a separate thread is used for the execution of each test method, and a maximum of 4 threads are concurrently used.

Annotate the order of execution

@BeforeSuite: The annotated method will be run before all tests are run
@AfterSuite: The annotated method will be run after all tests are run
@BeforeTest: The annotated method will be run before the test is run
@AfterTest: The annotated method will be run after the test is run
@BeforeGroups: The configured method will run before the gourp in the list. This method is guaranteed to be executed immediately before the first test method belonging to these groups is called
@AfterGroups: The configured method will run after the gourp in the list. This method is guaranteed to be executed immediately after the last test method belonging to these groups is called
@BeforeClass: The annotated method will run before the first test method of the current class is called
@AfterClass: The annotated method will run after all test methods of the current class are called
@BeforeMethod: The annotated method will run before each test method call
@AfterMethod: The annotated method will run after each test method call

Testng the life cycle of a method

@BeforeSuite (execute once)
@BeforeClass (execute once)
@BeforeMethod (N Test methods executed N times)
@Test Test method (this annotation may indicate multiple on the class, and one on the method)
@AfterMethod (N Test methods executed N times)
@AfterClass (execute once)
@AfterSuite (execute once)

Introduction to testng.xml

testng.xml is a file that records all tests in xml. It describes the runtime definition of the test suite and is also the largest unit of work for running tests in testng. Although there is no testng.xml file, the test is easy to execute. But with the growth of the test code, testng.xml provides a convenient way to store all runtime configurations, such as setting related classes, tests, methods, parameters, grouping inclusion and exclusion, etc.

The main structure of Testng.xml: The root tag is that the tag contains one or more tags, the tag contains one or more tags, and the tag contains one or more tags.

Suite: Represented by an XML file, which can contain one or more Tests, suite can run other testng xml files through tags
Test: Represents a test, which can contain one or more TestNG classes
TestNG class: is a simple Java class that contains at least one TestNG annotation
Method: An ordinary Java method, marked by @Test before it
Groups: TestNG can classify different Methods into different Groups, or classify Class into different Groups

Software Testing-TestNG must know and know

Guess you like