Java automation testing framework -10 - TestNG's test results published

1.- Test Results

1.1 success, failure and assertions

Test is considered successful if it does not throw any exceptions complete, it is still expected to throw an exception (please see the comments on the document expectedExceptions @Test properties found).

Your test methods typically call throws an exception or assertion by the various possible (using Java "assert" keyword) components. "Assertion" failed to trigger AssertionErrorException, which in turn will mark methods fail (if you do not see an assertion error, please remember to use -ea on the JVM).

This is a sample test method:

/ ** 
 * @author Beijing - Hong Columbia 
 * 
 * the Java automation testing framework -10 - TestNG's test results published 
 * 
 * November 9, 2019 
 * / 
@Test 
public  void verifyLastName () {
   the Assert "Beust" .equals (m_lastName ): "Beust the Expected name, for" + m_lastName; 
}

TestNG also includes the JUnit Assert class, the class allows you to perform complex object assertions:

/ ** 
 * @author Beijing - Hong Columbia 
 * 
 * the Java automation testing framework -10 - TestNG's test results published 
 * 
 * November 9, 2019 
 * / 
Import  static . Org.testng.AssertJUnit * ;
 // ... 
@ the Test
 public  void Verify () { 
  the assertEquals ( "Beust" , m_lastName); 
}

Note that the above code uses a static introduction, to be able to without having to add a prefix before its use assertEquals class method.

1.2 records and results

Index.html file in the directory specified by the results of tests run at startup SuiteRunner created. This file contains the entire operating results point to test various other HTML and text files.

With TestNG with listeners and reporters to generate their own reports very easily:

Listener realize org.testng.ITestListener interface and real-time notification when the beginning of the test, passed, failed and so on.

Reporting procedures to achieve org.testng.IReporter interface and receive notification when TestNG has run all packages. Receiving a list of objects described in Examples IReporter entire test run.

For example, if you want to generate a test run of the PDF report, you do not need real-time notification test run, so you should use IReporter. If you want to write real-time reporting of tests, such as GUI or with a progress bar is displayed each time the test is called the point ( ".") The text reporting procedures (as described below), then ITestListener is your best choice .

1.2.1 Record listener

This is a show. "" Listener. For each test passed, for every failure, we are "F", for each skipped are "S":

/ ** 
 * @author Beijing - Hong Columbia 
 * 
 * the Java automation testing framework -10 - TestNG's test results published 
 * 
 * November 9, 2019 
 * / 
public  class DotTestListener the extends TestListenerAdapter {
   Private  int m_count = 0 ; 
 
  @Override 
  public  void onTestFailure (ITestResult TR) { 
    log ( "F." ); 
  } 
 
  @Override 
  public  void onTestSkipped (ITestResult TR) { 
    log ( "S" ); 
  } 
 
  @Override 
  public  void  onTestSuccess (ITestResult TR) {
    (log "." );
  }
 
  private void log(String string) {
    System.out.print(string);
    if (++m_count % 40 == 0) {
      System.out.println("");
    }
  }
}

In this example, I chose to extend TestListenerAdapter, the method uses empty method implementation ITestListener, so I do not have to cover other methods from the interface I'm not interested in. You can directly implement this interface as needed.

This is the method I call TestNG to use this new listener:

java -classpath testng.jar;%CLASSPATH% org.testng.TestNG -listener org.testng.reporters.DotTestListener test\testng.xml

And output:

........................................
........................................
........................................
........................................
........................................
.........................
===============================================
TestNG JDK 1.5
Total tests run: 226, Failures: 0, Skips: 0
===============================================

Note that when you use -listener, TestNG will automatically determine the type of listener you want to use.

1.2.2 Record Reporter

The org.testng.IReporter interface has a single method:

public void generateReport(List<ISuite> suites, String outputDirectory)

When all packages have been run, TestNG will call this method, you can check its parameters in order to access all the information just completed running.

1.2.3-JUnitReports

TestNG contains a listener, the listener obtains and outputs a result TestNG XML file, then it may be fed to JUnitReport. This is an example, and create ant task this report:

<target name="reports">
  <junitreport todir="test-report">
    <fileset dir="test-output">
      <include name="*/*.xml"/>
    </fileset>
  
    <report format="noframes"  todir="test-report"/>
  </junitreport>
</target>

Note: JDK 1.5 and JUnitReports not currently compatible, can not use the version of the framework, so you need to specify the "noframes" in order to make it work.

1.2.4-Reporter API

If you need to record a message to be displayed in the generated HTML report, you can use org.testng.Reporter categories:

Reporter.log ( "have called M3");

1.2.5-XML report

TestNG provides an XML reporting procedures, TestNG JUnit report specific information is not available for capture. When the user's test environment requires the use of XML results have TestNG JUnit format specific data not available, this feature is particularly useful. Reporters can -reporter TestNG by using the command injection.

This is an example of usage: -reporter org.testng.reporters.XMLReporter: generateTestResultAttributes = true, generateGroupsAttribute = true.

The following table details all the options that can be passed. Be sure to use:

: - the name of the reporter and their property separately

= - partition key attribute / value pairs

, - partitioning the plurality of key / value pairs

The following is an example of the output of such a reporter:

<testng-results>
  <suite name="Suite1">
    <groups>
      <group name="group1">
        <method signature="com.test.TestOne.test2()" name="test2" class="com.test.TestOne"/>
        <method signature="com.test.TestOne.test1()" name="test1" class="com.test.TestOne"/>
      </group>
      <group name="group2">
        <method signature="com.test.TestOne.test2()" name="test2" class="com.test.TestOne"/>
      </group>
    </groups>
    <test name="test1">
      <class name="com.test.TestOne">
        <test-method status="FAIL" signature="test1()" name="test1" duration-ms="0"
              started-at="2007-05-28T12:14:37Z" description="someDescription2"
              finished-at="2007-05-28T12:14:37Z">
          <exception class="java.lang.AssertionError">
            <short-stacktrace>
              <![CDATA[
                java.lang.AssertionError
                ... Removed 22 stack frames
              ]]>
            </short-stacktrace>
          </exception>
        </test-method>
        <test-method status="PASS" signature="test2()" name="test2" duration-ms="0"
              started-at="2007-05-28T12:14:37Z"= "someDescription1"description
              finished-at="2007-05-28T12:14:37Z">
        </test-method>
        <test-method status="PASS" signature="setUp()" name="setUp" is-config="true" duration-ms="15"
              started-at="2007-05-28T12:14:37Z" finished-at="2007-05-28T12:14:37Z">
        </test-method>
      </class>
    </test>
  </suite>
</testng-results>

The report program injected with other default listener, so by default you can get this type of output. Listener provides a number of properties that can be adjusted reporter to meet your needs. The following table contains a list of these attributes, along with a brief description:

Property	Comment	Default value
outputDirectory	A `String` indicating the directory where should the XML files be output.	The TestNG output directory
timestampFormat	Specifies the format of date fields that are generated by this reporter	yyyy-MM-dd'T'HH:mm:ss'Z'
fileFragmentationLevel	An integer having the values 1, 2 or 3, indicating the way that the XML files are generated: 1 - will generate all the results in one file. 2 - each suite is generated in a separate XML file that is linked to the main file. 3 - same as 2 plus separate files for test-cases that are referenced from the suite files.	1
splitClassAndPackageNames	This boolean specifies the way that class names are generated for the `<class>` element. For example, you will get `<class class="com.test.MyTest">` for false and `<class class="MyTest" package="com.test">` for true.	false
generateGroupsAttribute	A boolean indicating if a `groups` attribute should be generated for the `<test-method>` element. This feature aims at providing a straight-forward method of retrieving the groups that include a test method without having to surf through the `<group>` elements.	false
generateTestResultAttributes	A boolean indicating if an `<attributes>` tag should be generated for each `<test-method>` element, containing the test result attributes (See `ITestResult.setAttribute()` about setting test result attributes). Each attribute `toString()` representation will be written in a `<attribute name="[attribute name]">` tag.	false
stackTraceOutputMethod	Specifies the type of stack trace that is to be generated for exceptions and has the following values: 0 - no stacktrace (just Exception class and message). 1 - a short version of the stack trace keeping just a few lines from the top 2 - the complete stacktrace with all the inner exceptions 3 - both short and long stacktrace	2
generateDependsOnMethods	Use this attribute to enable/disable the generation of a `depends-on-methods` attribute for the `<test-method>` element.	true
generateDependsOnGroups	Enable/disable the generation of a `depends-on-groups` attribute for the `<test-method>` element.	true

To configure this report, you can use the command line -reporter option, you can also Ant tasks and nested <reporter> used with the elements. For each of them, you must specify org.testng.reporters.XMLReporter class. Please note that you can not configure the built-in reports is because the report only uses the default settings. If you only need

XML reports with custom settings, you must use one of two methods to manually add it and disable the default listener.

1.2.6-TestNG exit code

When TestNG finished executing, it will exit and return code.

You can check this return code to understand the nature of the fault (if any).

The following table summarizes the different exit code TestNG currently in use.

FailedWithinSuccess	Skipped	Failed	Status Code	Remarks
No	No	No	0	Passed tests
No	No	Yes	1	Failed tests
No	Yes	No	2	Skipped tests
No	Yes	Yes	3	Skipped/Failed tests
Yes	No	No	4	FailedWithinSuccess tests
Yes	No	Yes	5	FailedWithinSuccess/Failed tests
Yes	Yes	No	6	FailedWithinSuccess/Skipped tests
Yes	Yes	Yes	7	FailedWithinSuccess/Skipped/Failed tests

2.- Summary

Well, today on the results of TestNG, we share here.