1. Welcome to TestNG

TestNG is a testing framework inspired from JUnit and NUnit but introducing some new functionalities that make it more powerful and easier to use, such as:

  • Annotations.

  • Run your tests in arbitrarily big thread pools with various policies available (all methods in their own thread, one thread per test class, etc…​).

  • Test that your code is multithread safe.

  • Flexible test configuration.

  • Support for data-driven testing (with @DataProvider).

  • Support for parameters.

  • Powerful execution model (no more TestSuite).

  • Supported by a variety of tools and plug-ins (Eclipse, IDEA, Maven, etc…​).

  • Embeds BeanShell for further flexibility.

  • Default JDK functions for runtime and logging (no dependencies).

  • Dependent methods for application server testing.

TestNG is designed to cover all categories of tests: unit, functional, end-to-end, integration, etc…​

I started TestNG out of frustration for some JUnit deficiencies which I have documented on my weblog here and here Reading these entries might give you a better idea of the goal I am trying to achieve with TestNG. You can also check out a quick overview of the main features and an article describing a very concrete example where the combined use of several TestNG’s features provides for a very intuitive and maintainable testing design.

Here is a very simple test:

package example1;

import org.testng.annotations.*;

public class SimpleTest {

    @BeforeClass
    public void setUp() {
        // code that will be invoked when this test is instantiated
    }

    @Test(groups = {"fast"})
    public void aFastTest() {
        System.out.println("Fast test");
    }

    @Test(groups = {"slow"})
    public void aSlowTest() {
        System.out.println("Slow test");
    }

}

The method setUp() will be invoked after the test class has been built and before any test method is run. In this example, we will be running the group fast, so aFastTest() will be invoked while aSlowTest() will be skipped.

Things to note:

  • No need to extend a class or implement an interface.

  • Even though the example above uses the JUnit conventions, our methods can be called any name you like, it’s the annotations that tell TestNG what they are.

  • A test method can belong to one or several groups.

Once you have compiled your test class into the build directory, you can invoke your test with the command line, an ant task (shown below) or an XML file:

<project default="test">

    <path id="cp">
        <pathelement location="lib/testng-testng-5.13.1.jar"/>
        <pathelement location="build"/>
    </path>

    <taskdef name="testng" classpathref="cp"
             classname="org.testng.TestNGAntTask"/>

    <target name="test">
        <testng classpathref="cp" groups="fast">
            <classfileset dir="build" includes="example1/*.class"/>
        </testng>
    </target>

</project>

Use ant to invoke it:

c:> ant
Buildfile: build.xml

test:
[testng] Fast test
[testng] ===============================================
[testng] Suite for Command line test
[testng] Total tests run: 1, Failures: 0, Skips: 0
[testng] ===============================================


BUILD SUCCESSFUL
Total time: 4 seconds

Then you can browse the result of your tests:

start test-output\index.html (on Windows)

1.1. Requirements

  • TestNG Upto v7.5: JDK 8.

  • TestNG v7.6.0 and above: JDK 11 or higher.

1.2. Mailing-lists

1.3. Locations of the projects

If you are interested in contributing to TestNG or one of the IDE plug-ins, you will find them in the following locations:

1.4. Bug reports

If you think you found a bug, here is how to report it:

  • Create a small project that will allow us to reproduce this bug. In most cases, one or two Java source files and a testng.xml file should be sufficient. Then you can either zip it and email it to the testng-dev mailing-list or make it available on an open source hosting site, such as github and email testng-dev so we know about it. Please make sure that this project is self contained so that we can build it right away (remove the dependencies on external or proprietary frameworks, etc…​).

  • If the bug you observed is on the Eclipse plug-in, make sure your sample project contains the .project and .classpath files.

  • File a bug.

For more information, you can either download TestNG, read the manual or browse the links at the top.

1.5. License

Apache 2.0

2. Download

2.1. Current Release Version

2.1.1. Maven

<dependency>
    <groupId>org.testng</groupId>
    <artifactId>testng</artifactId>
    <version>7.9.0</version>
    <scope>test</scope>
</dependency>

2.1.2. Gradle

dependencies {
    testCompile 'org.testng:testng:7.9.0'
}

2.1.3. Snapshots

TestNG automatically uploads snapshots to Sonatype which you can access by adding the following repository:

repositories {
    maven {
        url 'https://oss.sonatype.org/content/repositories/snapshots'
    }
}

2.1.4. Eclipse plug-in

See the related documentation.

2.1.5. Build TestNG from source code

TestNG is also hosted on GitHub, where you can download the source and build the distribution yourself:

$ git clone https://github.com/testng-team/testng.git
$ cd testng
$ ./gradlew build

You will then find the jar file in the testng/build/libs directory

Some useful tasks:

  • ./gradlew tasks - to see what all tasks are available

  • ./gradlew test - to kick off the tests (incremental build)

  • ./gradlew clean test - if you want to have a clean build.

  • ./gradlew autostyleApply - Applies code formatting steps to sourcecode in-place.

  • ./gradlew autostyleCheck - Checks that sourcecode satisfies formatting steps.

  • ./gradlew check - Runs the below checks:

    • "Applies code formatting steps to sourcecode in-place." and

    • "Checks that sourcecode satisfies formatting steps.".

2.1.6. Build the TestNG Eclipse Plugin from source code

TestNG Eclipse Plugin is hosted on GitHub, you can download the source code and build by ourselves.

3. TestNG Documentation

3.1. Introduction

TestNG is a testing framework designed to simplify a broad range of testing needs, from unit testing (testing a class in isolation of the others) to integration testing (testing entire systems made of several classes, several packages and even several external frameworks, such as application servers).

Writing a test is typically a three-step process:

  • Write the business logic of your test and insert TestNG annotations in your code.

  • Add the information about your test (e.g. the class name, the groups you wish to run, etc…​) in a testng.xml file or in build.xml.

  • Run TestNG.

You can find a quick example on the Welcome page.

The concepts used in this documentation are as follows:

  • A suite is represented by one XML file. It can contain one or more tests and is defined by the <suite> tag.

  • A test is represented by <test> and can contain one or more TestNG classes.

  • A TestNG class is a Java class that contains at least one TestNG annotation. It is represented by the <class> tag and can contain one or more test methods.

  • A test method is a Java method annotated by @Test in your source.

A TestNG test can be configured by @BeforeXXX and @AfterXXX annotations which allows to perform some Java logic before and after a certain point, these points being either of the items listed above.

The rest of this manual will explain the following:

  • A list of all the annotations with a brief explanation. This will give you an idea of the various functionalities offered by TestNG but you will probably want to consult the section dedicated to each of these annotations to learn the details.

  • A description of the testng.xml file, its syntax and what you can specify in it.

  • A detailed list of the various features and how to use them with a combination of annotations and testng.xml.

3.2. Annotations

Type

Annotations

Description

Annotation

  • @BeforeSuite

  • @AfterSuite

  • @BeforeTest

  • @AfterTest

  • @BeforeGroups

  • @AfterGroups

  • @BeforeClass

  • @AfterClass

  • @BeforeMethod

  • @AfterMethod

Configuration information for a TestNG class:

  • @BeforeSuite: The annotated method will be run before all tests in this suite have run.

  • @AfterSuite: The annotated method will be run after all tests in this suite have run.

  • @BeforeTest: The annotated method will be run before any test method belonging to the classes inside the <test> tag is run.

  • @AfterTest: The annotated method will be run after all the test methods belonging to the classes inside the <test> tag have run.

  • @BeforeGroups: The list of groups that this configuration method will run before. This method is guaranteed to run shortly before the first test method that belongs to any of these groups is invoked.

  • @AfterGroups: The list of groups that this configuration method will run after. This method is guaranteed to run shortly after the last test method that belongs to any of these groups is invoked.

  • @BeforeClass: The annotated method will be run before the first test method in the current class is invoked.

  • @AfterClass: The annotated method will be run after all the test methods in the current class have been run.

  • @BeforeMethod: The annotated method will be run before each test method.

  • @AfterMethod: The annotated method will be run after each test method.

Behaviour of annotations in superclass of a TestNG class

The annotations above will also be honored (inherited) when placed on a superclass of a TestNG class. This is useful for example to centralize test setup for multiple test classes in a common superclass.

In that case, TestNG guarantees that the "@Before" methods are executed in inheritance order (highest superclass first, then going down the inheritance chain), and the "@After" methods in reverse order (going up the inheritance chain).

Annotation Attributes

alwaysRun

For before methods (beforeSuite, beforeTest, beforeTestClass and beforeTestMethod, but not beforeGroups): If set to true, this configuration method will be run regardless of what groups it belongs to. For after methods (afterSuite, afterClass, …​): If set to true, this configuration method will be run even if one or more methods invoked previously failed or was skipped.

dependsOnGroups

The list of groups this method depends on.

dependsOnMethods

The list of methods this method depends on.

enabled

Whether methods on this class/method are enabled.

groups

The list of groups this class/method belongs to.

inheritGroups

If true, this method will belong to groups specified in the @Test annotation at the class level.

onlyForGroups

Only for @BeforeMethod and @AfterMethod. If specified, then this setup/teardown method will only be invoked if the corresponding test method belongs to one of the listed groups.

Annotation

@DataProvider

Marks a method as supplying data for a test method. The annotated method must return an Object[][] where each Object[] can be assigned the parameter list of the test method. The @Test method that wants to receive data from this DataProvider needs to use a dataProvider name equals to the name of this annotation.

Annotation Attributes

name

The name of this data provider. If it’s not supplied, the name of this data provider will automatically be set to the name of the method.

parallel

If set to true, tests generated using this data provider are run in parallel. Default value is false.

Annotation

@Factory

Marks a method as a factory that returns objects that will be used by TestNG as Test classes. The method must return Object[].

Annotation

@Listeners

Defines listeners on a test class.

Annotation Attributes

value

An array of classes that extend org.testng.ITestNGListener.

Annotation

@Parameters

Describes how to pass parameters to a @Test method.

Annotation Attributes

value

The list of variables used to fill the parameters of this method.

Annotation

@Test

Marks a class or a method as part of the test.

Annotation Attributes

alwaysRun

If set to true, this test method will always be run even if it depends on a method that failed.

dataProvider

The name of the data provider for this test method.

dataProviderClass

The class where to look for the data provider. If not specified, the data provider will be looked on the class of the current test method or one of its base classes. If this attribute is specified, the data provider method needs to be static on the specified class.

dependsOnGroups

The list of groups this method depends on.

dependsOnMethods

The list of methods this method depends on.

description

The description for this method.

enabled

Whether methods on this class/method are enabled.

expectedExceptions

The list of exceptions that a test method is expected to throw. If no exception or a different than one on this list is thrown, this test will be marked a failure.

groups

The list of groups this class/method belongs to.

invocationCount

The number of times this method should be invoked.

invocationTimeOut

The maximum number of milliseconds this test should take for the cumulated time of all the invocationcounts. This attribute will be ignored if invocationCount is not specified.

priority

The priority for this test method. Lower priorities will be scheduled first.

successPercentage

The percentage of success expected from this method

singleThreaded

If set to true, all the methods on this test class are guaranteed to run in the same thread, even if the tests are currently being run with parallel="methods". This attribute can only be used at the class level and it will be ignored if used at the method level. Note: this attribute used to be called sequential (now deprecated).

timeOut

The maximum number of milliseconds this test should take.

threadPoolSize

The size of the thread pool for this method. The method will be invoked from multiple threads as specified by invocationCount. NOTE: this attribute is ignored if invocationCount is not specified

3.3. testng.xml

You can invoke TestNG in several different ways:

  • With a testng.xml file

  • With ant

  • From the command line

This section describes the format of testng.xml (you will find documentation on ant and the command line below).

The current DTD for testng.xml can be found on the main Web site: testng-1.0.dtd. Here is an example testng.xml file:

<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">

<suite name="Suite1" verbose="1">
  <test name="Nopackage">
    <classes>
       <class name="NoPackageTest"/>
    </classes>
  </test>

  <test name="Regression1">
    <classes>
      <class name="test.sample.ParameterSample"/>
      <class name="test.sample.ParameterTest"/>
    </classes>
  </test>
</suite>

You can specify package names instead of class names:

<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">

<suite name="Suite1" verbose="1">
  <test name="Regression1">
    <packages>
      <package name="test.sample"/>
   </packages>
 </test>
</suite>

In this example, TestNG will look at all the classes in the package test.sample and will retain only classes that have TestNG annotations.

You can also specify groups and methods to be included and excluded:

<test name="Regression1">
  <groups>
    <run>
      <exclude name="brokenTests"/>
      <include name="checkinTests"/>
    </run>
  </groups>

  <classes>
    <class name="test.IndividualMethodsTest">
      <methods>
        <include name="testMethod"/>
      </methods>
    </class>
  </classes>
</test>

You can also define new groups inside testng.xml and specify additional details in attributes, such as whether to run the tests in parallel, how many threads to use, whether you are running JUnit tests, etc…​

By default, TestNG will run your tests in the order they are found in the XML file. If you want the classes and methods listed in this file to be run in an unpredictable order, set the preserve-order attribute to false.

<test name="Regression1" preserve-order="false">
  <classes>
    <class name="test.Test1">
      <methods>
        <include name="m1"/>
        <include name="m2"/>
      </methods>
    </class>
    <class name="test.Test2"/>
  </classes>
</test>

Please see the DTD for a complete list of the features, or read on.

3.4. Running TestNG

TestNG can be invoked in different ways:

This section only explains how to invoke TestNG from the command line. Please click on one of the links above if you are interested in one of the other ways.

Assuming that you have TestNG in your class path, the simplest way to invoke TestNG is as follows:

java org.testng.TestNG testng1.xml [testng2.xml testng3.xml ...]

You need to specify at least one XML file describing the TestNG suite you are trying to run. Additionally, the following command-line switches are available:

3.4.1. Command Line Parameters

Option Argument Documentation

-configfailurepolicy

skip,continue

Whether TestNG should keep attempting @Before*/@After* methods after one of them has failed once, or skip the remaining ones. Default behavior is skip. With continue, a configuration failure invalidates only the narrowest scope it belongs to — the failing instance for @BeforeClass, the failing test method invocation for @BeforeMethod — so sibling instances and sibling test methods still get their configuration methods invoked. A test method whose own setup failed is still skipped, with one exception: a failed @BeforeTest invalidates no instance at all, so under continue the test methods of that <test> do run. A @BeforeSuite failure always stops the suite regardless of this setting. In every case the configuration failure is reported, so continue never turns a broken setup into a green run.

-d

A directory

The directory where the reports will be generated (defaults to test-output).

-dataproviderthreadcount

The default number of threads to use for data providers when running tests in parallel.

This sets the default maximum number of threads to use for data providers when running tests in parallel. It will only take effect if the parallel mode has been selected (for example, with the -parallel option). This can be overridden in the suite definition.

-excludegroups

A comma-separated list of groups.

The list of groups you want to be excluded from this run.

-groups

A comma-separated list of groups.

The list of groups you want to run (e.g. "windows,linux,regression").

-listener

A comma-separated list of Java classes that can be found on your classpath.

Lets you specify your own test listeners. The classes need to implement org.testng.ITestListener

-usedefaultlisteners

true,false

Whether to use the default listeners

-methods

A comma separated list of fully qualified class name and method. For example com.example.Foo.f1,com.example.Bar.f2.

Lets you specify individual methods to run.

-methodselectors

A comma-separated list of Java classes and method priorities that define method selectors.

Lets you specify method selectors on the command line. For example: com.example.Selector1:3,com.example.Selector2:2

-parallel

methods,tests,classes

If specified, sets the default mechanism used to determine how to use parallel threads when running tests. If not set, default mechanism is not to use parallel threads at all. This can be overridden in the suite definition.

-reporter

The extended configuration for a custom report listener.

Similar to the -listener option, except that it allows the configuration of JavaBeans-style properties on the reporter instance. Example: -reporter com.test.MyReporter:methodFilter=insert,enableFiltering=true You can have as many occurrences of this option, one for each reporter that needs to be added.

-sourcedir

A semicolon separated list of directories.

The directories where your javadoc annotated test sources are. This option is only necessary if you are using javadoc type annotations. (e.g. src/test or src/test/org/testng/eclipse-plugin;src/test/org/testng/testng).

-suitename

The default name to use for a test suite.

This specifies the suite name for a test suite defined on the command line. This option is ignored if the suite.xml file or the source code specifies a different suite name. It is possible to create a suite name with spaces in it if you surround it with double-quotes "like this".

-testclass

A comma-separated list of classes that can be found in your classpath.

A list of class files separated by commas (e.g. org.foo.Test1,org.foo.test2).

-testjar

A jar file.

Specifies a jar file that contains test classes. If a testng.xml file is found at the root of that jar file, it will be used, otherwise, all the test classes found in this jar file will be considered test classes.

-testname

The default name to use for a test.

This specifies the name for a test defined on the command line. This option is ignored if the suite.xml file or the source code specifies a different test name. It is possible to create a test name with spaces in it if you surround it with double-quotes "like this".

-testnames

A comma separated list of test names.

Only tests defined in a <test> tag matching one of these names will be run.