How to Run Grouped TestNG Tests Using Gradle

By Gaurav Singh

Hello!

When building a test framework, one of the most crucial decisions is the choice of the test framework in your language ecosystem. For instance, in the Kotlin/Java world, we could choose JUnit, TestNG, Cucumber or pure Kotlin frameworks like kotlintest and spek. Each of these frameworks offer some basic constructs to achieve similar results along with their own implementation idiosyncrasies.

At GOJEK, I chose to work with TestNG since it is fairly mature and many devs/testers who work with JVM languages are already aware of its features. This also results in a lower learning curve.

Once you have wired up some tests and ensured they work fine inside the IDE (IntelliJ in this case) the next logical step is obviously to promote them into your CI environment and run via Gradle.

While doing so, I faced a small hiccup.

This post explains how I debugged and finally arrived at a solution.

Let’s start at the beginning:

Test Grouping

An often used feature of any test framework is their support for grouping similar tests together and providing the ability to run this subset of tests at will.

Let’s take an example to understand this a bit better

Lets say we need to test a Person class having name and age properties such that we expect the Person to have certain fixed name and age. Also we would want to run all the cases which belong to person_test group.

PersonTest.kt

import org.testng.Assert
import org.testng.annotations.AfterMethod
import org.testng.annotations.BeforeMethod
import org.testng.annotations.Test
class PersonTest {
    private var name = "Rob"
    private var age = 23
    @BeforeMethod()
    fun before() {
        println("Performing setup...")
        name = "John"
        age = 25
    }
    @Test(groups = ["person_test"])
    fun personNameTest() {
        Assert.assertEquals("John", name)
    }
    @Test(groups = ["person_test"])
    fun personAgeTest() {
        Assert.assertEquals(25, age)
    }
    @AfterMethod()
    fun after() {
        println("Performing teardown...")
    }
}

In TestNG, we can pass a list of group names to the Test annotation to uniquely identify them under this common term.

@Test(groups = ["person_test"])

The next step is to setup a simple build.gradle file which should be able to run these tests. We can follow a structure like shown below:

Note: This config sets up your IntelliJ project to work with Kotlin

plugins {
    id 'org.jetbrains.kotlin.jvm' version '1.3.31'
}
group 'com.example.testnggradle'
version '1.0-SNAPSHOT'
repositories {
    mavenCentral()
}
tasks.withType(Test) {
    systemProperties = [
            tag: System.getProperty('tag', 'person_test')
    ]
}
task runTests(type: Test) {
    useTestNG() {
        testLogging.showStandardStreams = true
        includeGroups System.getProperty('tag', 'NONE')
    }
}
dependencies {
    implementation "org.jetbrains.kotlin:kotlin-stdlib-jdk8"
    compile group: 'org.testng', name: 'testng', version: '6.14.3'
}
compileKotlin {
    kotlinOptions.jvmTarget = "1.8"
}
compileTestKotlin {
    kotlinOptions.jvmTarget = "1.8"
}

A few more things to note here are:

While running the runTests Gradle task, we would want to pass a command line flag as -Dtag to be able to run the tests belonging to the group that we want.

We achieve this by adding this line in useTestNG() method inside our task:

includeGroups System.getProperty('tag', 'NONE')

We are setting this as defaulted to person_test in order to ensure that all our tests are run. This could be set to a sensible default like regression in case that is what you use to tag all your cases.

tasks.withType(Test) {
    systemProperties = [
            tag: System.getProperty('tag', 'person_test')
    ]
}

Let’s try to run this via command line to check that this would work fine when we push this as a command line job on some CI.

/gradlew clean runTests -Dtag=person_test

Aaannnd, here’s our result:

> Task :runTests FAILED

Gradle suite > Gradle test > PersonTest.personAgeTest FAILED
    java.lang.AssertionError at PersonTest.kt:24

Gradle suite > Gradle test > PersonTest.personNameTest FAILED
    java.lang.AssertionError at PersonTest.kt:19

2 tests completed, 2 failed

FAILURE: Build failed with an exception.

Wait a minute. Why did the it fail? If you go back and see PersonTest.kt, you can see the test should pass since we are setting up the correct value for person’s name and age in the@BeforeMethod annotation.

So what went wrong here?

Let’s get into investigation mode and run the same command with --infoCMD line switch to get more info.

./gradlew clean runTests -Dtag=person_test --info

Hmm. 🤔 We can see the actual value for name and age is still the initialization value and also the println method’s content is not printed. This is basically a hint that the setup/teardown methods are not getting executed, causing these failures.

Gradle suite > Gradle test > PersonTest.personAgeTest FAILED
    java.lang.AssertionError: expected [23] but found [25]
        at org.testng.Assert.fail(Assert.java:96)
        at org.testng.Assert.failNotEquals(Assert.java:776)
        at org.testng.Assert.assertEqualsImpl(Assert.java:137)
        at org.testng.Assert.assertEquals(Assert.java:118)
        at org.testng.Assert.assertEquals(Assert.java:652)
        at org.testng.Assert.assertEquals(Assert.java:662)
        at PersonTest.personAgeTest(PersonTest.kt:24)
Gradle suite > Gradle test > PersonTest.personNameTest FAILED
    java.lang.AssertionError: expected [Rob] but found [John]
        at org.testng.Assert.fail(Assert.java:96)
        at org.testng.Assert.failNotEquals(Assert.java:776)
        at org.testng.Assert.assertEqualsImpl(Assert.java:137)
        at org.testng.Assert.assertEquals(Assert.java:118)
        at org.testng.Assert.assertEquals(Assert.java:453)
        at org.testng.Assert.assertEquals(Assert.java:463)
        at PersonTest.personNameTest(PersonTest.kt:19)
2 tests completed, 2 failed

This was weird. Initially, I couldn’t comprehend why TestNG and Gradle were behaving this way. 🤷🏻‍♂️

After poring over multiple discussions on GitHub, Stack Overflow and Gradle docs, I finally found a solution to achieve the expected behaviour.

In order for setup/teardown methods to work with the above config we need to add alwaysRun = true in them.

If we follow TestNG documentation for alwaysRun we can see as below:

For before methods (beforeSuite, beforeTest, beforeTestClass and beforeTestMethod, but not beforeGroups): If set to true, this configuration method will be run regardless of what groups it belongs to.
For after methods (afterSuite, afterClass, …): If set to true, this configuration method will be run even if one or more methods invoked previously failed or was skipped.

Here is the final code with the changes.

import org.testng.Assert
import org.testng.annotations.AfterMethod
import org.testng.annotations.BeforeMethod
import org.testng.annotations.Test
class PersonTest {
    private var name = "Rob"
    private var age = 23
    @BeforeMethod(alwaysRun = true)
    fun before() {
        println("Performing setup...")
        name = "John"
        age = 25
    }
    @Test(groups = ["person_test"])
    fun personNameTest() {
        Assert.assertEquals("John", name)
    }
    @Test(groups = ["person_test"])
    fun personAgeTest() {
        Assert.assertEquals(25, age)
    }
    @AfterMethod(alwaysRun = true)
    fun after() {
        println("Performing teardown...")
    }
}

Sure enough, Gradle build and test pass (validated by the debugging messages printed from setup and teardown methods).

Gradle suite > Gradle test > PersonTest STANDARD_OUT
    Performing setup...
Gradle suite > Gradle test STANDARD_OUT
    Performing teardown...
    Performing setup...
    Performing teardown...
Finished generating test XML results (0.0 secs) into: .../build/test-results/runTests
Generating HTML test report...
Finished generating test html results (0.003 secs) into: .../build/reports/tests/runTests
:runTests (Thread[Task worker for ':',5,main]) completed. Took 0.527 secs.
BUILD SUCCESSFUL in 1s
3 actionable tasks: 3 executed

If you have ever been frustrated by the same problem, hopefully this post might help you save some time.

Here are some of the resources that helped me arrive at this solution:

• Gradle docs — Test grouping
• GitHub

If you liked this or feel someone else can also benefit from this, do share it along. Until next time. Happy coding! 🖖

Psst! We’re hiring!

It’s been barely four years since the GOJEK app went live, but we’ve not been sitting around. Our ecosystem has grown to 19+ products, which account for over 100 million transactions a month. We’re still shooting for the stars, and we need the best minds to steer our rocket ship. If that’s your kind of jam, gojek.jobs is where you sign up for the ride.

See you on the other side 🙌