endorphins endorphins - 3 months ago 13
Java Question

Getting average runtime of instances of test method using a TestNG Factory

I'm currently running a set of tests using a Factory. How can I record the time taken for each test, and then report the average time take for each test over every instance of the class. For example:

public class TestFactory {
@Factory
public Object[] create() {
return new Object[] {
new TestClass("1"), new TestClass("2")
};
}
}

public class TestClass {
private int x;

public TestClass(int i) {
x = i;
}

@Test(priority=1)
public void test1() {
System.out.println("test 1 : " + x + "!");
}

@Test(priority=2)
public void test2() {
System.out.println("test 2 : " + x + "!");
}
}


This would produce the output:

test 1 : 1! (assume this ran in 1 second)
test 2 : 1! (assume this ran in 2 seconds)
test 1 : 2! (assume this ran in 0.5 seconds)
test 2 : 2! (assume this ran in 1 second)


I know I can use @AfterMethod and ITestResult to calculate the runtime for each method in an instance and get the runtimes for both test1 and test2. But how would I calculate the average runtime of all instances that test1 and test2 ran so that I know that the average runtime for test1 was 0.75 seconds and for test2 it was 1.5 seconds?

Answer

You can use @org.testng.annotations.Listeners.

Implement suite listener

public class AverageReport implements ISuiteListener {
    @Override
    public void onStart(final ISuite iSuite) {
    }

    @Override
    public void onFinish(final ISuite iSuite) {
        // group runtimes per method
        Map<String, List<Double>> runtimes = new HashMap<>();
        for (IInvokedMethod method : iSuite.getAllInvokedMethods()) {
            ITestResult result = method.getTestResult();
            long runtime = result.getEndMillis() - result.getStartMillis();
            String methodName = method.getTestMethod().getMethodName();
            runtimes.computeIfAbsent(methodName, (name) -> new ArrayList<>())
                    .add((double) runtime);
        }

        // calculate averages and report
        averages.forEach((methodName, value) ->
            value.stream()
                 .mapToDouble(a -> a)
                 .average()
                 .ifPresent(avg -> 
                       // Put your reporting code here
                       System.err.println(
                           methodName + " average runtime is " + avg + "ms"
                       )
            ));
    }
}

Annotate your suite

And then you can use this annotation on your factory like this:

@Listeners(AverageReport.class)
public class TestFactory {
    @Factory
    public Object[] create() {
        return new Object[] {
            new TestClass(1), new TestClass(2)
        };
    }
}

After all tests will finsh you'll get an output like this

test2 average runtime is 31.0ms
test1 average runtime is 26.5ms
Comments