JaCoCo vs Cobertura performance and coverage results

We did recently migrated from Cobertura to JaCoCo as the test code coverage tool for our Spring batch Components. After some initial troubles with Powermock the tests were running FINE and coverage results were coming back as expected. The results looked good too. That was the end of that in terms of official tasks that needed to be done for this switch but I was still curious about any performance or coverage differences between the two tools, so I dug a little deeper...

I compared old Cobertura configuration for Maven and ran some builds with test code coverage enabled. I also ran the same tests with the new JaCoCo configuration. The numbers surprised me!

For these tests I disabled Cobertura's report aggregation since that's not being done with JaCoCo. Both tools were generating HTML and XML reports. JaCoCo was set up with offline instrumentation to ensure that Powermock enabled tests were picked up.

JaCoCo Timing

From Local Machine:

From Jenkins Server:

Cobertura Timing

From Local Machine:

From Jenkins Server:

That's quite the difference! Cobertura covered builds were taking more than twice the time to complete! This was not something I paid much attention to while we were still using that tool.On slower machines this difference in time would be even more profound.

So I choose JaCoCo in terms of performance, it was fast! Next I wanted to compare their coverage reports. Right off the bat it was obvious that each tool reported very different statistics and so it was hard to compare them...

Cobertura Coverage Info:

Jacoco Coverage Info:

Looking over the numbers it seemed that each tool was calculating the number of lines per class differently, which explained some variation. 

Cobertura is good tool but will not cover java 1.8 code properly. And its stopped with 2015 last version. It has issues in understanding lambda expressions.

[ERROR] net.sourceforge.cobertura.javancss.parser.ParseException: Encountered " ">" "> "" at line 6, column 54.
Was expecting one of:
    "assert" ...
    "boolean" ...
    "byte" ...
    "char" ...

Cobertura also skips classes which have import from java.util.stream package. For example, if the class imports java.util.stream.Stream, Cobertura will not generate coverage report for the class.

So which tool do I think is better? In terms of productivity I think I'd have to go with JaCoCo. In terms of setup, Cobertura wins. In terms of the generated reports - I like a bit of each one so they're on par here though I do like Cobertura's package layout (JavaDoc style). In terms of coverage - it looked like JaCoCo was showing more accurate results. In the end productivity and accuracy won since setup is a once-off activity and the reports from both tools show the same basic information. So JaCoCo is my choice then.

Setup: in pom.xml

<properties> <appName>App Name</appName> <jacoco.skip.instrument>false</jacoco.skip.instrument> <jacoco.version>0.8.4</jacoco.version></properties>

dependency:

<dependency> <!-- must be on the classpath --> <groupId>org.jacoco</groupId> <artifactId>org.jacoco.agent</artifactId> <classifier>runtime</classifier> <version>${jacoco.version}</version> <scope>test</scope></dependency>

Plugin:

<plugin> <groupId>org.jacoco</groupId> <artifactId>jacoco-maven-plugin</artifactId> <version>0.8.4</version> <configuration> <excludes> <exclude></exclude> </excludes> </configuration> <executions> <execution> <id>default-instrument</id> <goals> <goal>instrument</goal> </goals> </execution> <execution> <id>report</id> <phase>none</phase> </execution> <execution> <id>default-restore-instrumented-classes</id> <phase>prepare-package</phase> <goals> <goal>restore-instrumented-classes</goal> </goals> </execution> <execution> <id>post-unit-test</id> <phase>install</phase> <goals> <goal>report</goal> </goals> </execution> </executions></plugin>

<plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <configuration> <includes> <include>**/*.class</include> </includes> <systemPropertyVariables> <jacoco-agent.destfile>target/jacoco.exec</jacoco-agent.destfile> </systemPropertyVariables> </configuration></plugin>