The State of Code Coverage for Groovy

Last year we finally decided to update from JDK6 to JDK7. This was mostly a painless process, however when we upgraded, we started getting very strange code coverage numbers. We were using cobertura 2.0.3 at the time. I also created an issue at the time, and more recently a second issue was added.

Recently we started looking in to getting our coverage statistics working again so I’ve done a bit more digging with John Engleman. We’ve tried jacoco 0.7.0.201403182114, cobertura 2.0.3 and clover 3.1.12.1.

I wanted to do an apples to apples comparison of jacoco and cobertura for JDK6 and JDK7 on a simple groovy project.

The project has one class:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
class CodeCoverageExample {
  def usedMethod(def a) {
      if (a) {
          dispatchToPrivateMethod()
      } else {
          dispatchToPrivateMethod2()
      }
  }

  def unusedMethod(def a) {
      if (a) {
          dispatchToPrivateMethod()
      }
  }

  private def dispatchToPrivateMethod() {
      1
  }

  private def dispatchToPrivateMethod2() {
      2
  }

}

It also has one test:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
import spock.lang.Specification

class CodeCoverageExampleSpec extends Specification {
  def "calls usedMethod"() {
      setup:
      CodeCoverageExample cce = new CodeCoverageExample()

      expect:
      result == cce.usedMethod(givenValue)

      where:

      result | givenValue
      1      | true
      2      | false
  }
}

Here are some statisics:

JDK Version Coverage Tool LOC covered Branches covered Comments
6 Cobertura 71% 25% This seems pretty legit to me.
7 Cobertura 42% 12% This is so broken. It didn’t count any line in the private methods, and also didn’t count a line hit inside the else branch.
6 jacoco 50% 21% Jacoco is saying 50% of instructions were executed but no lines of code had 100% of their instructions executed. I don’t know how to determine what instructions were missed and if they are important.
7 jacoco 50% 21% Hurray consistency!
6 clover 78% 69% I had to calculate these percentages by hand using the XML data.
7 clover 78% 69% Hurray consistency! However Clover doesn’t work on our non-trivial codebase and errors on classes with @CompileStatic. I couldn’t reproduce this in my trivial example however.

Honestly none of these are perfect. Clearly cobertura doesn’t work for groovy on the jdk7. The instrumented code appears correct to my eyes but I am (obviously) not an expert. I don’t really like the jacoco instructions measure because I don’t understand where some of the missing instructions are and if they are important. Clover isn’t opensource, is fairly expensive, and also doesn’t work on our real applications.

Jacoco seems to be the best option right now, but I’m hoping the cobertura defects are fixed soon.

Comments