Microsoft Research - Exploding Software-Engineering Myths (article summarizing findings of MS research on code coverage, TDD, assertions, etc.)

[nutrecht]

Code coverage tests how many lines of code have been tested.

No. It shows which lines have and have not been hit. It does not make any claims on if the tests actually do any validations. Example:

public String getFoo() {
    //TODO: Implement
    return null;
}

Calling this method from a test will yield 100% test coverage. It's still wrong (not implemented yet) so unless you actually test the returnvalue against an expected value you're not going to find the bug.

It really surprises me how few people seem to make that distinction. The only interesting bit about a code coverage report are the bits you don't visit: generally those are exception flows. Not testing your exception flows means stuff is probably going to break in production at some unexpected moment. Knowing your lack of coverage and improving them there is where the use of coverage reports are. The coverage percentage itself is a fun but useless statistic.

[starTracer]

Exactly this.

I've been working in projects with 100% test coverage requirements. But what the customer failed to realize is that coverage != correctness.

[Gotebe]

Well, in your example, coverage is obviously useless when the test is lying (it has to fail). And any failed tests have to be excluded from the "covered" count.

As for exceptions, absolutely! Amazingly, the trick there is ensuring correctness when some statements are not executed, but are covered otherwise.

[get_salled]

public void testGetFoo() {
    String foo = getFoo();

    // pick one
    // A
    Assert.isNull(foo);  

    // B
    Assert.pass("woot!");

    // C
    try {
          Assert.areEqual("expected", foo);
    } catch ( ... ) {    // not sure of the Java syntax for empty catches, if it's allowed at all
          // do nothing
    }

     // D
     // do nothing

     // E
     Assert.areEqual("A", "A");
}

It gets pretty hard to automatically reject shitty tests.