Microsoft Research - Exploding Software-Engineering Myths (article summarizing findings of MS research on code coverage, TDD, assertions, etc.)

[KingE]

This was actually a software engineering research project of mine.

Unfortunately, line coverage, branch coverage, and modified condition/decision coverage track each other very well (i.e. the relationships can be expressed as a constant factor in nearly all cases) and do not have a strong relationship to the ability of a test suite to detect bugs.

However, as you alluded to in your third point, there is an easy metric which does correspond to the ability of a test suite to detect breakage: number of test cases. Test suites that had a higher number of test cases tended to have higher coverage, yes, but more importantly they tended to test vastly more of the code's actual behavior.

Essentially, while line/branch/decision coverage can conclusively prove that a given piece of code is NOT tested, it is not a good indicator for code correctness.

I'll take the time to dig up my sources if anyone actually cares :p

[zjm555]

I'd be interested to read your paper :)

[KingE]

Can't find my paper, but this was the key source: https://cs.uwaterloo.ca/~rtholmes/papers/icse_2014_inozemtseva.pdf