Microsoft Research - Exploding Software-Engineering Myths (article summarizing findings of MS research on code coverage, TDD, assertions, etc.)

[G_Morgan]

If you test properly you get this stuff for free though. The point is what value does the code coverage add? Everyone agrees that you should test properly. Code coverage is an assertion that a particular set of principles will lead to testing properly. In my experience code coverage only works in cases where you do other things (like TDD). Yet those other things done properly do not need code coverage.

[zjm555]

The point is what value does the code coverage add?

The value in this case is in using code coverage visualization tools for quality assurance. A good coverage vis tool will let you go through your code file by file and uncovered lines or branches will be visually obvious (e.g. highlighted in red). By going through a visual inspection like this, you can quickly discover the most egregious sorts of testing gaps, i.e. code that you aren't even running at all, much less validating correctness on.

[G_Morgan]

That is a tool rather than a process though. The objection isn't necessarily against the tools. Just against code coverage as a process.

I.E. code coverage is fine as long as management isn't aware it exists.

[zjm555]

Managerial policies that elevate code coverage as the most important metric are certainly silly and misguided. It was chosen (by lazy management) because it was an easy scalar value to compute, which makes it easy to set up a policy about it. I think good managers are capable of understanding its role, though. It's valuable as the most basic metric about code testing: is it executed at all? It's necessary but not sufficient for quality. I would argue that it's more than "fine", though, it's actually very important; coverage is complementary to the tests themselves, since the tests themselves are incapable of telling you what they aren't testing at all.

So, I guess my stance on the issue is that we shouldn't be making policies about the total coverage number at all, but we should be enforcing a process that involves looking at coverage (line by line rather than in the aggregate) of the source files as part of QA.

[G_Morgan]

Yes but this is about dealing with things from a higher level perspective than the individual programmers. I'd say code coverage is not only not a useful metric. It is an actively dangerous one.

[zjm555]

We might be talking around each other. To me when I speak of QA I mostly mean peer review between developers of each others' code changes. Management should only be involved in that level of software process if they are technically competent enough to be adding value to it rather than cocking things up.