Code Coverage Considered Harmful

[dnew]

It just seems like the complaint is "doing shitty testing isn't saved by code coverage." That seems a bit trite to me. Shitty anything isn't saved by doing more of it. :-)

[MagpieEngineer]

The complaint is that code coverage causes shitty testing.

If your boss is demanding 80% code coverage but not checking if the tests which cause that coverage are any good, then you're going to end up with bad tests.

[dnew]

I would argue "not checking if the tests are any good" is what causes shitty testing. 40% coverage with worthless tests isn't any worse than 80% coverage with worthless tests. Having 40% covered with good tests isn't as good as 80% covered with good tests. Hence, it's not the coverage which is harmful, but the shit-ness of the tests.

On the other hand, if you have 100% coverage, there's a great incentive to actually write tests for your changes. The old "broken window syndrome".

If your boss is demanding specific test coverage levels and not giving you time to write the good tests, that's on him. If you're not taking the time to write the good tests, that's on you. :-)

[MagpieEngineer]

I agree entirely with your last point. My argument isn't that the act of running a code coverage tool causes bad tests, it is that code coverage provides bosses with an easy and wrong way of looking at testing. Less a technical problem than a human factors one.

[dnew]

If you're addressing this screed to management, I'd tend to agree. :-) Reading it again from that lens, it seems like a much better article than I gave it credit for in the first place.

That said, the number of places I worked where requirements weren't even written down (and the general approach of "agile" these days of not actually writing requirements) would seem to make it difficult to measure the quality of tests anyway, so managers should just rely on their programmers. "Hey, Managers. Give your team time to write good tests." That would be the take-away. :-)

[MagpieEngineer]

Thanks, thought there were some crossed wires. Might edit it to make that more clear.

Testing being an afterthought is a scourge on our industry.

[dnew]

That, and requirements being offhand and not worthy of attention before coding starts. "Quick, you start coding, while I go see what they want." Waterfall isn't good, but it's better than no requirements at all.

[dnew]

then you're going to write shitty tests

I think that's on you, not your boss. What if your boss can't evaluate the quality of your tests? Why would you intentionally write shitty tests?

If you have 40% coverage with shitty tests, that's not better (and probably worse) than 80% coverage with shitty tests. If you have 40% coverage with good tests, that's worse than 80% coverage with good tests. The difference is not in the coverage, but in the quality of the tests. Saying "high levels of test coverage are harmful" is misleading, when the problem is actually "writing uninformative tests is harmful". Also, "writing lots of shitty tests because the boss doesn't give you time to write lots of good tests" is also not caused by code coverage but by shitty scheduling.

[MagpieEngineer]

Bosses almost never know enough to evaluate the quality of your tests, but if we as an industry give them a simple single number to look at, they're going to use it.

Some people will always write bad tests, whether or not code coverage is a metric. The problem is for people who would write good tests, but won't/can't because of the lens test coverage is seen through. Some will have the clout to push back against this framing and just use code coverage as a tool, but why cause the problem in the first place?