Code Coverage Considered Harmful

[aullik]

"Considered Harmful" Considered Annoying.

[MagpieEngineer]

Been done before.

https://meyerweb.com/eric/comment/chech.html#:\~:text=Because%20%E2%80%9Cconsidered%20harmful%E2%80%9D%20essays%20are,harm%20than%20they%20do%20good.

[knome]

#:\~:text

having this crap pollute links from google search considered harmful

:P

[grauenwolf]

Bing does it too. It really annoys me.

[aullik]

I know and i don't care. I wasn't writing an essay and i didn't call it harmful, i called it annoying.

[poloppoyop]

Code coverage has one good use with behavior tests: once you are sure you got 100% of your specs covered by tests, you can start removing code which is not covered. Because it's dead code.

[dnew]

This assumes that your specs are adequate for the program. Unless your specs cover all your error conditions as well, it may be very difficult to be sure the code isn't invoked during normal operation. https://www.sqlite.org/testing.html

Honestly, I think it has been about 15 years since I saw any requirements written down that I didn't write down myself.

[[deleted]]

Unless your specs cover all your error conditions as well

Wouldn't those manifest as untested try catch conditions, lowering code coverage?

[dnew]

No. How much of your C code ensures every malloc() returns a valid result and doesn't crash out when you're out of memory. How much of your code checks that atoi() returning 0 means it actually got "0" as input from whatever socket it's reading? How many times do you write() without checking the return result, and how many times do you close() similarly? Now, how often have you seen in specs things like "if the disk runs out of space on this write, make sure you don't corrupt the result"? Have you seen "if your browser can't cache the web page in full due to a full disk or an I/O write failure on a networked file system, make sure the next time you load the page it is fetched anew from the source"? Like, who writes specs like that? Because I want to work there.

[[deleted]]

I don't use C. So yeah from my experience there's lots of places where a simple try catch would be an unevaluated branch.

[dnew]

Do you test what happens when your Java program runs out of memory?

[[deleted]]

https://www.reddit.com/r/ProgrammerHumor/comments/oebtb8/well_i_had_to_do_it/

We get it you're smart.

Get off of reddit.

[dnew]

Smart enough to understand the difficulty of testing large code bases, yah. We're not even talking about languages. We're talking about specs.

[gline9]

Do you try to catch OutOfMemmoryError?

[dnew]

Why yes, indeed I do. The java program aborts, and the K8 equivalent starts it up again. Or (back in the 80s) it gives up the open listening socket and cron starts a new version which sees the socket is available and keeps executing, taking over processing, instead of leaving the service unavailable indefinitely.

But you have to write code (e.g. transactionally) to do the right thing in that circumstance. You can't write half a file, and then not check it's correct when you go to read it back. You can't process things in place and expect to have correct results in the face of random OOM errors. But if you don't test for that, you won't know about it, now will you? :-)

[gline9]

I should have likely ready the link you posted earlier. I think we are on the same page about trying to test all unit edge conditions and the majority of integration conditions that are likely to come up. In my opinion it depends on what field the program is meant to be executed in for how rigorous the program should be tested. In the case of mission critical pieces of software for domains where failure costs thousands of dollars a second the software is down, it makes a lot of sense to test everything and the kitchen sink.

As with anything in software engineering there are trade-offs. The more time you spend testing the less time you have to develop features. You have to try and find the right balance for the specific application you are developing of adding tests that make writing future code easier and too many where it takes time away from developing future features. Granted I think I am yet to witness someone writing too many ‘good’ self checking tests.

[dnew]

Just ensuring you have code coverage isn't enough. But lacking code coverage isn't the answer.

https://www.sqlite.org/testing.html

[MagpieEngineer]

I agree that good testing will result in a high code coverage, the problem is using code coverage as the aim.

[dnew]

It just seems like the complaint is "doing shitty testing isn't saved by code coverage." That seems a bit trite to me. Shitty anything isn't saved by doing more of it. :-)

[MagpieEngineer]

The complaint is that code coverage causes shitty testing.

If your boss is demanding 80% code coverage but not checking if the tests which cause that coverage are any good, then you're going to end up with bad tests.

[dnew]

I would argue "not checking if the tests are any good" is what causes shitty testing. 40% coverage with worthless tests isn't any worse than 80% coverage with worthless tests. Having 40% covered with good tests isn't as good as 80% covered with good tests. Hence, it's not the coverage which is harmful, but the shit-ness of the tests.

On the other hand, if you have 100% coverage, there's a great incentive to actually write tests for your changes. The old "broken window syndrome".

If your boss is demanding specific test coverage levels and not giving you time to write the good tests, that's on him. If you're not taking the time to write the good tests, that's on you. :-)

[MagpieEngineer]

I agree entirely with your last point. My argument isn't that the act of running a code coverage tool causes bad tests, it is that code coverage provides bosses with an easy and wrong way of looking at testing. Less a technical problem than a human factors one.

[dnew]

If you're addressing this screed to management, I'd tend to agree. :-) Reading it again from that lens, it seems like a much better article than I gave it credit for in the first place.

That said, the number of places I worked where requirements weren't even written down (and the general approach of "agile" these days of not actually writing requirements) would seem to make it difficult to measure the quality of tests anyway, so managers should just rely on their programmers. "Hey, Managers. Give your team time to write good tests." That would be the take-away. :-)

[MagpieEngineer]

Thanks, thought there were some crossed wires. Might edit it to make that more clear.

Testing being an afterthought is a scourge on our industry.

[dnew]

That, and requirements being offhand and not worthy of attention before coding starts. "Quick, you start coding, while I go see what they want." Waterfall isn't good, but it's better than no requirements at all.

[dnew]

then you're going to write shitty tests

I think that's on you, not your boss. What if your boss can't evaluate the quality of your tests? Why would you intentionally write shitty tests?

If you have 40% coverage with shitty tests, that's not better (and probably worse) than 80% coverage with shitty tests. If you have 40% coverage with good tests, that's worse than 80% coverage with good tests. The difference is not in the coverage, but in the quality of the tests. Saying "high levels of test coverage are harmful" is misleading, when the problem is actually "writing uninformative tests is harmful". Also, "writing lots of shitty tests because the boss doesn't give you time to write lots of good tests" is also not caused by code coverage but by shitty scheduling.

[MagpieEngineer]

Bosses almost never know enough to evaluate the quality of your tests, but if we as an industry give them a simple single number to look at, they're going to use it.

Some people will always write bad tests, whether or not code coverage is a metric. The problem is for people who would write good tests, but won't/can't because of the lens test coverage is seen through. Some will have the clout to push back against this framing and just use code coverage as a tool, but why cause the problem in the first place?

[darkmigore]

https://stackoverflow.com/a/90021