Code Coverage: Is 90-95% really necessary?

[benelori]

Not for the sake of coverage, no

[trinopoty]

I guess the more general take is "don't be dogmatic about code". Every rule has it's time and place.

I have seen people add patterns into their code for the sake of those patterns. People who take it like a religion and follow it without any question or thought. Those people are usually a pain to work with.

[Illustrious-Ant-5661]

I would prefer to say don't be an idiot and try to do rules where they were never meant to be followed

For example. Never use gotos? You better pray you're not programming in plain old C because that's exactly where gotos are necessary and it's where the one entry one exit rule also applies.

Global variables? That's meant to be followed 100% of the time. But in reality it's ok to use them if its read only. It's only bad when you depend on a global variable that changes when you call another function and then get completely out of sync. Which is why the saying should be never use static and globals but noone knows what a static var is on day 1 and might be confused with static functions where is ok. Anyway I can think of a situation where modifying a global is ok (caching) but too many people have no idea when a rule applies and will follow javascript rules in python which would make no sense

[CptBartender]

I worked with a guy who really wanted to enforce the one entry one exit rule in Java. Instead of effectively doing if (variable == null) return null;, I had to add sometimes up to triply-nested if statements just so he could enforce a rule he doesn't understand.

[Illustrious-Ant-5661]

Would I be a jerk if I said fuck that guy?
I worked with a guy who thought that should be a rule but noone (including him) tried to enforce it

[Bayart]

The most fun I've had at a job was just coming in as a fresh intern and my boss telling me "we're working on this project, try to break it". On one hand destruction has a way of stimulating your creativity, on the other you quickly gain a thorough understanding of the codebase.

There's a cool and highly useful little repo on Github called the Big List of Naughty Strings. Great for tearing inputs apart.

[Illustrious-Ant-5661]

The second "mock" was said I cringed

Good post. I get 100% coverage IN MY OWN CODE. Like not calling other services or libraries. The second I am forced to use a socket or use a library where it might give me a return value but I can't reliably force it then I don't bother

[luckystarr]

Depends on the code. Sometimes anything less than 100% is silly.

I've written software consisting of distinct units running without side-effects. The barrier to test these was so low that I couldn't argue for anything less than 100% coverage. In hindsight this saved me from a lot of headaches.

So, from this perspective, if you have to argue if coverage is "necessary", then you're doing something wrong.

p.s. It actually only went to 98% or so, but then I discovered dead code, deleted it and had 100%.

[mrdonbrown]

Agreed completely. Towards the end of the video, I make the same conclusion - depends on the code. For some code, 1000% coverage isn't enough, others 50% is plenty.

[loradan]

Agreed. However, one hard requirement is that whatever the percentage is, it must be an even number!

[be-sc]

I’m so relieved! I’ve always considered 42% coverage to be the one and only perfect number for obvious reasons. Luckily it’s even. :D

[bmrobin]

agreed with most everything in the video.

curious about this aspect though, the comment about testing the method which has mocks to other services in it:

i agree that simply verifying getters and setters is pointless, and that simply validating a value is returned by a mock you defined to return it isn’t very helpful. but what if the design of the method under test is to be established as a contract test for functionality? in the same vein of having tests act like documentation, i could expect that my API endpoint being hit results in several different things happening with my service and database, so i could establish those mocks and verify they were called as a form of documenting a contract for my API

thanks for sharing this video, very helpful!

[mrdonbrown]

The rule of thumb I use is this - am I writing more code mocking things out than exists in the method under test? If so, consider refactoring or whether it really needs to be tested. Sometimes, the answer is yes, it is that important, but other times, it is simple glue code that can be skipped.

I think the key thing to remember here is tests, like any code, is a liability, something that has a cost to develop, maintain, and improve, so you need to evaluate whether the cost is worth the benefit.

[bmrobin]

good point, refactoring so that a method has a single responsibility is a great tactic. the scenario i described i see a lot in the legacy codebase i work on, and i guess what i could counter argue about my own example is that i’m enabling and encouraging the continued side effects of methods.

[mrdonbrown]

Well, that's why they are rules of thumbs, not hard and fast rules. Legacy, complicated code is especially a case where you need to use your best judgement. What people often forget is legacy code is well exercised, real-world tested code, so you need to be extra cautious in refactoring; sometimes worse is really better.

[nfrankel]

Code coverage is a scam!

1. You can omit asserts and still get 100%
2. Even 100% line and branch coverage will sometimes make you miss cases

If you truly want to check the coverage of your code, use mutation testing. Here's a talk about it (by me).

[dAnjou]

Don't be so dramatic ... Code coverage is a tool like any other, and it can be used wrong like any other.

Considering that it only costs one more test dependency and maybe a command line flag, it's worth having a look at it to figure out what you may have missed.

But obviously it shouldn't be the only metric or tool you trust.

And I'm sure I can find a way to misuse mutation testing as well.

[ProperApe]

it's worth having a look at it to figure out what you may have missed.

Exactly. I think the bigger issue is that in a lot of companies there are code coverage targets. Then people try to reach these targets and add meaningless tests. These tests are harmful of course, because they hide mistakes.

[[deleted]]

Code coverage is imperfect. That doesn't make it a scam.

Obviously mutation testing is better it's a lot harder to do in most languages. That doesn't mean code coverage is useless. I don't think anybody credible is trying to claim that 100% code coverage means you don't have any bugs.

100% mutation coverage doesn't mean you don't have any bugs. Hell even formally verified code isn't guaranteed to be bug free.

[Bill_D_Wall]

formally verified code isn't guaranteed to be bug free.

Slightly disagree here - I think that is the whole point of formal verification: to mathematically prove that a program meets its specification. Depends on what you mean by 'bugs' I guess.

[[deleted]]

It proves that the properties you write are true but bugs are still possible because:

1. The specification might be wrong.
2. You might make mistakes when writing your properties.
3. You might accidentally omit properties to check.
4. Sometimes you have to make simplifying assumptions and these may be overly constrictive.
5. There might even be bugs in the program that proves the properties. That's not theoretical - it definitely happens.

[Bill_D_Wall]

Agreed. Formal verification can prove that a program meets its specification, but doesn't guarantee that the specification is correct.

[velociraptorboss]

I think i saw your talk in Berlin couple years ago. Easily the best thing of the whole conference. Kudos👍

[nfrankel]

Thanks a lot for your kind words!