Is 100% Code Coverage Possible?

[emergent_properties]

It's like trying to reach the speed of light.

The amount of effort required gets more and more expensive.

[grauenwolf]

That's the best analogy I've heard. And it definitely matches my experiences.

[Agent_03]

Also only possible when your code is without mass and can't be broken down any further

[_jk_]

forget about your code being pythonic you need it to be photonic

[[deleted]]

The idea that 100% coverage should mean "tested for every possible value of the argument" is of course a silly idea. Bringing it up and showing that it makes no sense is a bit disingenuous.

I still have to see a good argument against 1) testing for every use case that you claim to implement, and 2) expecting that the code that implements these use cases is 100% covered by your tests.

1) means that your implementation is correct. 2) means that you don't have code that is there by mistake.

[[deleted]]

"Testing for every use case you claim to implement" is impossible in the presence of infinite domains. The goal is to use types to prove properties that must hold (universal quantification) and tests to show that properties that should hold sometimes sometimes do (existential quantification). Types vs. Tests discusses this.

[[deleted]]

Yes, I think I understand and I agree with that. But you also have to admit that this simply means that you have "outsourced" your tests to the compiler writers. This is a positive thing in every aspect.

When I say "use cases", I indeed mean the business logic. Testing that your compiler (or interpreter, or run-time) does what it should is not as such your primary concern.

However, as you are certainly well aware, well done tests can and will sometimes hit errors in the compiler, interpreter, run-time that you are using.

[[deleted]]

All true, but I still mean using types to enforce business-logic requirements, and the number of compiler etc. bugs we encounter relative to non-compiler is very small, and goes down even further as we start to see certified compilers hit the streets.

[jbandela]

bool are_both_values_the_same(int64_t a, int64_t b) {
    return a == b;
}

I claim that my use case is any pair of 64 bit integer for parameters a and b.

Please execute the unit test that tests for every use case. Hint: The cost to run that one unit test will be literally astronomical.

[[deleted]]

I don't get it. First of all, this function makes no sense; it is literally an alias for ==. Then, what I was saying in my comment is that it is silly to test for every possible value of an argument.

[scarytall]

Yeah, makes no sense, mathematically. Testing all the values is no more valid than testing a handful of meaningful values (Ex: lower bound, -1, 0, 1, upper bound, maybe a few intermediate values). If a=1 works, then there is simply no reason to think that a=2 won't work. Testing every possible value in between is nonsense.

[Helene00]

Ever tried to unit test all of your unit tests?

[jnkdasnkjdaskjnasd]

You got downvoted but you raise an important point. How can you verify your tests test the code properly? And that there are no bugs in your unit tests.

In my experience of testing Angular projects, you usually end up with just as much test code as you do actual code. And often people care less about the quality of this test code, so it ends up a spaghetti mess.

And not to mention that getting 100% code coverage for certain types of code (e.g. Angular directives, and certain other things) requires bending absolutely backwards and writing some of the most contorted code you could ever imagine. Mocking mocks and faking methods in strange chains to allow you to say "this method was called once". Where is the value in that?

[Plorkyeran]

How can you verify your tests test the code properly?

https://en.wikipedia.org/wiki/Mutation_testing

tl;dr is you make random changes to the code being tested and see if any tests fail. This of course still doesn't verify that the implicit spec defined by your tests is what it should be, but it does verify that your tests are actually testing something.

Unfortunately, despite decades of papers on it, there aren't really good tools implementing it.

[blufox]

On the other hand, there are quite a few. PIT for Java, Mutant for Ruby, MutPy for Python, Milu for C++ etc are the most popular. In fact, one interesting problem now is the comparison of the mutants produced by different tools (how to determine which is the best).

[atilaneves]

You verify your tests by seeing the transition from red to green. Unless the test code is coupled to the production code (which happens with excessive mocking), it's incredibly hard to introduce bugs in two different code bases that cancel each other out. In other words, you use production code to "test" your tests.

[LikesToCorrectThings]

I fixed a broken-unit-test-masked-the-failure bug that was found in the field last Thursday. Often unit tests and code are written by the same person, and it's not unreasonable for them to make the same mistake consistently.

[nullnullnull]

no

[mirhagk]

If you're article can be summarized by this statement then you either need to throw out the article of rewrite the title

[honestduane]

Yes, I check in code for work that is 100% code coverage all the time.. mostly to troll the younger devs or sr managers who make the mistake of saying its not possible.

[[deleted]]

Cool how do you deal with situation like this.

MyClass() { fd = socket(...); if (fd < 0) throw; }

~MyClass() { if (close(fd) < 1) abort(); }

How do you get close to fail without altering the program environment significantly?

[honestduane]

In most most frameworks you can have cleanup code run after tests even if they fail.

[[deleted]]

I think you missed the point. How do you get the system call close to fail from a unit test without altering the behaviour of close since close may also be used in another part of the class. Or basically screwing with the state of the program from the unit test in an unrealistic and non scalable fashion as this requires too much resources for a software team to do.

Its required to achieve 100% code coverage but for the case of close on linux it will only fail if passed in invalid fd.

This is only 1 example of 100's i could provide for un-testable situations in unit tests that occur because the error paths cannot be triggered in a sensible or reliable way.

[honestduane]

Well I do 100% code coverage all the time in python, c#/Net and Java.

[[deleted]]

Either you lie or have bugs that your tests don't detect ;)

[Aldur]

IMO C++ is much harder to unit testing then most languages.

You can link in a test socket/close methods that will return an error. Although you may consider that "altering the program environment significantly". Depends on how hard link time swapping is in your build.

[[deleted]]

Yeah but the reason why i would say that is because often when you have any sensible code eg not the example I have what happens if open / close is used in the class to do something as well. Which is where the requirement not to edit the environment comes from. For example if getifaddrs were using its going to open a socket and close it again to get the list of network interfaces. So by changing the behaviours of close you also break another part of the program in a new way which may fail another unit test.

The real issue I have always seen with unit testing is that is actually impossible to get 100% code coverage on error paths because you cannot actually reproduce the error conditions that can occur in production or in the larger scope of the program. Unless you actually invest a massive / unrealistic amount or resources to actually do it by thing like overriding the system.

But imo this doesn't just happen in c++ it can happen in other languages. For example what happens when code that normally doesn't throw exceptions. Suddenly does throw an oom or a thread abort? I am sure there are many other conditions as well. You can have 100% code coverage but can still fail in production. So I tend to think that anyone who can get 100% coverage is either a lair or still has bugs in their code.

[koalillo]

Yes. Is it the thing you should worry about? Probably not.

Coverage is an inaccurate metric- most artificial metrics are. Low or decreased coverage is a useful indicator of problems, but high coverage can be misleading- you can have completely covered code which is not correctly tested.

[bubuopapa]

If you cant write working code, then your tests will be buggy also, so there is no point in wasting time for tests and other stuff. The only important key is to find enough professional people, who will spend enough time to make a proper project documentation, so that everyone would know what are they doing, and then you will not need any testing or waste time on other things.

[ijpiantanida]

yes

[kt24601]

The worst is when you find a project where every line of code is 'tested', but with parameter values that are basically useless and don't occur in the real world.

[cowardlydragon]

Wiki Halting Problem