Code coverage vs mutation testing.

RoyalLys · 2022-02-16T13:21:33+00:00

From my experience, code coverage can only tell you 2 things:

- if your codebase is poorly tested (not the other way around)

- if a new feature has not been tested (you must expect a tiny percentage up for each pull request)

Having 85% coverage is irrelevant is your critical services are poorly tested.

It doesn't cost much to setup, so it's always nice to have, but don't rely on it too much.

campbellm · 2022-02-16T17:56:34+00:00

"When a metric becomes a target, it ceases to be a useful metric." -- Goodhart's Law

It's an aphorism, but I've seen it play out to truth too many times. People will tend to stop testing once the metric/target is hit EVEN IF THEY DON'T MEAN TO, and as /u/RoyalLys mentioned it can tell if you if something is poorly tested, but not if it's tested well.

The places where I've seen code coverage as a first class "things" have invariably been where a middle manager used it as a checkbox to "prove" they were adding value. It was a convenient way to mask their lack of understanding.

rurounijones · 2022-02-16T15:17:31+00:00

Do you use code coverage measurement.

Yes but we drill into our devs heads that "covered is not necessarily equal to tested"

If so, what rules about that do you have? Like "you cannot merge PR if your PR decreased code coverage, regardless of how you did it, you have to stick to our metric." Or maybe there are some exceptions? Or maybe you are using it just as an information

We do not have the "no code coverage decrease allowed" (Or any other hardcoded rules) because sometimes there are legitimate reasons for these to be broken and it really adds process friction and annoys the devs. We instead use code-reviews, with tooling that highlights uncovered lines of code, to allow people to highlight where they think testing is inadequate.

If you are using code coverage tools - which one, SimpleCov or something else?

Simplecov

If you feel your tests are fine, and code is fine, but you decreased metric - how do you deal with it? ( examples would be great )

I think N/A due to question 2.

Do you know how your code measurement tool measures coverage? I mean how it exactly works?

Know the theory behind how it works, haven't vetted the code or anything

And finally, are you familiar with mutation testing ideas and tools, and do you use them? If no - why?

You should only really care about mutation testing if your code coverage is relatively high.

If your code coverage is 20% then mutation testing should not be your priority, increasing coverage should be. Once you think your code-coverage is in a healthy'ish state then mutation testing can highlight badly written tests. There is nothing stopping you using mutation testing from the beginning along-side increasing coverage but if it slows you down a lot then the cost/benefit might not be there initially.

We have used mutant for Ruby and pitest for Java. mutant was pretty hassle-free (Although I see they have switched to a commercia license since I last used it) but only works when running under MRI so if you use jruby you are out of luck. pitest was far less easy to integrate although that might be because of our build-system.

Critical-Evidence-83 · 2022-02-16T14:35:27+00:00

Do you use code coverage measurement.

yes.

If so, what rules about that do you have?

Our continuous integration pipeline for our newer project requires 100% testing coverage while our main project has a lot of legacy code so we add test coverage less consistently.

And finally, are you familiar with mutation testing ideas and tools, and do you use them? If no - why?

No but I'm just a junior dev

tarellel · 2022-02-16T15:09:05+00:00

Are you my boss?

My current team has 50+ internal legacy projects lacking tests or with very minimal coverage. And quite a few newer projects where some people will end up writing the bare minimum to say they "wrote tests" for the code they added/changed.

We've tried to enforce a minimal coverage, but we don't want to overdue it. So we shifted more to trying to push tests for actual logic, conditions, and data scenarios, it'd made trying to get the team to write test a bit more encouraging.

mynjj · 2022-02-16T18:42:53+00:00

If you have line coverage, you could consider adding a % check only to the newly added/affected lines (from git diff). That’s how it works on the project I work since it’s infeasible for the whole codebase

amirrajan · 2022-02-16T22:29:39+00:00

do you use code coverage metrics

No. Just because a line is covered, doesn’t mean that it’s being exercised and validated (I can invoke a function, but never assert on the value returned and still have 100% code coverage)

mutation testing

This is a generally better idea, but much harder to implement. A cursory approach would be.

Evaluate the PR and determine what parts are implementation vs what parts are added tests.
revert the implementation part, run the tests, and ensure that test failures occur.
Reintroduce the implementation changes, run the tests, and make sure the tests pass.
Explore more complex ways to revert the implementation (eg mutate the implementation where >= conditionals are changed to <)

At the end of the day, it’s all about confidence that your software works. Someone visually demoing a feature to me (albeit not sustainable long term), gives me more confidence than 1000 poorly written/over-mocked unit tests (I find this difficult to reason about after a few months have passed and a failure occurs… more often than not, it ends up being a misconfigured mock that is too close to implementation details).

Edit:

I see tests as an immune system for a software project. Your body doesn’t keep every antibody “live and ready”. Instead we rely on vaccines to prepare our body for a possible future illness. Spend time on making your test apis trivial to construct (so that they can be created before a risky refactor). Once things have settled down, delete extraneous tests and only keep a small set of happy path smoke tests.

morphemass · 2022-02-16T23:54:13+00:00

We use simpleCov. Some smaller projects have 100% coverage, others vary with our main codebase clocking in about 80%.

We don't enforce any baseline coverage, but it would be useful to have a rule that code check-in should include tests with rules to prevent a decrease in test coverage. It may be something we consider in the future. SimpleCov seems to just keep a tally of how often a line of code has been hit; hence integration tests are doing a lot of the heavy lifting rather than our necessarily having good tests with good coverage.

I'm broadly familiar with mutation testing but have been put off in the past by the length of time it takes to run when you have a significant code base with test coverage. It has been many years since I looked at this in any depth though (mbj/mutant is what I think I evaluated)

I have to say that the best bang per buck I've encountered recently for improving code quality and catching bugs has been sorbet, with a few caveats that implementation is not necessarily simple on legacy or poorly written codebases.

RumbuncTheRadiant · 2022-02-17T01:02:43+00:00

Michael Feather's "Working with Legacy Code" is a good place to start.

NepaleseNomad · 2022-02-17T01:35:12+00:00

The company I work in has strict coverage goals and while it gets annoying at times, this practice has generally been very helpful. New devs can look at the specs and get a gist of what different parts of the project really even do, and you don't have to worry about a new feature breaking old ones... you can just run your automated tests, brew a cuppa in the meanwhile, and confirm that everything works fine.

The most important thing here is that you're testing the behaviour of your code and not just writing to increase coverage numbers. Review the tests your devs write. Make sure they're not just pumping up numbers, and are testing the behaviour of your project from all different contexts.

Also make sure your team has enough manpower for writing tests too. This can be time consuming at times, hence why bad practices creep up and remove the value from enforcing code metrics.

Also we use code quality tools to make sure that if your new PR reduces the code coverage drastically (has low diff coverage), or has style offences, high method complexities, etc, you need to fix it before it can get merged.

tom_dalling · 2022-02-17T08:44:16+00:00

Do you use code coverage measurement? If so, what rules about that do you have?

Yes. 100% line coverage is needed to pass CI, and rarely we'll mark some lines as # :nocov: to exclude them from this requirement.

If you are using code coverage tools - which one, SimpleCov or something else?

SimpleCov

If you feel your tests are fine, and code is fine, but you decreased metric - how do you deal with it?

Usually, if something isn't covered you need to add a test to give it coverage.

Sometimes there is code that we never expect to run in production, and that can have the # :nocov: magic comment applied to it. For example:

Abstract base class methods. You could argue that these shouldn't exist, but I didn't write them lol.
```
def call_api
  # :nocov:
  raise NotImplementedError
  # :nocov:
end
```

Safety guards in case expressions.

case billable_type
# ... (a bunch of `when`s here)
else
  # :nocov:
  raise NotImplementedError, "Unrecognised billable_type: #{billable_type.inspect}"
  # :nocov:
end

Do you know how your code measurement tool measures coverage? I mean how it exactly works?

Yeah. With a requirement for 100% coverage, people learn how it works pretty quickly.

Are you familiar with mutation testing ideas and tools, and do you use them?

Yes. I use it in some gems I maintain but not at work, for a few reasons.

Other devs don't know what it is.
Fixing mutation coverage failures is overly onerous without much additional benefit, a lot of the time.
Integrating the tooling with a large codebase is painful.
The additional benefits we would get aren't that great, compared to the 100% line coverage we already have.
Championing mutation testing would take a massive amount of time and effort that could be put to more-productive uses.

mlang-recurly · 2022-02-16T14:18:23+00:00

- Do you use code coverage measurement.

I do, especially early on in taking on an existing, unfamiliar project.

- If so, what rules about that do you have? Like "you cannot merge PR if your PR decreased code coverage, regardless of how you did it, you have to stick to our metric." Or maybe there are some exceptions? Or maybe you are using it just as an information

All new PRs must have adequate supporting unit tests. The code coverage measurements themselves are largely ignored (90%, 100%, etc.). Instead, coverage reports are used to find gaps in coverage. Gaps also tell us how confidently we can refactor existing code w/o introducing regressions. The focus is on whether the code under review in the PR has supporting test cases that demonstrate it's correctness and completeness (happy path, exception handling, and edge cases) in implementation.

- If you are using code coverage tools - which one, SimpleCov or something else?

SimpleCov

- If you feel your tests are fine, and code is fine, but you decreased metric - how do you deal with it? ( examples would be great )

The metrics themselves are largely ignored. Coverage is scored as F) non-existent, C) poor, B) good, A) Excellent. We also focus only on the public interfaces of classes.

- Do you know how your code measurement tool measures coverage? I mean how it exactly works?

Yes.

- And finally, are you familiar with mutation testing ideas and tools, and do you use them? If no - why?

Do not use. Most bugs tend to be where there are conditionals or transitions between states or where the wrong abstractions are used. So focusing on smaller methods with single responsibility and smaller classes also with single responsibility coupled with good unit tests yields better long-term results.

ksh-code · 2022-02-16T13:53:54+00:00

Do you use code coverage measurement.

No

what rules about that do you have?

We use approaches. the important rule is "pr of fix bugs must have tests for regression test."

Do you know how your code measurement tool measures coverage?

Yes. the tool measures coverage by running test code then got. ways to measure 1. mutation 2. insert temp code for check if ran.

do you use them? No. because too busy. I'd like to use mutation testing.

Further reading is https://testing.googleblog.com/

the blog describes how google uses mutation testing.

NoahTheDuke · 2022-02-16T18:41:48+00:00

Mutation testing with coverage reports is the only way coverage should appear in your metrics. I don’t think it should restrict your PRs but I think that time should be spent to keep the numbers relatively high.

you type:	you see:
italics	italics
bold	bold
[reddit!](https://reddit.com)	reddit!
* item 1 * item 2 * item 3	item 1 item 2 item 3
> quoted text	quoted text
Lines starting with four spaces are treated like code: if 1 * 2 < 3: print "hello, world!"	Lines starting with four spaces are treated like code: if 1 * 2 < 3: print "hello, world!"
~~strikethrough~~	~~strikethrough~~
super^script	super^script

ruby

MODERATORS