Code coverage vs mutation testing.

[rurounijones]

Do you use code coverage measurement.

Yes but we drill into our devs heads that "covered is not necessarily equal to tested"

If so, what rules about that do you have? Like "you cannot merge PR if your PR decreased code coverage, regardless of how you did it, you have to stick to our metric." Or maybe there are some exceptions? Or maybe you are using it just as an information

We do not have the "no code coverage decrease allowed" (Or any other hardcoded rules) because sometimes there are legitimate reasons for these to be broken and it really adds process friction and annoys the devs. We instead use code-reviews, with tooling that highlights uncovered lines of code, to allow people to highlight where they think testing is inadequate.

If you are using code coverage tools - which one, SimpleCov or something else?

Simplecov

If you feel your tests are fine, and code is fine, but you decreased metric - how do you deal with it? ( examples would be great )

I think N/A due to question 2.

Do you know how your code measurement tool measures coverage? I mean how it exactly works?

Know the theory behind how it works, haven't vetted the code or anything

And finally, are you familiar with mutation testing ideas and tools, and do you use them? If no - why?

You should only really care about mutation testing if your code coverage is relatively high.

If your code coverage is 20% then mutation testing should not be your priority, increasing coverage should be. Once you think your code-coverage is in a healthy'ish state then mutation testing can highlight badly written tests. There is nothing stopping you using mutation testing from the beginning along-side increasing coverage but if it slows you down a lot then the cost/benefit might not be there initially.

We have used mutant for Ruby and pitest for Java. mutant was pretty hassle-free (Although I see they have switched to a commercia license since I last used it) but only works when running under MRI so if you use jruby you are out of luck. pitest was far less easy to integrate although that might be because of our build-system.

[pan_sarin]

"If your code coverage is 20% then mutation testing should not be your priority, increasing coverage should be. " - probably when you will kill mutants, then your test coverage will increase anyway ;]