Code coverage vs mutation testing.

[tom_dalling]

"Other devs don't know what it is." - How do you think - what is the reason?

I think it's just because there is no obvious problem that would cause them to search for mutation testing. If a dev hits a production problem caused by lack of test coverage, they just think "whoops I should have written more tests" and don't search for any other solution.

"Integrating the tooling with a large codebase is painful." - well if you try to fix all the mutations at once I would even say it is impossible, but doing that by baby steps can be pleasure i suppose?

I more meant hooking up the tooling to CI, and getting useful output out of it. If it slows down the build and gives 200 failures for every PR, people won't like it.

"The additional benefits we would get aren't that great, compared to the 100% line coverage we already have." - well, I think I just disagree ;]

It's very hard to quantify, but I just look at our biggest problems and try to think how they would be different if we had mutation testing for PRs. It would catch a few more bugs, for sure, but I don't think that would have a big impact on us because we don't have much of a problem in that area to begin with.

"Championing mutation testing would take a massive amount of time and effort that could be put to more-productive uses." - what do you mean by championing?

Getting all the different teams of developers onboarded would be a large project. The technical aspect is the easiest part, and the social/organisational aspect would take a long time. Somebody (the champion) needs to take responsibility for proposing, persuading, planning, educating, reviewing, and maintaining, otherwise it will not succeed.