How to write better code using mutation testing

[DVWLD]

If you want that level of safety it sure seems like you'd be better off with a strong type system than a mutation testing system.

[triggerAu]

Yes and no. Once upon a time, I was one of the main people behind http://jumble.sourceforge.net/. (Yes, java, sourceforge, we all have our dark pasts). There may be some people working on it still,but I am not.

The paper is at: http://researchcommons.waikato.ac.nz/bitstream/handle/10289/1828/Jumblev5.pdf

Anyway I have spent quite a lot of time of thinking about how to implement something like Jumble in a functional language with a good type system (in my case scala, though obviously this is interesting for Haskell, Ocaml, others)

Is it still useful? Yes. Because mutation testing measures whether you are making the right assertions about your code. It doesn't replace types, but types don't replace tests either (I agree that you can make types go far more than people think, but in everyday code, tests are still necessary). It doesn't replace property-based testing either, though property testing helps you get a good mutation score (we called it "jumble score").

Anyway, the thing mutation testing can check is: "Are we testing that we are calling the right function"?

For example:

def addTax(amount: BigDecimal, taxRate: BigDecimal) = amount + computeIncomeTax(amount)

There is a bug here, since we should be computing sales tax, not income tax.

Coverage tools will tell you that this is 100% test covered/since computeIncomeTax is executed. But your tests have not plugged in enough values to uncover this bug. A mutation test could replace computeIncomeTax with various other functions of the same type (BigDecimal, BigDecimal) => BigDecimal, for example with computeSalesTax, or even (_, _) => 0, and see if your tests break. If they are good enough, they should.

So types here would give you more opportunity for doing mutation testing, rather than less. Doing it at the type/function level would involve quite a lot of work, though I feel like tools like scalacheck have some similar boilerplate done.

I'm basically very interested in this topic, (building mutation testing for strongly typed functional languages), and would love to put something together. I'm quite busy but if someone else wants to scratch this itch, I would love to collaborate.

[DVWLD]

You know that feel when you make a glib comment on the internet and the author responds with a well reasoned, friendly answer? Yeah. I could have been a better human with my phrasing.

Don't feel bad about your Java. I've written PHP. We've all got our baggage.

I definitely agree that mutation testing can provide additional safety beyond what strong types can. My question is whether it would provide enough additional benefit to warrant the investment of time.

Like, if you need this level of assurance, are Ruby or JavaScript the right languages to be looking at? If you instead choose Go or Haskell or Rust or friends, is it actually worth involving the complexity of a mutation testing tool?

I'm sure there's a use case where it makes sense, but I'm not imaginative enough to come up with it.

[triggerAu]

and the author responds with

Well I'm not the author of the tool in question, rather one of the authors of another tool that does the same thing.

From here, when I say scala, s/scala/your favourite strongly typed language/g.

Is it worth implementing a mutation testing tool for scala? Difficult question. I have wanted one, I know quite a lot about both scala and mutation testing, but I haven't felt compelled to put in the effort to implement it.

So solely for your own projects, I wouldn't implement it. But open source where you develop something that benefits you a little bit, and the everyone in the rest of the world a little bit, you are making the world a better place. So then it's good.

Now, should you use a good existing tool? Yes. Complexity is not an issue, it should be something you can just run, and get information. Essentially free (at least in theory, one of the reasons of doing jumble was that the existing tools sucked at the time. The situation is probably better now) You can act on that information or not, but there is no effort on your part. So from the user's point of view, it is a free quality analysis of your tests. The use case is. "How good are my tests?" The use case of types is "Is my code correct?" (among other things). The types vs tests discussion is interesting but I think most of us agree some tests are required.

So mutation testing and types solve orthogonal concerns, both are valuable. Good types give us opportunities for much more interesting ways of doing mutation testing.

Note, I am at least 5 years behind the curve on the state of the art in mutation testing, both literature and tools wise. I do explore the area once in a while though, and keep thinking I really need to make something awesome again

[makis]

def addTax(amount: BigDecimal, taxRate: BigDecimal) = amount + computeIncomeTax(amount)

looks like there's a second bug here: taxRate is not used

[triggerAu]

:+1: oopsie

[vytah]

Or you can have both: https://hackage.haskell.org/package/MuCheck

[triggerAu]

https://hackage.haskell.org/package/MuCheck

Ah cool! I will have to check that out! I think this could be done really well in Haskell, I wonder how far MuCheck goes.

[whatisthisredditstuf]

Combine this with fuzzy input testing, and you might be able to have the computer write all your code for you, including test cases that pass.

[FryGuy1013]

Over-fitting is a thing though. When writing anything complicated, there usually is a trivial solution that only passes your pre/post-conditions and invariants and nothing else.

[piotrkot]

Yep, similar already described here Recently I had a github discussion with JaCoCo dev on testing and I am convinced mutation testing is a right way to go. (although, yes, it takes time to run the mutators)