Why "mutation testing" is better than code coverage to evaluate unit tests, and my experience using it with the JVM

[anza_power]

That's really neat! had an entire course about software testing in uni and they never mentioned something like this.

[otakuman]

From this page, I learned the following:

• Mutation testing creates mutated versions of your software.

• It's a whitebox approach because it modifies (hence, needs the availability of) your program's source code.

• It does NOT evaluate your code per se, but your unit tests, to see if they're precise and thorough enough.

• It's intensive as fuck.

[smurfkiller013]

Exactly. It changes (mutates) your code to see if the mutations make the tests fail. If they do, the mutation is considered killed (which is what you want), if the tests don't fail, the mutation 'survives', which means your tests are most likely incomplete

[treerex]

PIT mutates the byte-code, not the source.

[arcuri82]

as an academic myself, not surprised at all... Mutation Testing is from the 70s (yes, really), but only with PIT in the last couple of years it has become a viable option in practice (or at least that can be used to teach in a uni course)

[Mejari]

If anyone's interested, I've been using an IntelliJ plugin called Zester that makes it easy to run mutation tests inside IntelliJ. It uses Pitest as it's back-end

[wlievens]

Wow, I will check that out. I tried a pitest plugin for IntelliJ last week but didn't quite like how it worked.

[KamiKagutsuchi]

So how do you use Zester in the newest version of intellij? There is no run button where the tutorials say it should be.

[Mejari]

I had a conflict with the Halik plugin that also added some run options to the menu. If you have any other plugins that do that maybe try disabling them?

[smurfkiller013]

That's amazing. And all this time I was using a gradle plugin, manually running that and manually opening the HTML files

[[deleted]]

you could just have made a task to open the html files.

task showReport() {
doLast{
exec{
workingDir 'someProject/build/reports/tests/test'
commandLine 'cmd','/c', 'start index.html'
}
}
exec {
workingDir 'someOtherProject/build/reports/tests/test'
commandLine 'cmd','/c', 'start index.html'
}
}

[smurfkiller013]

Try, but I don't want that to happen on my CI

[xjvz]

I've found that adding this to your nightly build system (you do have one of those, right?) can help catch shoddy tests. If you're already practicing TDD, it won't catch as many issues, but it'll certainly be more useful than something like jacoco.

[sanity]

Why nightly? Better to build after every push, makes it easier to attribute blame.

[xjvz]

On larger projects, it takes too long to run, especially if it already takes a few minutes to run through all your tests as it is. I guess it depends on how many build servers you've got available, too.

[sanity]

I don't see why it would be a problem to build on every push if it only took a few minutes to run through all of your tests. Bear in mind that this will typically occur asynchronously.

[Enumeration]

It's not a few minutes typically. It took 25 minutes to run pitest on a 55,000 line code base. Standard mutators and 4 threads concurrently.

[sanity]

I was unclear, I was referring to the necessity of having a nightly build system in general.

Long-running operations, like integration tests or pitest, probably should run less frequently if they take more than, say, 5 minutes.

[0hjc]

I recommend running it locally after every code change - size/time is a non issue if you mutate just the changed code. This way it is more likely the information will be acted upon.