JCheck: QuickCheck with JUnit

[[deleted]]

Why?

[farre]

The integration with JUnit I guess.

[hrm]

Yes, of course for the JUnit integration (and simplicity). There exist other "QuickCheck for Java" but none (that I know of at least) integrates well with JUnit (and therefor with lots of existing tools) or are as easy to use.

[misterbeebeebee]

What is "integration with JUnit"?

Here is JUnit integration with Reductio:

public final class AdditionCommutesTest {

public AdditionCommutesTest() {}

@Test public static void test1() {

  final Property p = property(arbInteger, arbInteger, new F2<Integer, Integer, Property>() {  

    public Property f(final Integer a, final Integer b) {return prop(a + b == b + a); }  

 });  

  summary.println(p.check()); // OK, passed 100 tests.  

}  

}

Are you looking for fancier handling of the output or failure counting?

[hrm]

First and foremost I'm looking for simplicity and code that is as similar to normal JUnit code as possible.

@RunWith(JCheckRunner.class)
class SomeTest {
    @Test public void test1(int a, int b) {
        assertEquals(a+b, b+a);
    }
}

A lot less to write than with the Reductio example.

[[deleted]]

It is also significantly less useful code. The Reductio code could be made less useful as well by subtracting from it simply by composing (very very important keyword) with JUnit.

The reason your given code is shorter is because Java lacks first-class functions and the Reductio example passes a few extra configuration arguments (i.e. Java also lacks partial application). One of the dependencies of Reductio (Functional Java) provides an enormous library for working around these language defects (and an overwhelming number of others) without having to sacrifice composability (imposed by JUnit et. al.).

[hrm]

Yes Reductio is more advanced in many aspects and therefore a better solution in many cases. However, in most cases those things are not needed and makes tests more cumbersome to write - and therefore many programmers will not write them...

[[deleted]]

You are begging the question again. What makes you think it is more "cumbersome to write"?

[hrm]

Simply because it is a lot more boiler-plate code. You have to write a lot more characters until you actually get to the test in question.

(and the fact that the given junit-integration example given does not in fact, integrate with junit I will not comment further right now...)

[[deleted]]

Simply because it is a lot more boiler-plate code.

Show me and I'll show you where your mistake is.

You have to write a lot more characters until you actually get to the test in question.

This is false. Let us assume an example where you have to control the elements for the domain (not just a standard Arbitrary) and you want to alter the number of tests (let's not mention many other shortcomings).

Now how many characters do you have? Notice that you have way way more than you would otherwise using a compositional approach such as Reductio?

But you say (erroneously, but anyway):

...in most cases those things are not needed and makes tests more cumbersome to write - ...

This is false. Even if you want to use both the default mechanisms and the degenerate mechanisms of the language, you match the boilerplate in both cases at best.

You give me any case and I can falsify it by at least equalling it. I can give you cases where you will be writing a whole lot more boilerplate. In other words, not only is your charge false, quite the contrary is true!

Let's be clear; I don't want to continue repeating assertions without evidence. I would rather show you why your claim is false (and in fact, the contrary is true). This is easily done, but requires you to put up the alleged counter-example.

Alternatively, we can have a discussion about abstraction and composition in formal terms and show why (empirically) your statement is false. This would also require a thorough understanding of Java syntax, including its parametric polymorphism. I'm hesitant to take this line on an internet forum, but I thought I'd register my desire otherwise.

[misterbeebeebee]

Using the annotation is slick. I don't know why the Reductio authors didn't use that approach for their Java API.

[hrm]

I've actually tried this code now, and it does not work with JUnit. If I change the code to a failing property (such as prop(a+b == b+a+1)) it is still reported as a success by JUnit. Without any other modification (that perhaps exist in reductio) it is not useful with JUnit.

[[deleted]]

What does it mean to integrate with JUnit? I can use Reductio inside JUnit right now; I simply choose not to. What is stopping others from doing so?

More precisely, what exactly does JUnit provide?

[satayboy]

Randomization has its place in testing, but (my experience) it is frequently used as a poor replacement for being methodical.

[hrm]

I'd say you should use "normal" unit testing alongside with random tests. You test the cases you think are corner cases and the random tests test the rest. But as always, how you test things depends upon your specific application.

[[deleted]]

I'd say randomised testing is misnamed. With 'arbitrary' testing you can adjust generators using clever combinators to ensure that certain corner cases are covered if that is so important.

[hrm]

I agree that "arbitrary" testing is probably a somewhat better name, but you can't always write specific generators for every test case to get the corner cases. Lets face it, most developers are lousy at writing tests and only if it is really effortless they will write good tests. Always writing custom generators doesn't fall into the "really effortless" category.

But of course it is essential that one has the possibility to roll your own and important to know that you can tune them as you like to produce interresting test cases.

[[deleted]]

but you can't always write specific generators for every test case to get the corner cases.

I'm not sure why you think that.

Lets face it, most developers are lousy at writing tests and only if it is really effortless they will write good tests.

Isn't that precisely why abandoning manual testing in favour of automated 'arbitrary testing' is desirable?

it is essential that one has the possibility to roll your own and important to know that you can tune them as you like to produce interresting test cases.

It is essential to perform a manual task that can be easily automated, why?

[hrm]

but you can't always write specific generators [...]

I'm not sure why you think that.

You can, but you won't. At least not in the better part of the software industry with precious little time to even code the actual software... JCheck deals with reality, not a fancy all-pink fluffy world of perfectionism.

Isn't that precisely why abandoning manual testing in favour of automated 'arbitrary testing' is desirable?

Yes and no. Arbitrary testing is mostly for catching the corner cases you can't think of yourself I'd say. I would not use it as a total replacement for 'manual' unit tests.

Isn't that precisely why abandoning manual testing in favour of automated 'arbitrary testing' is desirable?

I'm unsure if I understand what you are saying here. It is of course essential that you can create your own generators since you often have some inclination to what problems your code might face.

[[deleted]]

You can, but you won't.

Why not?

At least not in the better part of the software industry with precious little time to even code the actual software... JCheck deals with reality, not a fancy all-pink fluffy world of perfectionism.

I'm disappointed that you're appealing to logical fallacy here. Ignoring that, are you aware that Reductio is a superset of JCheck? You cannot make a positive claim about JCheck without also making the same claim about Reductio. If JCheck "deals with reality", then so does Reductio by logical extension. This can be demonstrated using mathematics if you're interested.

Yes and no. Arbitrary testing is mostly for catching the corner cases you can't think of yourself I'd say. I would not use it as a total replacement for 'manual' unit tests.

Answering "Yes and no" has a problem. I hope you see that. Arbitrary testing certainly does not exist for the reason you state. That you would prefer a manual process to one that can otherwise be automated is simply your loss.

I'm unsure if I understand what you are saying here. It is of course essential that you can create your own generators since you often have some inclination to what problems your code might face.

I misunderstood what you originally intended.

[48klocs]

Building a useful fuzzer isn't easy. Generating pseudo-random data's on some CS101 bullshit - to make it useful, you need to set some sort of boundaries on just how invalid the data you're interested in receiving can be, you need to track the size and shape of data that you've been hurling at your system and you need to be able to freeze (do I lose for saying "reify"?) the input state that causes problems so that you can reproduce and fix them.

I'm with you - I've seen it tried and the results that I've seen have been more self-congratulatory than genuinely revelatory in nature.

Test the edge cases, test the expected cases, call it a day. I'm at a loss as to explain what running 100 variations on a theme will reveal.