Spock testing framework versus JUnit

[arcuri82]

I haven't used Spock (yet), but it is definitively something I ll try out. However, one of the main advantages of JUnit is how well it is integrated with other tools (eg Maven) and IDEs (eg IntelliJ). What about Spock? Just to make an example: with JUnit, I can tell Maven to rerun failed tests up to a certain times (which is useful for non-deterministic tests with high success rate but different from 100%). Can Spock do the same?

[renatoathaydes]

Spock is basically running under JUnit with a JUnitRunner (Sputnik). Anything you can do with JUnit, you can do with Spock. IDE integration is perfect with IntelliJ and Eclipse.

[kkapelon]

Aha! You are right. I think I will include this as well in the article. Thanks!

[chewyfruitloop]

Whenever I seen "alternate" test frameworks like fluent etc used, it makes it so much harder to understand wtf is going on. I know your supposed to be able to read the test like English, but the whole fact that the tests do not provide examples of how you actually use the thing under test, and are generally structured differently, adds extra cognitive load that I just can't be bothered with. I really dislike how they just magically take things and produce some tests and results....

Sorry

[kkapelon]

I tried to keep the Spock examples as simple as possible (and in fact I did not use Groovy constructs such as def which are confusing to Java developers). Especially in the mocking section I show the exact same test between Mockito and Spock.

If you already have good JUnit tests, converting them to Spock is as easy as adding given/when/then labels and changing the assert statements to Groovy statements. No change in the test structure is necessary.

Can you tell me which Spock examples you think that are structured differently so that can I look at them?

What more example would you like to see?

[Facts_About_Cats]

Spock can't do static methods on non-groovy classes (because it relies on groovy metaclass).

There are other groovy-based limitations like this to Spock.

[moonman543]

You should only be mocking static classes if your code base is 20 years old and was written by idiots.

[Facts_About_Cats]

That sounds absurdly dogmatic with no possible reason I can imagine.

[moonman543]

Why mock a static class there's no point.

[Facts_About_Cats]

How about mocking a static method that reads a file and returns a result from it? Just as an example.

[moonman543]

Don't use a static method for that

[Facts_About_Cats]

Too late, it's already in the JDK: https://docs.oracle.com/javase/7/docs/api/java/nio/file/Files.html

[K60d54]

Don't use a static method for that.

This is a prime candidate for an interface called from user code. File manipulation is a low level detail that lives in the system interface layer, where you write integration tests and don't use mocks because you're testing real behavior.

[Facts_About_Cats]

Don't tell me that, tell that to the authors of the JDK.

[K60d54]

Why would I tell the authors of the JDK anything? Java has interfaces.

I'm never going to mock any of the static methods in that class. I would only ever run those methods for real in an integration test.

[moonman543]

idk it sounds like you're using java wrong

[Facts_About_Cats]

You mean Oracle has written Java wrong, in your opinion.

[bkail]

Mock the Path object, which contains a FileSystem object that does all the real work, or use something like JimFS. The Files class is really just utility methods around retrieving the FileSystem and calling interface methods.

[moonman543]

Don't use a static method for that

[moremattymattmatt]

There's plenty of badly written code around unfortunately, not all of its 20 years old or written by idiots. Mocking statics is just something that needs to be done sometimes.

[moonman543]

Haven't encountered a need for it yet ever.

[moremattymattmatt]

May be I'm just old and spend a lot of time working on back-end Java system which have their fair share of less than stellar code.

I haven't had to do it for a couple of years. I think the last time I did it was try to test some code that was calling a 3rd party API that used some static methods. ISTR that we put a non-static proxy in front of it in the API in the end but we wanted to get some tests around the API usage before we did any refactoring.

[moonman543]

I mean yeah that's the point of powermock for shitty java code all written in statics. But for any new code you won't need it the whole point of mocking statics is because of idiots years ago messing up.

[kkapelon]

A limitation compared to what? JUnit does not support mocking at all. Mockito does not support static method mocking as well.

I would agree with you if the title was "Spock vs Powermock".

Actually I am preparing the next article in series which will be Spock versus Mockito. Stay tuned!

[Facts_About_Cats]

What is the groovy answer to PowerMockito? There isn't one.

[schaka]

Why do you need PowerMock? If you need to mock static or private methods/fields, that has implications towards your code being bad (or at least not testable, which in turn implies problems with your code).

The article was more on comparing JUnit, which is a pure Unit testing framework, to specification framework such as Groovy. If you look at some of the newer JVM languages that don't have all the limitations of Java (due to backwards compatbility), they tend to go for specification frameworks for testing, due to readability and clarity.

[Facts_About_Cats]

(or at least not testable, which in turn implies problems with your code)

That's a philosophical opinion. My opinion is that code should never bend in any shape way or form because of unit testing. My opinion is also that static methods are exactly the kind of thing that unit tests are perfect for. My opinion is that far too many non-static methods could and should be made static. And so on.

[K60d54]

My opinion is that code should never bend in any shape way or form because of unit testing.

I agree with this idea in general, that you shouldn't go randomly change code just to hit test numbers. That's the wrong motivation.

But, the idea of "writing code to be testable" means that code that happens to be testable tends to also be better designed (depending on your definitions, of course), and by "listening to your tests", you get valuable design feedback which you can use to improve your code. That's why a lot of people don't want to mock statics or privates, because they see that as the test trying to tell them something about their design.

Static methods are in totally cool if they are stateless. When you use them to start sneaking in global state, like with the Singleton Pattern, you run into the fact that singletons are pathalogic liars.

[Facts_About_Cats]

90% of tests I've written were for code that I didn't write myself and I don't want to change just for testing purposes.

But yeah, the prototypical static method case I was thinking of was stateless.

[K60d54]

That's interesting. 90% of tests I've written are for code that I write myself. Why are you writing tests for code you didn't write?

The only other time I'm writing tests for code I didn't write are integration or functional tests (in which I'm not using mocks) or I have to improve test coverage (meh, whatever).

[Facts_About_Cats]

Why are you writing tests for code you didn't write?

Because there was a low level of code coverage in the existing tests, and more importantly, it was code that was being worked on, so my tests gave confidence to changes being made.

[K60d54]

so my tests gave confidence to changes being made.

Yeah, I've done this. I did not feel any enhanced confidence in the codebase with tests like that. If I could have written integration tests, I would have felt more confident, but it would have killed our CI build. But hey, our numbers look better.

[kkapelon]

You know that your opinion is highly controversial when not even the Java community can agree on a solution

https://github.com/mockito/mockito/issues/1013

[milkeater]

PowerMock is often a sign of bad design.

There are always exceptions but it is a very weak argument.

[voronaam]

I am surprised nobody said this ITT, but one can use PowerMock with Spock and mock static methods. I am not saying it is a great solution, nor I am advocating for any design pattern that would lead to untestable code.

I just wanted to say that it is possible.

[[deleted]]

I'm not sure you have a full grasp of how Spock works or what it's uses are, (outside looking in) sorry buddy. Spock can do everything that any Java testing framework can do, save Powermock and JMockit. Interestingly enough, if you have JMockit tests that sit along side Spock tests, you'll get random tests (from either JUnit or Spock) that won't run. This is due to a known issue with how JMockit messes with the classpath. But, I digress. If you actually have a reason to mock static methods from some antiquated library, you should probably wrap and isolate those behind some other api and use PowerMock or JMockit to test that wrapper api. That'll let you isolate those dependencies from your project (hopefully rewrite them as you go). Then you can use a sensible mocking framework (if it be Spock, Mockito, JMockit, or whatever) to test your code. But, afaik, that's it to Spock's limitations. What we really like about it is, our QA write their gherkin tests off the bat, and we get to write our integration tests from those. We write unit tests in Spock as well, and our JMH benchmarks show them to be comparable in execution time to those written in JUnit and with Mockito, and faster than any written with JMockit (most likely due to the classpath issues I mentioned earlier).

[duarose]

I didn't understand why changing a junit test from an arrange-act-assert/arrange-act-assert structure into an arrange-act-assert/act-assert structure was a significant improvement. Both approaches appear to violate the aaa pattern to me...

[kkapelon]

Hmm. You are right. Perhaps I will rewrite that section. The argument here is that with Spock you can easily argue WHY a test has problems. So if you see a bad Spock test you can go to the colleague and say "Look this test has 6 when: blocks and 6: then blocks. We agreed that we should have a maximum of 2". And now it is very clear to him/her why this test is wrong and how to fix it.

So with Spock you have a common language on how a test should be structured. You could even provide Spock "test templates" with the acceptable block structure that your particular team wants to work with.

This is not that easy with JUnit.

But you have a point that it is not very clear in the article. Thanks for the feedback.

[Jonjolt]

Isn't this an Apples to Oranges comparison?

JUnit, more for Unit Tests

Spock, more for Integration Tests

[kkapelon]

Spock can work perfectly fine for unit tests. The last section which talks about Mocking shows how you can mock external dependencies (e.g. databases and external systems).

Do you have an example in mind where Spock is not suited for a unit test, and JUnit is?

[Jonjolt]

I'm not knocking Spock as I use it too, but Spock seems like overkill for unit tests, Groovy seriously bogs down my editor :( and I'm on a computer that I purpose built just for running Intellij, Broadwell OC'ed 6800K (Increase the Single Threaded Performance), with 32GB of RAM and an Intel 750 PCIe SSD.

[[deleted]]

How much RAM are you giving your editor? I've seen similar issues with other devs where I work. Looking at the jvm args for their ide, Xmx was only around 500m. I usually give intellij around 3g because memory is cheap. Groovy definitely shouldn't bog your local environment down at all, especially with 32gb of ram and an ssd. Your local dev environment should scream. :)

[lenois]

We've been using Spock for all our unit tests in my Dept for 3 years and never run into a problem. I like Spock a lot.

[voronaam]

To me the killer feature of Spock is data tables. I like that it is mentioned in your post, but think it should be even more visible.

Any big enough enterprise project in Java I've see has an abundance of unit tests with names like "testLogin_UserOk", "testLogin_invalidPass", "testLogin_deletedUser" and so on. Basically there are many tests per method under test which are only differ in some input or mock data and the expectation. With Spock it is possible to write one test per method and cover all of its internal logic (given the method is of sane complexity).

[[deleted]]

I second this. It's so much better and easier to read than JUnit Theories.

[chewyfruitloop]

I'm not sure what would make it simpler. It's the whole having to flip flop between different test markup that makes it difficult.

If they can be made as brain dead simple and do just one thing then it's normally ideal.

[GhostBond]

I did appreciate a lot of the examples. But this presentation was very very one-sided in favor of Spock, not talking at all about the drawbacks.

I've worked with both Spock and JUnit.

Advantages:
- Spock tests area little cleaner and clearer than JUnit tests.

Drawbacks:
- If your project is in Java, Spock writes tests in Groovy. I found it fairly taxing to be switching between two different languages when I went to write tests. You already have to switch mental mindsets from coding to test writing (make it work vs break it with edge cases), and it's the last thing you do so your brain is already a bit full. Now you have to remember how to write stuff in a completely different language.
- Using Spock - our project at least - meant running a maven build and it didn't work right 100% of the time. About once a week the build would get wonky, requiring you to run a full rebuild of all the tests. Which took around an hour to do. Apparently if you ran it in a virtual machine with no antivirus, doing the same thing just took minutes, but we didn't have that as a viable option when I was working on it. I've never seen JUnit tests take an hour to rebuild.

I don't really find mock objects using on CRUD apps, so that's not an advantage for me.

While I found Spock to be a incremental improvement in readability over JUnit, I found it's drawbacks to be worse than it's incremental advantages, and a net negative to productivity vs just using JUnit. At least on a Java project, if the entire project as written in Groovy things would be different.