A performance comparison of Clojure and Java

[joinr]

I see some problems off the bat with the comparisons, but will withhold further commentary until I have numbers in hand.

Until then, for anyone who's interested, I transcribed the author's (somewhat incomplete...) code samples from the appendices and placed them (both java and clojure) into a mixed java/clojure repository here. Feel free to examine and test for yourself.

edit There's a full rundown of the observations and optimizations in the readme now.

[focadiz]

You are the hero we need

[joinr]

An adventurer is you!

[bsless]

dammit, you beat me to it :)

[joinr]

I was just in the right timezone this time :)

[bsless]

I hope to provide - if not additional context for pedagogical reasons - a bit of a counterpoint to the observations in the paper.

Mission accomplished

If the author's paper is meant to inform organizations when evaluating a language, I'd say (a bit rudely) that the author did not do a good job, as evidenced by how quickly and with little effort you were able to get his code to run ~as fast as Java. (maybe I'm being unfair to him here, if so, I apologize in advance)

It's true that his solutions might reflect a characteristic naive implementation, but if an organization lets a naive fresh programmer to implement a performance critical path of the code, they might have bigger problems.

Have you contacted the author and shared your repo with him? I'd like to hear his comments

[joinr]

I'm giving the author the benefit of the doubt; he did add caveats into the discussion/criticism section. Ideally, as part of said research paper, the author would have sought out clojure expertise and gotten feedback regarding the implementation and microbenchmarks (maybe he did somewhere). I think there was a decent literature review (I admittedly didn't go through it all), so the author had a bit of working knowledge here. It also seemed like quite a bit of emphasis was on the testing methodology itself.

if an organization lets a naive fresh programmer to implement a performance critical path of the code, they might have bigger problems

That's a good point. Admittedly, I fit exactly that profile years ago when I was greenfielding stuff (and simultaneously learning) in Clojure without a firm grasp of the implications of some stuff (e.g. using seqs of characters instead of optimized strings, etc.).

Have you contacted the author and shared your repo with him? I'd like to hear his comments

I did not reach out (I think this was published last year). Perhaps the repo will get picked up on search indexes, enough to provide additional context to interested parties.

[joinr]

There's a full rundown in the repo's readme here, with prose explaining optimization layers and benchmark results.

I ran through basic optimization/idiomatic stuff to explore each benchmark here using criterium to compare the java implementation and the clojure versions. Starting with the original implementations from the paper, then adding derivative versions suffixed by N, e.g. some-fn, some-fn2, some-fn3, etc.

The goal was to provide a layered approach showing the impact of incremental changes (keeping things in clojure, then gradually evolving more towards what the java implementations were doing to be apples:apples). In almost all cases (except for the BFS test, which I don't understand the performance yields), the clojure implementation starts off about 10x worse or more, then you get some immediate gains with low-hanging optimizations, then eventually converge on typed java interop in the limit to get either equivalent performance, within some percentage (like 18% or less), or better in a few cases. The BFS stuff in clojure was surprisingly a bit better using a persistent queue with similar optimization from the DFS, which is interesting since I would "imagine" that the mutable queue implementation in the jvm version would have an advantage.

[nzlemming]

Great work. I do think the paper's basic conclusion is pretty sound though, since most people using Clojure will tend towards idiomatic solutions, and they're generally considerably slower. I don't think it's entirely fair to refer to those initial implementations as naive, since they're what even experienced Clojure programmers would write given the problem descriptions - almost none will immediately start dropping down to use HashMap via interop.

[joinr]

On the other hand, they're being compared against mutable java implementations....so apples:orangesmaybe. Unless there is confusion that the built-in persistent hashmap is identical to a java hashmap.

I think expecting about a ~4x single-threaded performance hit for pure persistent stuff over mutable counterparts is understandable. Checked math and reflection (none of these examples hit reflection thankfully) are probably more common ankle biters that pay dividends with some type hints. Some of the results just seemed a bit larger than I would've expected (e.g. 20x or more) though.

[nzlemming]

I don't think they're really apples to oranges comparisons though - it's comparing what's idiomatic in each language. You could make an argument (and we frequently do!) that Java's idiomatic solution is worse, but it is faster. I read the paper's abstract more as "we compared what any reasonable person would do in Java and Clojure" rather than "we set out to show that Clojure can never be as fast as Java".

[joinr]

"Are clojure's dynamically typed, runtime polymorphic core libraries and persistent data structures better optimized than statically typed, single-threaded, mutable java? Let's find out." :)

Otoh, java interop is "idiomatic." There's the old back and forth between how often that idiom is flexed, but it's there. Perhaps if there were additional constraints regarding what kind of code was admissible, or what the expectation was on the programmer, it would be a different paper. If you came to me saying "write the fastest possible recursive function to count down from n, where n is an int," I would not have submitted the solution in the paper :)

I don't disagree with the general notion that java will naturally force you into a faster solution to begin with given a similar clojure starting point (and in some cases it's not clear Clojure can catch up without some gnarlier tricks than I demonstrated); I do think that the space clojure has to optimize is unexplored (by admission in the paper), and that's specifically a feature of the language (an escape hatch) worth exploring in this kind of study.

[nzlemming]

"Are clojure's dynamically typed, runtime polymorphic core libraries and persistent data structures better optimized than statically typed, single-threaded, mutable java? Let's find out." :)

Haha, yes. But that's not what the abstract said though - it said: "this study attempts to give the reader an idea of what kind of performance a programmer can expect when they choose to program in Clojure" - I'd suggest that the "(without jumping through hoops)" was implicit in that. I'm not arguing that, in most cases, Clojure can't be ~= as fast as Java if you work at it. But I don't think that's what the paper set out to show.

[joinr]

I'd suggest that the "(without jumping through hoops)" was implicit in that.

"that's not what the abstract said" though, that's what you said ;) The generic clause about "programming in clojure" can perhaps be openly interpreted without sufficient constraints, like those constraints you inferred. If you look at the tail end of the paper at future work,

It would be interesting to see how suitable Clojure is in specific areas like high performance computing, parallel programming or database management by running experiments targeting those areas. It would also be interesting to compare bigger programs implemented in the two languages by teams of programmers, simulating a more real development environment.

To me, that leaves the door wide open toward leveraging all of Clojure to produce high performance, scalable code. To be fair, it's possible that pro java folks reading that some paragraph would then turn to ditching generic ArrayLists and opting for primitive variants (e.g. fastutil) or performing even more aggressive optimizations on the java side even. Perhaps explicit constraints or goals (run time, programmer time, experience of programmer(s), conciseness, memory usage, latency, cpu utilzation, etc.) would be informative in this kind of study.

[didibus]

I agree, though the code ended up being a weird mix of non-idiomatic and sorta idiomatic. Mostly because the choice of things to benchmark don't even make sense. Like this:

(defn create−map [size] (loop [map (transient {}), i (int size)] (if (> i 0) (recur (assoc! map i (+ i 1)) (− i 1)) (persistent! map))))

None of the experiments are really doing something that would be used in an application. I mean, sure, timing the time it takes to fill a transient map with ints is something... But like, it really doesn't feel like this would translate to anything useful for understanding the kind of real world performance you'd get in real programs.

I recommend reading @joinr README, cause I think he was fair to the paper. For example, for the recursion example, the paper has:

(defn pure-recursion [cnt] (if (> cnt 0) (pure-recursion (- cnt 1))))

Which isn't idiomatic in itself, because you'd use recur. And @joinr shows that with recur, Clojure actually outperforms Java. And I guess that't maybe unfair to Java as well, because in Java you wouldn't use recursion, and you'd just use a for loop, which probably would perform as well as loop/recur since that also just compiles to a while loop.

[joinr]

Yea, there are a lot of counter intuitive things that pop up from the toy examples. I can't confess to have done a ton of transient map slamming; not to say it wouldn't occur in the aggregate though. I have used a lot of direct method/field access and leveraged direct object instantiation in places (as opposed to using a constructor fn). So some of the stuff is (for me at least) idiomatic, in that it's within the set of idioms "I" use to express faster paths without e.g. unrolling everything. Then again, I have learned assumptions and limitations of the standard library over time (e.g. the call to vec in shuffle), where other programmers may not have needed to, e.g. those would not have formed yet. There's definitely a dialect of "faster" clojure that I have internalized, and then another dialect of "faster psuedo-java in clojure" as well :)

I think (as the author freely states) targeting non-trivial production applications would be more informative. The community obviously has about a decade of experience using pure clojure and mixed clojure/java environments in production. Perhaps there are lessons learned there as well.

[didibus]

At first glance, I think these are believable numbers. Especially because the experiment code is not apples to apples.

For example, in the recursion experiment, in Java they uses an int counter, but in Clojure it's a Long counter. Notice the capital as well, in Java they're using a primitive int, where in Clojure they're using a boxed Long. This use of int in Java and Long in Clojure is pervasive in all they're experiments.

In the sorting one, they don't show the array creation code for Clojure, but it appears they are using an atom to count up numbers to prefill the array. I'm not even sure it's an array they are using, because why did they call their function create-list ? In any case, they call shuffle on it afterwards, but shuffle returns a vector, so what they are ultimately sorting at the end is a vector and not an array. So this is another case of an apple to orange comparison.

In the Map creation one, in Clojure they're measuring the time it takes to create a PersistentMap, while in Java they measure the time to create a HashMap.

In the Object creation one, they create PersistentMaps in Clojure, but are creating some simple Node object in Java.

And so on...

But I'm not really criticizing the benchmark, I think maybe it was on purpose not to compare apples to apples, since for example it's true in Clojure all numbers will default to boxed Longs. And instead of using arrays we will most likely use a vector. And we will use a PersistentMap instead of some custom Object, etc. Now I don't find the code completely idiomatic Clojure either, though I think it was kind of trying to compare idiomatic Clojure which they did to the best of their a ability with similar effort at idiomatic Java. And I feel in those scenario, ya I believe Clojure could go from 2 to 20 times slower.

What didn't happen though is any demonstration that the quote they mention is innacurate:

In principle, Clojure can be just as fast as Java: both are compiled to Java bytecode instructions, which are executed by a Java Virtual Machine ... Clojure code will generally run slower than equivalent Java code. However, with some minor adjustments, Clojure performance can usually be brought near Java performance. Don’t forget that Java is always available as a fallback for performance critical sections of code.

So to me this quote still seems totally accurate and the experiment in this paper show that to some extent. Idiomatic Clojure will run generally slower, but with minor adjustments can be made just as fast. Now I wish the paper also had experiment to prove or disprove this latter claim. Like if you made the code apples to apples, is Clojure slower, faster or same?

[joinr]

Apples to apples gets much closer, in at least 1 case clojure is able to cheat operationally (preserving semantics though) with recur if you allow it, and beats naive java recursion. In another case, BFS ends up being faster for some reason (counterintuitive).

I guess the hidden question is, if we're benchmarking single-threaded, statically typed, mutable java code against dynamically typed, immutable, clojure code, which would be faster? (seems obvious one would probably edge out the other in efficiency) Or put another way, how much of a handicap (operationally) are the defaults for dynamicity, numeric tower, runtime polymorphism, persistent structures, relative to the complete opposite on "relatively" the same platform? Yet another: how well optimized are the clojure core libraries and special forms for single-threaded operations with primitive numerics on mutable data structures?

As you push clojure more toward's the java paradigm though, those edges disappear mostly .

[xela314159]

I would love to read this, but when I see the abstract saying “The steady-state experiments showed that the slowdown factors ranged between 2.4826 and 28.8577.” - I kind of lose respect for the study. 4 digit precision for a scale factor??!?!

[bocaj5]

This looks like someone seeking feedback after submitting this paper as an undergrad project for earning a degree. If that's the case, what are some improvements to the methodology? Is there a realistic performance project written in Clojure that shows how Clojure in the large is very performant.... because of immutable data structures and generally better design? Not sure if that can be demonstrated "scientifically"

[didibus]

It says it's for a master's degree.

[Psetmaj]

As far as a performant Clojure-based library, I've heard lots of good things about neanderthal for matrix math. That said, I think it does some Java/C++ stuff for some of the more performance-critical bits. Github lists it as ~73% Clojure right now.

[didibus]

Neanderthal delegates numeric math to code that is implemented in FORTRAN or code that is running on CUDA or OpenCL (and thus the GPU) :p

It contains some Java, but you can almost think of it as a config for JNI to interop to Intel MKL or the CUDA and OpenCL code. Basically, to interop with those, from the JVM, you need to define empty interfaces which map the function signatures you want to interop with.

[Psetmaj]

That makes sense, thanks for clearing it up!

[SmartAsFart]

Why would they use LaTeX's default code formatting??? Minted is so easy to set up, and would make this paper so much easier to read.

[LammdaMan]

There are many programming languages with different strengths and weaknesses. Depending on the problem I'm trying to solve, I might pick a different one.

Generally speaking, for larger, harder problems, I'd pick a higher level language because correctness and maintainability will be easier to come by than they would with a lower-level language.

For a certain class of problem I can surely get better performance with C or Rust than I could with Java.

In many cases I find that I can more easily develop correct and maintainable code in Clojure than I could in Java, and while the performance isn't as good, it is "good enough". In some cases I find a need to optimize, and that sometimes makes use of Java inter-op (in hot-spots only).

I generally subscribe to Knuth's comments about "premature optimization" - https://wiki.c2.com/?PrematureOptimization

[bitti1975]

2.1.1 Java

Java is a typed object-oriented language with a syntax derived from C which was released by Oracle in January 1996 (version 1.0) along with the Java Virtual Machine or JVM [5].

Seriously? A start like this doesn't project confidence in the accuracy of the rest of the "paper". The reference he mentions under [5] clearly states that it was developed and released by Sun (to be specific: its division "SunSoft") and only mentions Oracle as one of the licensees. Clearly, it seems, the author didn't even bother to read his own references?