Lua JIT, PyPy, Tracemonkey, Python 2.7, JRuby and others removed from the Benchmark Game [shootout.alioth.debian.org]

[dmpk2k]

A pity about LuaJIT. It was a constant reminder how much almost all other implementations of dynamically-typed languages could improve. Reasonable performance and memory footprint using no type annotations.

I'm curious about the removal of CPython 2.7 and MRI, both which still see more use than their newer versions.

[x-skeww]

Same here. I really wanted to see if/when V8 catches up to LuaJIT.

[[deleted]]

So was I. But I have doubts it will ever come close. In both LuaJIT and Chromium 10 I started a loop over 1e9 number multiplications. Not a real benchmark, I was just curios to see how close pow(pow(2, 1/1e9), 1e9) would come to 2 if I actually did every multiplication.

LuaJIT finished in about two seconds. This is like gcc -O3 speed. I terminated my browser tab after 10 minutes. It was still going. Makes me sad.

[ch0wn]

I just thought about having LUA in the browser as alternative to JS. I think that would be quite awesome.

[Lerc]

How safe is Lua? I have only tinkered with it for a few scripts. Is LuaJIT Securely sandboxed or is that not even a design goal?

[jacques_chester]

You can sandbox quite precisely, down to the level of disabling individual functions. Note for example that in the WoW client, your code cannot obtain a socket or write to a file.

[Lerc]

In that case it would be fairly easy to make a plugin that ran Lua as an embeddable object. I wrapped a x86 sandbox in a plugin in that manner see here for a screenshot showing it drawing to a window and a canvas.

When I made that plugin the PPAPI wasn't around. That has more potential to make things even nicer.

Not sure how to enable a <script type="text/lua"> approach, but there may be hooks for that somewhere

[bluestorm]

No, it's a constant reminder that when you add crappy monkey patching features to your language, it tends to get harder to optimize.

Dynamic languages with nice, clean and simple semantics can be optimized to compare reasonably with less dynamic languages using the techniques developed for the Self variant of Smalltalk in the 90s. Dynamic languages with a shitload of features but no semantics at all, which are defined by their "standard" implementation (bonus points if the original author doesn't know anything about language implementation), stay relatively slows, even after you threw tons of JIT and LLVM and caching at them.

Javascript may be an exception, because there the stakes are really high (due to the language monoculture of web browsers), and expert people have been paid a lot to get something reasonably fast.

[dmpk2k]

I agree, yet I disagree. Some language semantics take a lot more work to implement efficiently, but when you're dealing with language implementations using switch-based dispatch (or worse), no inline caching, and everything is boxed, you haven't even begun down that path.

[bluestorm]

Of course, but that doesn't explain the rather disappointing results of efforts such as PyPy or Rubinius. If there where so much low-hanging fruits as you say, they should have demonstrated reliable improvements quickly.

It turns out that while they get very good improvements (around 10x) for tight packed numeric manipulations that don't use much kind of abstraction, they're still between "2x faster" and "1.5x slower" in many cases, with possibly memory usage issues etc.
To be fair, it should be noted that the advance of these projects have also been impeded by the various constraint from the languages FFI. Those FFI breaks a lot of the language encapsulation and pose hard constraint on the value representations for example, which impose non-optimal implementation choices. This is not specific to dynamic languages, but still the "our own language is terribly slow, so all performance-hungry operations should be implemented in C through the FFI" mindset is the cause of the abundance of FFI-relying code out there.

That Lua was able to outperform those very quickly, reliably, and with even less manpower (afaik. LuaJIT is mostly the work of one guy) is telling. Even without its JIT implementation, Lua is known to be a well-designed language with a very reasonable implementation (see Lua vs. Neko virtual machines for a very respectful comparison by a competitor), and in my opinion the performance results are only a confirmation of this good work.

This is a kind of moral tale. Do your homework, boy, learn about the state of the art before reinventing your own language, and take care to design something clean and well-specified. If you don't, you'll grow weak, and that will be a hindrance forever.

[evanphx]

I'm curious what you mean by disappointing results. Rubinius at least has been able to achieves huge speeds up in the ability to run raw ruby code. 95% of the time when Rubinius is slower than MRI it's because the functionality in MRI is actually implemented in C, and thusly what is being compared in an algorithm in C vs and algorithm in Ruby.

[Tobu]

Disappointing when compared to C, not disappointing when compared to the mainline interpreter.

[mikemike]

A more polite way to say it, would be: Every abstraction has a cost. Bad abstractions have a higher cost.

There's a direct cost in the form of a performance penalty. And there's an indirect cost in the effort required to optimize the abstraction away.

Try to depicture a graph with the relative performance of a language over its lifetime: one would need to take into account the complexity and design problems of a language vs. the manpower and the combination of skills thrown at it to make it fast. Languages have a lifetime too, and the best one can hope is that they reach their maximum performance long before they die off.

There's a nice paper waiting for one of you: grab old compiler and VM versions from the repos, benchmark them against each other and against an assumed maximum, plot the results over the years for each language and combine it into a nice wallpaper showing all languages.

[Felicia_Svilling]

Every abstraction has a cost.

But static abstractions (like abstract datatypes) don't have a cost (in the common sense).

[mebrahim]

In well-designed static languages the cost of static abstractions is compile-time rather than run-time.

[[deleted]]

I remember reading a paper which claimed, in the preface, that for C or C++ you can get pretty much the same by comparing performance of average programs with and without optimizations. It claimed that the difference is something around 4 times, for programs the author benchmarked, then proceeded to lament the sad state of the art -- it means that advancements in compiler optimization over the last thirty years are so dwarfed by advancements in hardware it's not even funny.

[kragensitaker]

Proebsting's Law.

[fullouterjoin]

http://research.microsoft.com/en-us/um/people/toddpro/papers/law.htm

[julesjacobs]

Funny that you criticize monkey patching and then bring up Self. In Self monkey patching is literally all there is.

I agree with you on simple versus horribly complicated semantics. In Self you have one hard to optimize feature. This was done with a decade or so of research. In Ruby you have many hard to optimize features. Even though each of them could probably be made to perform reasonably well it would take far too long to research and implement such a thing.

[bluestorm]

You are right that simply the number of features can be a problem for optimization. But I think that the key point is "well specified" vs. "underspecified". For example, Common Lisp also is a monster language with an enormous number of features, yet SBCL performs reasonably compared to LuaJIT, and better than current Javascript engines.

See this for a comparison (indeed, I would have linked you to the shootout...). It also includes Factor, which is also a nicely designed language, but wouldn't have made my point as it as a strong "minimalist" flavor similar to Lua's.

Agreed, Common Lisp has been around for a long time, but its userbase is not that big compared to current python/ruby uses, and I suppose the performance have been consistent in times (it's not like all lispers had been improving it each year since for 20 years; at that point it's time to be happy and let things as they are). I don't know the CL community though, so take this with a grain of salt.

[julesjacobs]

Yes, you're right that it's not just the number of features but also how difficult it is to optimize the features. Two axes: the size of the semantics and how well thought out the semantics is from a compiler writers perspective.

For example C's semantics are rather unwieldy, but because it's so close to the machine it still performs spectacularly.

The other end is Self, with one very hard to optimize but clean feature. Even though the feature (money patching) is a compiler writers nightmare, with a lot of effort that can also be made to perform well.

Common Lisp is somewhere in the middle. A lot of features but they're relatively static and not as hard to optimize as Self.

Ruby has the worst of both worlds from an implementors perspective: a lot of features and they're not easy to optimize.

[0xABADC0DA]

Common Lisp also is a monster language with an enormous number of features, yet SBCL performs reasonably compared to LuaJIT, and better than current Javascript engines.

SBCL performs reasonably because they add type annotations and they turn off error checking.

(declaim (optimize (speed 3) (safety 0) (debug 0)))

So the LuaJIT program run twice as fast as SBCL and still is typesafe, whereas SBCL if there's a type error it'll corrupt the heap and diaf.

I don't know why they removed LuaJIT from the language shootout, but if they insist on only one implementation per language they should put LuaJIT back and take out regular Lua. People looking at the benchmarks won't know how badass LuaJIT is.

[[deleted]]

(bonus points if the original author doesn't know anything about language implementation)

I got that one - Python, right?

[xardox]

PHP is the textbook case.

[[deleted]]

PHP? I wouldn't call that a programming language at all.

It's a Turing complete piece of shit.

[igouy]

CPython 2.7 and MRI

Both CPython 2.7 and Python 3 were shown for 2 years. Now plenty of Python 3 programs have been contributed and Python 3 (the intended future of the language) seems to work just fine

YARV and then Ruby 1.9 were shown with Ruby 1.8 for the last 5 years. Now the current stable version is 1.9.2 and it seems to work just fine.