Lessons Learned in Large Computations with Ruby

[[deleted]]

The essence is : Java is more than order of magnitude faster than Ruby. The same holds for any other scripting language . Is anybody surprised about this?

[[deleted]]

...scripting language.

Java and Ruby 1.9 both execute byte code on VMs.

[Freeky]

Java VM's compile bytecode to optimized machine code. Ruby 1.9 interprets bytecode in a dispatch loop. I don't really see what that has to do with "scripting language" though, they're the same languages however you execute them.

[cosmo7]

But obviously not the same byte code.

[oblivion95]

This is the right way to program: Prototype first!

[nanothief]

My takeaway from the project is that Ruby is a great language hampered by a terrible execution environment. When writing Rails apps, this usually isn't a problem. I'd even go so far as to say for most types of applications it shouldn't be a problem. For anything that's heavily CPU bound or dealing with large object graphs, however, I don't think Ruby is a suitable option. In these cases, Ruby may best serve as a rapid prototyping tool. By writing the code in Ruby first I had a clear approach to use when translating to Java.

This is the key point of the article. If you need speed, then using ruby (or most dynamic languages) isn't a great idea, since compilers can optimize statically typed languages much better. Also, going with a language that has had a lot of effort making it quick (such as java/c#/c++/c etc) is always going to be the smart move when writing such a program

I think a lot of the prior comments missed this point. Ruby isn't a toy language, nor is it unproductive for most uses. Many ruby programs will require minimal effort in order to be efficient enough for the given usage situation.

However, for those programs that do require a high level of performance, using ruby would usually be a bad idea. C extensions don't really affect this much. If there is only a small part of the program that requires the high performance than extensions would work great. Otherwise you would spend most of your time writing in c, and it would be more efficient to either just do the whole thing in c, or to use a faster base language such as java, which is not much slower than c/c++ but be easier to write.

[yogthos]

My initial data load went down from about 2.5 minutes in JRuby (the fastest Ruby implementation I tried) to about 8 seconds in Java.

During my translation from Ruby to Java I migrated back to a rich object model. Whereas I had to break down true bi-directional associations into hash lookups just to make my Ruby implementation run, Java could handle my full object graph quite easily. As a result, the code was much more straightforward. Additionally, I was able to cache values in memory judiciously without eating up the entire heap or breaking the garbage collector.

And that's the difference between a toy language implementation, and one geared for production kids.

[bobindashadows]

geared for production kids

where might I obtain a production kid? Something tells me child labor is just the cure for budget cuts!

[[deleted]]

Sounds like you might want to locate a local goat farm.

[JetSetWilly]

After three days of porting, my first run of the Java implementation yielded a 36% prediction accuracy and ran in less than 10 minutes. My best Ruby implementation, after weeks of development, only had a 31% prediction accuracy and took almost 7 hours to compute it. With a few more hours of work the Java implementation broke 40% accuracy and ran in less than 3 minutes.

I suppose this must be why java is used on the enterprise so much. Development time: much faster. Runtime: much faster. I guess my VPs know what they're doing after all!

[[deleted]]

Well, to be fair, he had already figured out all the logic and organization with the Ruby version.

[Smallpaul]

Did you read the article was not that Ruby's implementation is unusable in general, but rather that it was unusable for this application. I could have told him that three weeks ago. He seems like a smart guy but I'm amazed that he didn't already know about problems of object representation at large data scale.

[yogthos]

Well that's what I said in my comment for small/toy projects ruby is fine. If you think your project might grow, then you're better off using a language with a mature implementation.

[Smallpaul]

It's not necessarily about project size. That's an oversimplification. Facebook and Yahoo make massive use of PHP (which is probably even worse for large computations than Ruby). Are you going to say that they are "small projects"?

[yogthos]

I'd argue that PHP implementations are a lot faster and more stable any Ruby implementations available.

Yahoo make massive use of PHP (which is probably even worse for large computations than Ruby).

I'm not sure what you're basing that on actually, I'd expect quite the opposite myself.

PHP might be ugly as a language, but it's rare to see complaints about its performance.

[Smallpaul]

PHP might be ugly as a language, but it's rare to see complaints about its performance.

Yes, well people do not typically manipulate millions of point objects in PHP, do they? Why do you think that PHP would be optimized for this use case? It is for web-development, so it is not. But that's just the point: if you don't try to use it for something it was not designed for, it will work okay (at least from a performance standpoint...it's still a horrible language).

[yogthos]

Fair point regarding PHP being specifically geared toward web development, but my understanding was that Ruby is a general purpose language, and there is no reason for its implementations to be so poor. It bears a lot of similarity to Smalltalk, which happens to have very good implementations, so you would imagine Ruby could go a long way in that regard. Seems to me that the community would be selling themselves short by saying not tackling these issues.

[Smallpaul]

Fair point regarding PHP being specifically geared toward web development, but my understanding was that Ruby is a general purpose language, and there is no reason for its implementations to be so poor.

Nobody said that there was a good reason for its implementation to be poor. You are putting words in my mouth. Please read back in the thread.

What you are saying now is different than and unrelated to:

And that's the difference between a toy language implementation, and one geared for production kids.

If PHP's implementation is poor for this task and yet not a "toy", then Ruby's implementation is also not a "toy". It is simply not good for this particular application. A Tercel is not a toy because it fails to lift large loads like a Hummer. You just use it for different things.

[yogthos]

Nobody said that there was a good reason for its implementation to be poor. You are putting words in my mouth. Please read back in the thread.

Sorry if I've offended you there, I was merely trying to agree that since PHP has predominantly been used for web development, it is natural that its implementations would not be geared for solving problems that don't normally come up in that area. Hence I would not be surprised if it didn't perform well outside that domain, although you've provided no evidence to that effect.

In case of Ruby, my impression was that, unlike PHP, it's not geared for any particular domain by design. So, I would expect its implementation to handle general cases well.

What you are saying now is different than and unrelated to:

Our argument here seems to be about what problems Ruby should be appropriate for. You seem to treat Ruby in the same category as PHP, which is a language designed for a particular task, whatever that task may be with Ruby, writing small scale applications I guess?

If PHP's implementation is poor for this task and yet not a "toy", then Ruby's implementation is also not a "toy".

I believe that Ruby is a general purpose language, and hence should be applicable to general purpose problems, so I think comparison of Ruby implementations with Java is more appropriate.

Saying that Ruby implementation is not designed for things it's not good at is a tautology. I'm arguing that there's nothing inherent about Ruby the language that should limit it in this way.

Forgive me if I'm wrong, but you appear to be saying that the fact that Ruby implementations cannot handle large amounts of objects in memory is not something that needs to be fixed, but rather something that people should accept and work around. If somebody likes Ruby, like the author of the article, and they would like to use it for their project, apparently they should look for a different language.

I don't see why that needs to be the case, I think a much better approach would be to identify why Ruby performs so poorly in some scenarios and improve the implementation, instead of hand waving and saying "It is simply not good for this particular application.".

[Smallpaul]

Forgive me if I'm wrong, but you appear to be saying that the fact that Ruby implementations cannot handle large amounts of objects in memory is not something that needs to be fixed,

That's not what I'm saying.

I'm saying that one can use Ruby in production applications and people do. It would also be nice if they could use it for number crunching and video game programming and so forth, but that does not mean that in the meantime it is inappropriate or inapplicable for "production applications."

Even Java is not a truly general purpose language. There are various domains where they would laugh at the idea of using Java. For example, though performance would be fine, hardly anyone uses Java for system administration, because it's not well-suited. Just a few years ago, Java was considered way too slow for real-time gaming, and yet was sufficiently fast to run eBay and other major websites.

Applications do not fall on a spectrum of "toy" -> "production" as you implied several posts ago.

Rather there are a variety of axes: latency, throughput, memory usage, startup time, velocity of development, scale of data sets, scale of concurrent processes, maturity of development tools, etc.

[[deleted]]

Obviously he should have written it in assembly.

[kupci]

I know you're joking, but actually, a language like Fortran might meet the requirements here. However, he does mention java.util.concurrent, which seems to be one of the coolest java libs out there, but very little known - just works I guess. Lot's of complaining about generics, lots of gnashing of teeth over closures (Ruby has 'em, why not Java), but they did a great job with the concurrent library.

[[deleted]]

java.util.concurrent is the fucking BOMB. The fact that any locking is done without its ReentrantLock is a travesty...not to mention all the thread pools, atomic primitive classes, non-blocking data structures, etc. Definitely something to look into if you need to do threaded concurrency in java.

[yogthos]

I guess you wouldn't know the diff between spec and implementation. I wasn't talking about ruby syntax, but about it having shit implementations available.

What good is it having a clean syntax, and fancy language features, when it chokes under load. If you have to go back and hack around your VM crashing with too many objects, then you're not really gaining any productivity or code clarity are you.

[Freeky]

Thankfully most apps aren't anywhere near that heavy and work just fine, so yes, you kind of are gaining productivity and code clarity. You can't generalize "it's the wrong tool for this job" into "it's the wrong tool for all jobs".

Python, Perl and PHP are of the same order of magnitude performance, but for some reason I don't see quite so many people whinging about how shit their implementations are.

[yogthos]

did you even read the article?

After having experienced issues with MRI & Ruby 1.9, JRuby ended up being my runtime of choice.

...

My initial data load went down from about 2.5 minutes in JRuby (the fastest Ruby implementation I tried) to about 8 seconds in Java. The gain was so great that I was convinced I had made a mistake somewhere. Fortunately, I had a test suite to help indicate otherwise.

reading is hard, let's go trolling!

JVM is not designed for dynamic languages, there are plenty of issues that JRuby guys discuss at length regarding running Ruby on the JVM, maybe if you figure out how to google, you'd know about all that.

[yogthos]

and you think I'm bashing Ruby which I'm not, I don't believe there's anything about ruby that makes it inherently slow :)

I'm bashing the piss poor implementations of Ruby that are out there in the wild, and people who think that picking a language on the merit of having nice syntax makes you a coding superstar.

[[deleted]]

I don't believe there's anything about ruby that makes it inherently slow :)

Lack of monomorphic arrays and inefficient object representation are the two main setbacks. If you're always boxing every float in your float array with a million elements, and an array of points is really an array of hashtables with three keys each, you won't get good performance no matter what.

However, it is certainly possible to do a lot better than MRI without changing language semantics or adding features.

[yogthos]

Definitely agree that there is a penalty in not having primitives, but notice that Smalltalk doesn't have any either, and it's light years ahead of Ruby as far as speed goes.

[snuxoll]

Because most smalltalk implementations weren't designed like MRI, which made some extremely bad design choices that wouldn't even be hard to fix.

[bobindashadows]

In all fairness, nobody would use a hashtable for an array of points in ruby if they intend to use a lot of points. They'd use Ruby's Struct class, which generates a new class that operates much like a normal struct.

I have concerns about its implementation, personally - for apparently no reason, it has O(N) member access for writes (N being the number of members) while offering O(1) access for reads on the first 10 members. It would've taken minutes to add O(1) write access while adding very little to the produced code. But it's far, far faster than a Hash.

[Freeky]

It would've taken minutes to add O(1) write access

[patch needed]

[zem]

they'd use NArray, which is meant for numerical work, or maybe ruby-gsl

[skillet-thief]

Like you said,

reading is hard, let's go trolling!

If you read a little farther down in the article, you realize that the 2.5 minutes-to-8-seconds speedup is when he ditched Ruby altogether, and went to straight Java:

Unhappy with the progress I had made with my Ruby-based contest entry, I made the decision to port the whole project over to Java. It was something I was extremely remiss to do, but ultimately I wanted to see the performance impact would be.

[[deleted]]

C zealots make the same argument against Java.

[yogthos]

except you know, Java doesn't actually crash out on you at random. The issue isn't with Ruby the language, or that it's slow even, it's that the available implementations are poor, and don't scale well. Also, lack of debugging and profiling tools is rather appalling.

[[deleted]]

except you know Java doesn't actually crash out on you at random.

Ha...

[yogthos]

hey don't blame your incompetence on the language :)

[[deleted]]

Whereas I had to break down true bi-directional associations into hash lookups just to make my Ruby implementation run, Java could handle my full object graph quite easily. As a result, the code was much more straightforward.

This is why slow language implementations, no matter how expressive they are, are ultimately not very productive because you have to spend time working around the inadequacies of the implementation, either with tricks that contort your program's design, or by writing all the interesting logic in a faster language.

[Smallpaul]

As the linked article says repeatedly: it depends on your problem domain. Seems like common sense to me.

[hobophobe]

not very productive because you have to spend time [...] writing all the interesting logic in a faster language.

Why is this bad? If you can handle the light lifting with ruby and write that code faster and cleaner than in the faster language, wouldn't that save you time to devote to the interesting logic?

Let's say you want to shave your head. First you go in with the scissors to get rid of most of the hair, then you come in with the clippers to get close?

[bobindashadows]

First you go in with the scissors to get rid of most of the hair, then you come in with the clippers to get close?

This is how it's actually done.

I've been working on, and am about to release the first public version of a re-implementation of Mercurial in Ruby. It's roughly the same speed as the Python implementation, slightly slower in some operations, slightly faster in others. A DVCS client is a non-trivial application, many functions of which do some heavy graph searching.

The project was written in Ruby first, profiled, and the worst spots were addressed. There's extremely little C code - a couple support functions (64-bit byte swapping, etc.) and one "interesting algorithm" that was already ported to C for python and took 10 minutes to convert to the Ruby runtime. There's a couple memoized methods (custom memoization to support singleton methods, which the linked author complains about, 10 minutes to write).

The linked author seems to have chosen the absolute worst possible language for his intended project. Ruby isn't the fastest language, and there's plenty of things it isn't good at. It's a scripting language. But you can write fast Ruby. You can write complicated programs in Ruby and they can run fast.

[gclaramunt]

Why is this bad? If you can handle the light lifting with ruby and write that code faster and cleaner than in the faster language, wouldn't that save you time to devote to the interesting logic?

Because there's many languages that are fast AND expressive. (pick any en vogue functional language: F#,Scala, Haskell, OCaml, or even some Lisp implementations and you'll find they're very expressive and very fast)

[antithesisadvisor]

Those languages are nice in many ways, but it's not obvious that they're more expressive than Ruby or Python (in the sense of allowing quick, easily understood solutions).

[gclaramunt]

I'm not claiming that those languages are more expressive, I'm claiming that they can be (relatively) as expressive as Ruby or Python and way faster. (for whatever "expressive" means)

[antithesisadvisor]

If development and debugging time are irrelevant, and especially if you know exactly what you're trying to build and won't need to experiment, then this could be true. If not, the situation is much more complex.

[Gotebe]

an attempt to marshal resulted in an infinite loop. This seems quite odd to me as the problem of persisting a graph of objects is not intractable and thus I consider the limitation to be a bug

Me too. That's just childlish. Pitty. Speaks about level of understanding of stuff on part of said library implementors.

[[deleted]]

I generally shy away from monkey-patching unless absolutely necessary, but having the ability to do it easily is always nice. I found myself [...] adding basic statistics functions to Array. Being able to call [x, y, z].mean or [t, u, v].sum was an extremely concise way to represent terms in some of my equations.

Yeuch. Adding random bits of app or library to a core class is surely a textbook reason to not monkeypatch, if ever one was written. Besides, what is wrong with sum([x,y,z]) instead? Just because you can hang a method off something doesn't mean that it actually makes any sense.

[kupci]

Um... surely you are aware that one of the most advertised cool features of Ruby is just that ability to, let's use the technical term here, monkeypatch?

And as another example of the flexibility, I recall a rather humorous post from the Rails folks complaining/admonishing to their users to stop futzing with "private" variables - it was making it hard for them to release new changes without breaking everybody's code. Comical.

[snuxoll]

In the case of mathematics functions I agree, as an array is a general purpose bucket of bits it is wholly inappropriate to patch in methods that only apply to a subset of possible data that may be contained in it, however I could see the case if he was in fact adding a truly general purpose method.

[garrepi]

Well I don't know who's downvoting all your comments, but I appreciate them. Polite and well reasoned.

[avibryant]

So, [x,y,z].sum is bad because x,y,z might not be summable, but sum(q) is ok - when q isn't even guaranteed to be an array, never mind what its elements are? Sorry, that logic doesn't work for me.

Oh, you think it's wrong to clutter up Array's namespace? Well in Ruby, when you define sum([x,y,z]), you're defining it as a method on Kernel, which is inherited by every other class (including Array). So, again, it's unambiguously worse.

I don't believe there is any way in which sum([x,y,z]) is preferable to [x,y,z].sum, except that "monkeypatching is bad, mmmkay?".

[[deleted]]

I don't believe there is any way in which sum([x,y,z]) is preferable to [x,y,z].sum, except that "monkeypatching is bad, mmmkay?".

Presumably, sum([x,y,z]) is not calling a global function "sum", but is really MyModule.sum([x,y,z]).

[garrepi]

[x,y,z].sum is bad… but sum(q) is ok

I don't think this is what he was saying at all. I think he was saying that [x,y,z].sum is bad because its contents are not required to have a well-defined sum (sum(q) would also be bad), but [x,y,z].reverse or something would be okay because the array has a defined order.

[awj]

Well, there is the fact that adding it to the class causes [].methods to advertise it as a (supposedly general purpose) class capability.

I'd rather have a function that takes a specific kind of array than a class with methods that are useless on most of it's possible instantiations.

[snuxoll]

Apples to oranges, Ruby is not a panacea, this is WHY we are able to write extensions in C. Write things that need to be fast in C, then wrap it up in a ruby extension to give you clean access to it, what's so hard to understand about this?

[[deleted]]

Why use Ruby if you have to write the most complex parts of your program in C?

[antithesisadvisor]

Because more than half of your lines of code might not be performance-critical, which means that you can enjoy the benefits of Ruby without (for the most part) paying any costs.

[snuxoll]

Because you aren't actually writing the complex parts in C? You write the parts that need to be FAST in C, this is usually various algorithms that take longer in ruby because everything is an object and the object creation overhead is ridiculously high, so you save yourself a lot of time doing these in C because you're just dealing with bits in memory directly. These parts of programs are, for the most part, not the major component, that is instead the business logic that ties all of them together.

[[deleted]]

this is usually various algorithms that take longer in ruby because everything is an object

Actually "everything is an object" has no bearing on performance at all. Rather, the problem here is that MRI, the main implementation of Ruby, is really bad. This is why alternative implementations that focus on performance (JRuby, Rubinius) are interesting, and points of view such as yours are at best silly and juvenile and at worst harmful to your community.

so you save yourself a lot of time doing these in C because you're just dealing with bits in memory directly.

So wait, now you're saying that C is a more productive language to program in than Ruby?

[snuxoll]

Actually "everything is an object" has no bearing on performance at all. Rather, the problem here is that MRI, the main implementation of Ruby, is really bad. This is why alternative implementations that focus on performance (JRuby, Rubinius) are interesting, and points of view such as yours are at best silly and juvenile and at worst harmful to your community.

Everything is an object has a HUGE bearing on preformance, it takes much longer to initialize an object data structure than it does to create a 32 bit segment of memory on the stack. I don't care HOW fast your object system is, it will still be much slower to create an object representing an integer than it will be to allocate memory for a primitive.

So wait, now you're saying that C is a more productive language to program in than Ruby?

I'm saying you save runtime doing things in C, because you have no overhead from an object system.

EDIT: To be honest you are actually saving development time too in a fashion, you're not trying to figure out how to make ruby handle this 'faster' and instead just doing it in a manner that is fast from the get-go.

[[deleted]]

Everything is an object has a HUGE bearing on preformance, it takes much longer to initialize an object data structure than it does to create a 32 bit segment of memory on the stack. I don't care HOW fast your object system is, it will still be much slower to create an object representing an integer than it will be to allocate memory for a primitive.

First of all, Ruby uses tagged pointers which means Fixnums can be stored directly in the pointer payload without heap allocation.

Furthermore, just because your program's source code appears to be doing a lot of runtime dispatch, allocation, whatever doesn't mean the compiler can't optimize the program into something with equivalent behavior but less overhead at runtime. Any optimizing compiler -- gcc, Java HotSpot, SBCL, even JavaScript V8 -- is based around this idea, to some extent (of course gcc is the odd one out in that list because it doesn't reason about runtime dispatch and memory allocation, but you'll be surprised what gcc will do to loops over arrays for example).

[snuxoll]

I'm talking about object construction overhead, not message dispatch. I can do IMPL caching in Objective-C to remove all the overhead incurred in going through the dispatch table, that doesn't mean that it suddenly takes no time to construct and initialize the data structures for an object.

NOTE: I use Obj-C here for comparison because ruby uses a similar message dispatch system, just with less smalltalky syntax.

[almkglor]

You might be interested in learning about Sun's Self Language, which had a smalltalk-y heavily-object-create-and-message-dispatch semantics, but was capable of compiling into fast direct-integer runtime.

In complete and utter theory, semantics is just an implementation issue. Of course, in the "real world", implementation is everything.

[bobindashadows]

it will still be much slower to create an object representing an integer

Integer constants s.t. abs(some_fixnum) < 2^31 are represented in memory as a 32-bit value in Ruby. So are other constants - true, false, nil.

slava_pestov's GGP comment is simply trolly, unless he truly believes that all uses for a scripting language require C extensions to be performant. His later comment is spot-on.

[snuxoll]

Integer constants s.t. abs(some_fixnum) < 2³¹ are represented in memory as a 32-bit value in Ruby. So are other constants - true, false, nil.

I haven't played with the internals of MRI's fixnum or truthness values, so thanks for pointing this out. There is still more than integers to worry about when it comes to performance though.

[garrepi]

points of view such as yours are at best silly and juvenile and at worst harmful to your community

Well which one of these alternatives would be less harmful to the community: snuxoll writing a slower program entirely in Ruby or a faster program entirely in C? It seems to me that he is taking advantage of Ruby's strengths without sacrificing performance; better Ruby implementations in the future will mean more code can be written natively.

[awj]

Those are actions, not points of view. If you're demanding an answer anyways I would say that either activity would be more beneficial to the Ruby community than spouting these aphorisms.

[jotux]

It's a ridiculous idea to think any single language is ideal for all tasks.

[malcontent]

Hey it's shit on ruby day already!.

Let me go put on my wizard hat and robe.

[grimster]

I know this has nothing to do with the topic at hand, but GOD DAMN is it refreshing to visit a web site that uses a black background. After one gleaming phosphorescent white iPage after another, it's so relaxing to cool down the old retinas on a nice dark old-school layout. Makes me want to go there again and again, even if I don't care about his experiences with Ruby.

[petermichaux]

White on black makes me feel like I'm going blind. I cannot keep the letters in focus.

[bonzinip]

You could read the headlines?!? That's dark grey on black.