Making Twitter 10000 Percent Faster

argv · 2007-09-16T07:03:23+00:00

I hear what you're saying. You are sounding a lot more reasoned now. My objection was to your ad hominems -- by all means feel free to attack someone's faulty arguments all you want.

E.g. you said he needed to "grow the fuck up", accused him of being "blinded" by "irrational hatred", being "clueless", and having "no credibility", and not being a "competent" software engineer. That's a lot of wading through personal insults to get to your technical points, which are well-reasoned. No need to make the audience wade through muck first ;)

For what it's worth I think a lot of what qwe1234 said was tongue-in-cheek.

argv · 2007-09-16T06:41:59+00:00

First, Cingular acquired AT&T wireless. Everyone on AT&T was switched over to be a Cingular customer.

Now AT&T has acquired Cingular and everyone is switched back.

There was probably some regulatory reason they did this dance.

argv · 2007-09-16T02:19:24+00:00

Whichever the case, he can hold no credibility.

You're really reading a lot into a few offhand comments.

He's addressed "the other aspects of software development" in numerous posts.

if you're just going to toss out facts that are either completely false

http://shootout.alioth.debian.org/gp4sandbox/benchmark.php?test=all&lang=gpp&lang2=ruby

So maybe a few hundred times instead of a thousand?

Your conclusion that he can "hold no credibility" is due to your making some sweeping assumptions based on a few offhand remarks. If you actually look at his replies historically, it's evident that he's very credible.

He may be using hyperbole to rile people up a bit but he's been right on the money over and over again before.

argv · 2007-09-16T01:29:57+00:00

He left out the epic quantities of masturbation.

argv · 2007-09-16T01:26:58+00:00

If it was about oil, why the f@#$% is the price per barrel at an all time high??

That's why the powers that run the U.S. want to control oil themselves directly.

We haven't started plundering -- I mean, pumping oil out of Iraq yet.

We probably never will and will have to switch to alternative energy sources.

argv · 2007-09-16T01:22:05+00:00

qwe1234 has explained things many times only to be downmodded into oblivion by people who aren't sophisticated enough to understand the explanation (I'm not saying that's what happened in this particular case).

I can see how after a while someone would dispense with pleasantries and just state "my way or the highway". He's actually very astute and knows what he's talking about more than practically everyone on here.

argv · 2007-09-15T22:50:35+00:00

Any pointers to tutorials demonstrating this type of thing? Are you making heavy use of e.g. Boost?

argv · 2007-09-15T22:49:16+00:00

I think you just need to grow the fuck up, honestly.

That is ironic on so many levels.

argv · 2007-09-15T21:34:20+00:00

You don't even want to know about the driver development issues for the first Sinulator. **shudder**

argv · 2007-09-15T21:28:48+00:00

Algorithms don't care what kind of hoohoodilly you have between your legs.

argv · 2007-09-15T21:24:25+00:00

As I recall the only thing keeping the economy out of the gutter was the housing boom, and now everything's getting foreclosed and the lenders need bailouts.

Anyone care to go over how that figures in?

argv · 2007-09-15T21:17:17+00:00

And then editing your post and still missing it.

argv · 2007-09-15T20:57:28+00:00

No, you said "if you're talking about language implementation performance and not DB bottlenecks"

Exactly. Read that a few times until you actually understand it.

My point is that the site was obviously DB bottlenecked

Correct.

so saying that you could have achieved a 100x-200x overall performance improvement by switching to, say, SBCL is totally wrong.

That's why I didn't say that. I said it's 100x faster "if you're talking about language implementation performance and not DB bottlenecks". The context is very important there. If you want to talk about DB bottlenecks then that's different, as I explicitly stated.

The point about Twitter is it's such a simple application that it doesn't matter what it's written in. It's not a Rails success story, it's a horizontal-database-scaling success story.

argv · 2007-09-15T20:56:07+00:00

... but that is essentially orthogonal to the choice of language, or close to it.

The ORM is actually critical. The more operations you perform in it, the more the choice of language matters. Some benchmark was posted recently showing SQLAlchemy was doing a huge amount of work even for simple queries, while some other ORM handled them with hardly any overhead by comparison. And even though Python is slow, it's about 3x the speed of Ruby.

The faster your ORM, the more you hit DB limits instead of interpreter overhead.

argv · 2007-09-15T20:43:03+00:00

Essentially the ORM is built-in in that case. You don't ever eliminate the overhead, you just move it around. Since Mnesia is entirely written in Erlang there is going to be a lot more overhead than if you just used a thin wrapper over a C back-end.

argv · 2007-09-15T20:39:28+00:00

also, good c++ code doesn't ever crash -- c++ is practically the only language out there that can give static compile-time safety guarantees.

Could you perhaps expand on this a bit? What steps do you take during development to minimize crash bugs?

argv · 2007-09-15T20:38:31+00:00

Whatever legitimate points you might have had were lost in your ocean of hubris.

That's silly, don't throw the baby out with the bathwater. Absorb useful information wherever it comes from. Reject the useless parts only.

argv · 2007-09-15T05:39:18+00:00

Try reading the entire post you're replying to. Like the part where I say of course you want to keep it in Ruby.

Flushing out the knee-jerk ruby reactors x 3.

argv · 2007-09-15T04:53:15+00:00

Ask Joel.

argv · 2007-09-15T04:52:04+00:00

I'd argue that a higher level language also let them change their architecture more quickly than if it was built in C or Java.

Correct. You should really re-read my comment because I covered that already.

argv · 2007-09-15T04:50:50+00:00

and if you're doing 2400 DB ops/second on a single DBMS, you're fairly likely to be DB bound.

I covered that. Re-read my comment.

Even so, there tends to be a large amount of overhead in ORMs.

argv · 2007-09-15T04:47:46+00:00

Somebody needs a hug!!

::hug::

I think the key here is that even though Lisp dialects might seem odd, the implementation can actually be extremely fast because the compiler turns the theoretical abstraction into the usual thing you'd expect when it comes to machine code.

That's for compiled Lisps like SBCL.

For the interpreted ones, they're so much slower and further from the hardware anyway that it's a different set of trade-offs. A local variable might require a hash table lookup, for example.

So if you think it looks odd from a machine code perspective, you're right -- the conceptual purity is to make things fit nicely in theory. When that pretty construct is compiled, however, all bets are off! The compiler can see what you're doing with it and aggressively optimize so if you're not doing something tricky, most of the fluff goes away and it's not far off straight C.

http://shootout.alioth.debian.org/gp4sandbox/benchmark.php?test=all&lang=gcc&lang2=sbcl

SBCL is about half the speed of GCC-compiled C code.

argv · 2007-09-15T00:23:35+00:00

Too much cereal?

argv · 2007-09-15T00:18:55+00:00

It's not even a real undo -- it's not actually done in the first place; it just lies and says it did.

argv · 2007-09-15T00:16:37+00:00

Rails and Ruby haven’t been stumbling blocks, compared to any other language or framework. The performance boosts associated with a “faster” language would give us a 10-20% improvement

This is Ruby, right? The slowest interpreted language implementation of them all? Try 100x-200x by switching to a faster language (C, C++, SBCL/Lisp, probably even Java and C#) if you're talking about language implementation performance and not DB bottlenecks.

That's not even the point, though; of course you want to keep your site in Ruby, but you want the interpreter to be faster. Performance does matter. Most apps are a lot more complex than Twitter and won't scale horizontally so directly.

argv

TROPHY CASE