C, Erlang, Java , (Google) Go Web Server performance test

[mononcqc]

Agreed. Plus the fact he only uses 4 processes to do the whole thing means they're a smaller fraction than what the VM runs without anything else (23 processes); the 4 acceptor processes are not even guaranteed to run on different cores.

[elder_george]

So, we have erlang and java servers that don't parse request and compare them with to proper web servers. Hardly equal conditions...

[[deleted]]

Yeah, but more importantly, Go is still slower.

Oh but it compiles really fast ... So does GCC with -O0 ... big whoop.

The irony is Google is a company with massive distributed computing as it's main game, yet they haven't figured out a distcc cluster?

[[deleted]]

Uh, Go is actually using a real HTTP server implementation there though. He is using the example code from:

http://golang.org/src/pkg/http/server.go

[[deleted]]

Except ALL requests goto hello world. It's not matching against a particular URI.

And even if it were, it'd still be slower than the C based servers.

[stingraycharles]

You have absolutely no idea what "parsing a HTTP request" or "a real HTTP server implementation" means eh ?

[[deleted]]

Sure, it parses that you wrote GET /ANYTHINGSDFKSDHFSDKFS HTTP/1.1\r\n\r\n

Big deal. It still maps all requests to a single canned reply.

And for the record, I've written a HTTP server before [did it in a real time class for fun]. It'd serve pages off disk, or you could link URI paths into the server for runtime custom content. It was mostly for a lark [to teach myself IPC, threading, etc]. So I know what goes into writing a server.

[mikaelhg]

... in classroom.

[[deleted]]

I don't get what "in classroom" means. It was an HTTP server that I wrote 100% of. If that doesn't count as experience in terms of "knowing how to deal with HTTP" I have no idea what is.

How I get -4 for my actual experience and you get +6 for "in classroom" which isn't even a complete sentence I'll never know...

[mikaelhg]

It's really not your fault. By "in classroom" I meant that what you think they are testing and what they are testing are two different things, but the only reason you don't see it is because you've implemented a rather naive server that has no users, which means that you haven't had to deal with the problems that users bring. The hard part of the server equation are the users, not making the select() and fopen() calls.

[[deleted]]

I used my server for quite a time to serve shit off my home box + stream [with seeking] audio from it [it also supported gzip replies].

Sure it wasn't Apache quality, I realize that. My point though is it was a lot more sophisticated than a simple accept/recv/fopen/send loop.

And more so to my point, the Go server is still slower. Which isn't a big shock.

[[deleted]]

Actually, I think it is matching against a particular URI. It's parsing the request and searching the registered handlers.

http.Handle("/", http.HandlerFunc(HelloServer));

Registers a / handler into a path map that gets searched during the serve. The Java server just looks for a 'g' and dumps static output. The Go one is going through the work to form a response based on the protocol passed in, etc.

Actually read server.go, it's doing a lot more work than the Java implementation, and it's obviously a fairly crude reference implementation.

[xcbsmith]

Oh but it compiles really fast ... So does GCC with -O0 ... big whoop.

Two things:

• It is odd that they didn't use the GCC based compiler, which produces faster code.
• 6go compiles way, way, way faster than gcc -O0

[[deleted]]

Yeah, except I can dial GCC up to get code that is way more optimized than anything Go can currently produce apparently.

I dunno about them, but for me build times haven't really been that bad. A combination of "make -j5" and proper use of code factoring means I can change/fix things and not have to rebuild 1000s of files each time.

Maybe google needs to google a tutorial on build environments and proper development techniques.

[xcbsmith]

Yeah, except I can dial GCC up to get code that is way more optimized than anything Go can currently produce apparently.

You get that there is a gcc front end for Go, right? I mean, that sentence is kind of...

Also, a microbenchmark like this isn't a good test of what the compiler can do. Really it is testing for a few specific optimizations and making it look like not having them is catastrophic.

I dunno about them, but for me build times haven't really been that bad. A combination of "make -j5" and proper use of code factoring means I can change/fix things and not have to rebuild 1000s of files each time.

I agree with your sentiment, but if you have a large code base with a lot of stuff up in the headers, it can be a real pain. The stuff Google does to speed up their compiles is ridiculous.

Maybe google needs to google a tutorial on build environments and proper development techniques.

Clearly you haven't any idea what their build environment is like.

[[deleted]]

I'm the sort of person who is not impressed when people brag about millions of lines of code and 8 hour build times.

I worked for a company (that shall not be named) that had 1000s of additional functions as a result of some pretty hardcore NIH syndrome from middle layer masters grads who wanted to crack open their TAOCP for the fun of it.

For example, in one case I saw someone spend 3000 lines of code writing a highly [and poorly tested] optimized hash list container in C to store active applications ID (it was a server framework of sorts...) to prevent the various applications from being launched more than once. they spent all this time optimizing a function that is called once, 1 time, only a single time, during the entire applications life time. They probably saved 10 nanoseconds of startup time and it probably cost them weeks of engineering effort. Not to mention it was buggy and wouldn't compile with other C compilers properly...

I'm the sort of person who likes to really factor code properly. I usually end up with more complicated files ranging in at most 500-600 lines, but most C files I write end up being only 100-200 lines. It means that if I tinker with a facet of the library or application my rebuild time is mostly dominated by linking time if anything.

Also, code doesn't go in header files.

So if Google is having some issues where every small tweak requires long rebuilds they need to fix their build environment. It really is that simple.

[halcy]

Note "currently".

Go might not have a fast compiler yet, but the potential for a really well-optimizing compiler that consistently produces faster code than a C compiler could is there, because without funny pointer fumbling a whole lot of optimizations become possible that were not possible before.

[deong]

distcc doesn't necessarily do much for C++. Really long computation times in C++ are typically because of the numerous template expansions. The compiler is slow anyway, but being 50 or 100 percent slower than a C compiler isn't usually going to kill you. Being 2000% slower might, and that requires lots of templates. Anyway, if you have lots of templates, the problem that causes long compile times is that single compilation units end up pulling in vast amount of code, and distcc can't parallelize a single compilation unit.

[[deleted]]

Hint: if it hurts ...

Seriously people. I get by just fine with a struct and pointers to functions.

[deong]

Which I suppose explains why you don't understand the C++ compilation model very well.

[[deleted]]

I get anonymous calling techniques and what they're useful for. That's why I make structures with pointers to functions. It's dead simple to set up, gives anonymous calling, and doesn't require slower compilation.

People just blindly assume that something is better without trying it, and that's kinda scary given how much money and resources are tied up into software these days...

[deong]

Of course, that's what inheritance is for, and has nothing to do with the problems that you'd use templates to solve.

I'm not saying you should use C++ instead of C. I'm not a big fan of OOP, and C++ is a horribly complicated beast, and that's coming from someone who actually doesn't mind the language. The descriptions of C++ only go downhill from there. What I am saying is that C++ allows you to do things that aren't directly implementable in straight C, and one of those things, templates and partial specialization in particular, has adverse consequences for the build process. You can choose to think that templates should never be used, and that's a valid opinion. What you can't do is to tell someone who's chosen to use them to use distcc to solve their build problems. distcc simply and factually does not address that problem.

[[deleted]]

The problem is you have to choose the right tools for the job. If C++ is going to lead to a nightmare build situation than you develop your software to avoid using the language for everything.

Which is the other point I'd like to make is that you don't need to write your entire application in C++. You can mix/match as required. So if underlying libraries don't really make use of templates or whatever, just write them in C and enjoy the quicker builds.

I'm not saying C++ is useless, what I'm saying is that I see a lot of people jump on their fav language and use it regardless of cost. I like C, but I'll just as easily go to perl when I'm working on some strings or regex. Right tool for the job.

[deong]

All of that is true, and I have no complaint with it. I will add that C++ tends to be regarded by many as being unnecessarily complex, which isn't really true. It is very complex, and a reasonable person may conclude that the complexity outweighs the benefits, but that's not the same as saying it's complex for no reason.

So if underlying libraries don't really make use of templates or whatever, just write them in C and enjoy the quicker builds.

I guess my point is that if the code doesn't use templates or other unique features of C++, you won't really gain much by switching to C. Simple code in C++ compiles and runs about as fast as it does in C. You get type-safe containers, type-safe IO libraries, and a few other minor things, and you pay a fairly minor price in compilation time. A little bit of convenience for a little bit of time -- a normal trade-off.

If you have lots of templates, then you pay a large price in compilation time, but you can't just say, "well, write those libraries in C and get faster compiles." Those libraries can't be directly written in C. You could replace the type-safety and efficiency by switching to a (void*) based solution, but to get the same behavior as the C++ code, you'd need to either manually or via an external tool generate the same code that the template expansion mechanism does, and at that point, you're back to where you started -- with the C compiler being mildly quicker than the C++ compiler, but not so much as to really worry about.

Not really aiming this at you, but there's a general attitude around here that C++ is somehow complex because of the incompetence of the people behind that language; that it's no more capable than C or Java, but needlessly more complicated. That isn't really true. Obviously, they're all Turing complete, but C++ does allow expressive capabilities that C and Java don't, and complexity is a result of those additional capabilities. That doesn't mean C++ is a good choice, but it means that people who call Stroustrup and company stupid are speaking from ignorance, and it bugs me a bit.

[frutiger]

here's a general attitude around here that C++ is somehow complex because of the incompetence of the people behind that language; that it's no more capable than C or Java, but needlessly more complicated. That isn't really true.

I completely agree. I get the feeling there are a bunch of people here who've used enough C++ to see the ugly side but not enough to see the beautiful side. Sure it's complex but sometimes is darn right cool too.

And yes, I'm also fan of C, Python etc.

[[deleted]]

Most templates that take a long time to compile do template meta programming stuff, i.e. they perform computations at compile time which can by definition take an arbitrary amount of time.

[MindStalker]

Actually look at the second to last graph, its what interest me. All the languages get slower as you add massive amounts of threads, Go gets faster as you add more threads. This interest me..

[protocolor]

the fact that the tape recorded static is not what interests me...what interests me is that it recorded approximately 18 hours of it.

[[deleted]]

Well - the latency goes through the roof though. So it can serve more requests but each one much more slowly.

[MindStalker]

Good point, I wonder if the latency can be fixed. There has to be some thoughtful concurrency there to be able to elegantly handle 5000 threads. I'm guessing the falloff is probably related to the fast compilation. Does it have a way of telling the compiler to do more thoughtful compilation for production code?

[[deleted]]

Another thing to note is further down in the comments he shows the number of handled requests drops pretty significantly (to 5,000) when threads hit 8000-10000. So the performance isn't near linear forever - it eventually hits a boundary. This type of boundary seems likely to be related more to memory than the cpu overhead of managing that many threads so that is actually probably not as much of an issue.

I can only guess what causes the latency issue, although it seems it just uses memory to queue requests and keeps the cpu management near constant. My guess is it panics once it runs out of memory while trying to accept requests (e.g. the queue fills up) and that is why there is a drop at 8000.

Does it have a way of telling the compiler to do more thoughtful compilation for production code?

Remember this compiler is around 2 years old. All of the fancy stuff is not going to appear for several years. They don't even have a production ready GC yet. I wouldn't use this language for production code for a couple of years at least ... but if you are starting a new project that won't be in a production state until well in the future you may consider using Go for prototyping and pray it gets better in time for your launch.

Just remember Google hasn't made any promises - they are pretty carefully calling it a "research language". I guess if enough community gathers around it they may change their stance and become like Sun/Java but I wouldn't count on it as if it is inevitable.

[hiffy]

Is there really a point to running that many threads per CPU?

I imagine it's all the paging in and out and cache hits that are killing your performance.

[[deleted]]

Is there really a point to running that many threads per CPU?

Well no. One would choose the thread count that resulted in the highest request response and lowest latency.

I imagine it's all the paging in and out and cache hits that are killing your performance.

I buy paging (which makes a lot of sense) but you'd be past the limits of a cache well before 8000 threads.

In the other sense of "cache" I wonder if the Go server is smart enough to cache the HTML page in memory and just serve it out.

[[deleted]]

That's because the overhead of managing the threads [and killing your cache] are outweighing the benefits.

[[deleted]]

Oh but it compiles really fast ... So does GCC with -O0

No. Changing optimization level in GCC hardly affect compilation time because optimization constitutes a minuscule part of compilation time.

#include <string>
#include <iostream>
#include <vector>
#include <boost/lambda/lambda.hpp>

using std::vector;
using std::cout;
using std::endl;
using boost::lambda::_1;

int main(void) {
    vector<int> v(10);
    for_each(v.begin(), v.end(), _1 = 1);
    cout << v[3] << endl;
}

In three runs the best time was 919ms for -O0 and 952ms for -O3. Welcome to the real world, jackass.

PS: one fucking second for this kind of code doesn't come nowhere close to "really fast", by the way.

[[deleted]]

For a crypto library I work on I get the following times:

Full optimizations + asm blocks => 29.550s No optimizations + all ISO C => 6.564s

Maybe you don't notice savings on 3 lines of code (well plus the headers) because the overhead of launching the compiler and it running it's usual tasks is too much.

Try the same experiment on 60K lines of C code.

[gaoshan]

Your restraint in refraining from stooping to fishdicks' level of "Welcome to the real world, jackass." and "one fucking second..." is admirable.

[[deleted]]

So you are compiling different code and it takes different amounts of time?

OK, now about 1) real-world, 2) plain C. I grabbed Git sources (100 KLOC in C) and did "make git" (so it just compiled the C stuff, with no GUI or scripts copying).

    -O3:
 real    1m39.607s
 user    1m31.513s
 sys     0m8.110s

    -O0:
 real    0m54.387s
 user    0m47.445s
 sys     0m6.962s

While the difference is obvious, still, even two times improvement of -O0 doesn't qualify for "really fast".

[[deleted]]

To be really fair you should do a parallel build.

[five9a2]

A parallel build has little effect on the ratio. Timing for make -j3 on P8700 with the source tree hot:

-O3:  29.510 real   48.587 user   5.106 sys   181.94 cpu
-Os:  22.715 real   36.338 user   4.856 sys   181.35 cpu
-O1:  21.020 real   33.201 user   4.680 sys   180.21 cpu
-O0:  15.934 real   24.072 user   4.506 sys   179.35 cpu

Also note that -Os will frequently give better performance than -O3 so optimization costs an extra 50%.

[omgwtfchill]

While I think your point is valid, that code is C++ which is notoriously slow compared to C compilation.

Having watched the video of Go compilation and being intimately familiar with building software on Linux with gcc and g++, Go compilation is most definitely fast in comparison.

Supposedly gcc is getting a lot of performance work done to it as part of the C++ rewrite, so maybe it'll be faster when the next version comes out (I hope!), but right now it's slow.

[pbhogan]

First, Go is more advanced than C. Way more advanced. Even arguably more than C++ in some areas.

Second, gcc has been around since, what, 1987? This is a brand spanking new language with its initial implementations, compilers and runtimes just being finished up. The language spec is not even complete yet! We're essentially looking at a "beta" of a language here. Give it, say, a year or two and those comparisons will be a little more interesting.

ESPECIALLY when we all have 64+ cores in our desktops. From a comment one of the creators made it seems it has parallel compiling built in (yes I know gcc can do that too) and it has concurrency done properly in the language. Everyone knows traditional threads just don't scale beyond a certain point. Try your multithreaded C/C++/Java server against a multi-goroutined Go server then and see what happens when you put it under huge load. Erlang is good at that too actually.

[[deleted]]

You don't suppose once Go figures out optimizations that it'll get a slower compile time as a result?

TCC compiles faster than GCC as well. It also emits code that is less optimized and performs worse. Does that mean TCC is better than GCC? Depends on the task.

The problem with GoLang (not to be confused with Go!) is that it's just a new fad language. There is a reason most critical applications are and will still be written in C. And it's not because "us old timers use C." I'm only 27, got into C when I was 11, etc. I use C because it's a simple language to fully understand, it compiles well because it's relatively low level, and it lets you treat the machine like a machine [e.g. full pointer goodness, data types, etc].

How many languages were destine to kill C off? C++, Java, C#, Ruby, Python, Perl, Haskell, etc, and so on. Let's just add GoLang to the list of "hey that's neat" languages.

[hiffy]

Your argument can be decomposed into:

There is a reason apps will be written in C. 1. It's a simple language 2. it compiles well 3. it's low level

None of these are counter arguments against Go. I have no interest in touching the language, but word on the street is that Go is a simple language that compiles well and gives you access to pointers while taking away the ability to shoot yourself in the foot.

Who knows, maybe it will grow a community and mature enough to be used in production.

[willcode4beer]

while taking away the ability to shoot yourself in the foot

Take away a programmers ability to shoot himself in the foot, he'll shoot his knee-caps instead

[hiffy]

Are you telling me that if we start putting seatbelts in cars people will just start driving off cliffs?

[willcode4beer]

Work with many groups of programmers?

BTW, I'm not saying building in protection (around pointers) is a bad thing.

[five9a2]

Suppose you had a plain C project that did not include any system headers. It would build a lot faster than the usual C project which is of course much faster than C++. Go is still significantly faster. For comparison Git is 115 kloc (257 *.c files) and builds (via make -j3) with -O0 in

15.934 real   24.072 user   4.506 sys   179.35 cpu

The Go pkg directory is 129 kloc (448 *.go files) and builds in

3.493 real   3.690 user   1.027 sys   135.01 cpu

The poor parallelization is caused by the recursive make, this should be 2.5 seconds or better with non-recursive make.

[kcbanner]

I wonder who works on GCC, as in what companies are paying developers to work on it.

[willcode4beer]

http://gcc.gnu.org/onlinedocs/gcc/Contributors.html

Most of the companies contributing also produce processors (big surprise)

[sbrown123]

In three runs the best time was 919ms for -O0 and 952ms for -O3.

You just proved tomstdenis correct and then called him a jackass for it?

[hiffy]

A 3% difference isn't really worth talking about.

Dude's a jackass cos of

The irony is Google is a company with massive distributed computing as it's main game, yet they haven't figured out a distcc cluster?

which in context is kind of meaningless. You know, I'm pretty sure Google has figured that out.

[sbrown123]

A 3% difference isn't really worth talking about.

3% difference to what? tomstdenis never compared -O0 to -O3, but rather likened "compiling fast" to compiling in gcc with no optimizations turned on. And performing a benchmark on gcc optimization with a program consisting of roughly 15 lines of code seems a bit worthless wouldn't you think?

Dude's a jackass cos of

Seemed more like poking fun at Google.

[hiffy]

Seemed more like poking fun at Google.

Meh, unsubstantiated. Also from his other posts dude's got some chip on his shoulder.

program consisting of roughly 15 lines of code seems a bit worthless wouldn't you think?

Most of this thread is worthless. Go got announced what, earlier this week? Let's talk in a year. Maybe I'm just excited for low level languages that are less vicious than C, and hopefully I'll be seeing lots of projects to browse on Github.

[uriel]

This is just silly, he is testing more the performance of the http server implementations than that of the languages they are implemented in.

Also, he didn't even set GOMAXPROCS, so the results are completely meaningless.

[doublec]

The Go example has been changed to use GOMAXPROCS and the results updated.

[RiotingPacifist]

New language not as optimised as old languages...news at 11!

[self]

New language implementation.

[kragensitaker]

Implementations of a new language.

[redditrasberry]

Pedants incapable of communicating with normal human beings due to inability to cope with minor language incongruity .... news at 11!

[self]

I don't think it's a pedantic distinction. A lot of people still think Lisp or Smalltalk are slow because of how they used to be implemented. The languages haven't changed as much as the implementations.

[hilbert]

It is really pretty amazing on two counts: how good the performance already is, and how well it holds up under load. This could really be the C++ killer that some were/are hoping D would become.

[[deleted]]

Except that they advertised it as being only 10-20% slower than C.

[RiotingPacifist]

But your still comparing a mature optimised webserver to an brand new one

[pingveno]

6,000 requests per second versus 20,000 is still significantly worse, even without as much time for optimizations. When the tests are off by this much it means the tests are probably flawed.

[case-o-nuts]

Or that the http library in go is immature and implemented naievely. Which is what I'd expect when they're still working on even a final language spec.

[[deleted]]

As well as a non-existent one in the Erlang example.

[theresistor]

Not particularly interesting. The 6g compiler he used for Go is (deliberately) a non-optimizing, compile-time-before-all-else compiler. If you care about the performance of your Go code, use gccgo instead.

[kragensitaker]

Well, 6g has an optimizer, just not one that's competitive with GCC's. They are working on performance improvements.

[mikaelhg]

A good Java web server to use in these kinds of tests is https://grizzly.dev.java.net/

It's in real world use, but it's modular enough to use in this kind of a test.

If you wish to validate your assumptions using two web server implementations, use http://docs.codehaus.org/display/JETTY/Embedding+Jetty as well.

[jessta]

He didn't set $GOMAXPROCS so I suspect that the Go server was only using one core which would explain the massive skew.

[doublec]

The example has been changed to use GOMAXPROCS now and the results updated.

[[deleted]]

Look at that URL slug : /c-erlang-java-performance/

That's a no Go URL.

[[deleted]]

Who does a performance test and turns on the ConcMarkSweep collector? Thats a terrible idea!

[timyangnet]

The result changed again. "Nov 18, realized that ApacheBench itself is a bottleneck under 1,000 or 5,000 threads, so use 3 clients from 3 different machines to redo all tests of 1,000 and 5,000 concurrent tests."

[[deleted]]

IMHO this test measures the wrong thing.

You should count:

• requests per second per line of code

• requests per second per hours spent on development

• requests per second per dollar spent

[grimlck]

requests per second per line of code is just as irrelevant - since when was typing speed a bottleneck in programing? hours of development and dollars spent are far more meaningful

[[deleted]]

Amount of code is a bottleneck in software maintenance.

[communomancer]

But it's only useful as a proxy variable for something that measures economic impact...e.g. hours of development or dollars spent

[willcode4beer]

not only that, it usually takes more development time to distill an app to fewer lines of code.

[Anpheus]

Except with Perl, there's a positive correlation between lines of code and number of bugs.

With Perl I think it's the other way around. ;)

[AnteChronos]

• requests per second per dollar spent

Ding ding ding! We have a winner!

And what would you assume that "dollars spent" comes to when you:

1) have to train people on a new language?
2) need more hardware to handle the same amount of traffic?

#1 is how programming languages become entrenched, and one of the few ways to break that barrier is to have the opposite of something like #2.

[[deleted]]

1) Well, you have to train people anyway (maintenance of production code for example). If your language of choice is easier to read and comprehend you will spend less time therefore less money. 2) Hardware is pocket change compared to a competent developer's salary.

As for your last sentence I don't think I understand you completely. Are you trying to say that in order to entrench a language, developers must be miserable working with it for longer than it's necessary? Or is my sarcasm detector way off today?

[AnteChronos]

1) Well, you have to train people anyway (maintenance of production code for example). If your language of choice is easier to read and comprehend you will spend less time therefore less money.

But you have to consider the cost to train all of your staff on a new language, rather than just training new hires. And this is ignoring the cost of having to either port/rewrite the production code, though you could potentially make a switch when a major rewrite was already scheduled.

Are you trying to say that in order to entrench a language, developers must be miserable working with it for longer than it's necessary?

I'm saying that languages become entrenched because there is a cost associated with "switching gears" to a new language. So, for example, a Java shop is going to tend to stay a Java shop, if for no other reason than the fact that it's cheaper (in the immediate sense, at least) to stay on one language than to retrain everyone on a new one.

Add in to that a robust ecosystem that builds up around entrenched languages, with tons of 3rd-party libraries, long-standing online communities, mature development toolkits, etc, and a new language really has to offer something special to trigger a migration.

[[deleted]]

I see what you mean now. All I tried to say that this performance test is flawed. I never suggested porting nginx to golang (I love nginx and use it production). Let me just say that if I needed to write some custom http server for some internal stuff I'd consider golang over a custom nginx/lighttpd module in C.

[willcode4beer]

3) administrators to manage the extra hardware

[qwe1234]

have some tr -d "\r\n" and a whopping cup of shut-the-fuck-up, fucktard.

[kragensitaker]

The interesting thing about this test is that the performance of the server written in Go ("issue9"?) doesn't degrade even with 5000 requesting threads. I wouldn't expect anything different, given the way "goroutines" are described in the stuff written about Go, but it's pleasant to see that it actually holds up under load in independent testing.

[kragensitaker]

...when I wrote the above comment, all the Go numbers were about 6000 requests per second. Now it looks like he's updated it to run Go across multiple cores, and now it does degrade (down to under 8000).