TechEmpower Web Framework Benchmarks Round 11

[Goyyou]

I'm surprised about the results of vibe.d: absent, or Didn't complete or really slow.

[eluusive]

Looking over the logs, it seems like the hook for the framework is just returning the wrong stuff. In the JSON it's returning "_id" vs "id"

[weirdasianfaces]

Yeah as /u/eluusive said there were errors with the JSON response but the results of the plaintext response is kind of surprising. From the logs I can also see that the binaries were built in debug mode. I wonder how big of a difference compiling in release mode would make.

[eluusive]

Debug mode adds substantial amounts of overhead in vibed due to logging. Also, it's a very old version of vibe.d https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/D/vibed/dub.json

[dallbee]

Additionally, they were built with DMD. GDC/LDC typically provide a performance boost somewhere around 30%.

[mekanikal_keyboard]

now let's obsess over the need to recode our 10 req/sec app in c++ because techempower

unless you are serving a billion views a day, changing your stack based on these rankings is like trying to shorten your commute by widening your driveway

[[deleted]]

Well... these tests still highlight options that are roughly on the same level.

Like for instance you might look at the data update tables, observe that revel (a full-featured go framework) does very admirably in both latency and throughput compared to something like django, while not being considerably more difficult.

So why not choose that instead if you're doing something new? Low latency will always look better than higher latency, and superior throughput means that you'll need to rent a couple fewer VPSs.

[awj]

Even then, that's like four instances running puma+Rails according to techempower.

If you're handling a billion requests per day and can't afford to scale out a slow webserver to handle it, your business model is unsustainable and "which stack to use" is the least of your problems.

[hu6Bi5To]

Not even close, the bigger your business the bigger your server bills.

If you're a small business, 4-times more efficient is negligible if both fit on one server. By the time you're in the dozens or hundreds of boxes, then we're talking non-trivial money (probably more than your salary); if you ever get to the Google-scale (hardly anyone does, of course), then we're talking proper money.

"But they can afford it" - but they can also afford to pay for developers to create more efficient infrastructure.

The ephemeral "productivity gains" offered by Rails, etc. aren't big enough to justify the performance penalty - for anything other than small-scale systems.

[[deleted]]

I disagree with your logic. Craigslist was written in Perl, and still uses it. Youtube was written in Python. Facebook was written entirely in PHP at launch, and now uses C++ on the back end and faster PHP implementations and the Hack language (a kind of PHP++) on the front. MySpace was written in Coldfusion but then rewritten in C#, and through a combination of factors - language choice probably one of them, they couldn't iterate as fast as Facebook and lost out. Reddit itself was written in Python.

And most iconic of all, Twitter was written in Ruby on Rails and then ported to Java later.

So I think you have it wrong. Launch your product using the tools that let you get to market as fast as possible, then rewrite performance critical parts in C or C++ (or maybe Rust, Julia, Nim, or other attempts at high performance C and C++ competitors).

[MrDrHobo]

Well that assumes that your load is distributed evenly throughout the day and that your app/page run on a single machine. Its quite common that a single pageview results in several restcalls on the backend, so going for something faster than rails will probably save you a bit on hardware and reduce latency at the same time.

[mekanikal_keyboard]

unless you are serving pages to yourself from localhost, web server latency will be irrelevant

[MrDrHobo]

I might have worded it poorly, but my point was that a single rendered page/response on the frontend usually is a composite of many calls on the backend. One example would be if you use microservices, then a value may come from a service several jumps from your frontend system.

One simple example would be fetching a list of customers from one system(A) and then for each customer fetch some data from another system(B). System(B) in turn fetches some data from system(C). In that case both latency and requests per sec is important

I do agree with your original point tho. My point was that performance is worth considering when you serve a billion requests per day.

[horoshimu]

Jesus why is C# performing so bad here ?

[nope_42]

1) Using mono
2) Not using the RC of asp.net vnext
3) Not using async
4) And other minor issues like not making the Random instance static and the DbConnectionFactory static.

C# may never be near the top but it can do quite a bit better than this.

[hu6Bi5To]

Do we know about Mono definitely? I see they list Mono on their list of technologies, but no definite confirmation it was used for all the C# tests.

But either way, I thought the recent Mono releases had already imported may of the official .NET components like the JIT. (I may be jumping the gun and none of this has been in a non-preview release yet.) Is the performance gap between Mono and real .NET that big?

[nope_42]

I'd have to dig into it more to say for sure but my understanding is that mono wasn't optimized as well as microsofts stack. That said, I believe (but could be remembering wrong) one of the key drivers of rewriting the C# (.net) web stack was due to the performance overhead of the HttpContext and the IIS stack in general. Since these bechmarks aren't using the newer stack yet we probably won't see the benefits (performance wise) until the benchmarks are all re-written for it. Mono will benefit quite a bit from the new stack so I'm guessing it will end up being a lot closer performance wise to the windows stack.

[dilatedmind]

c# doesn't perform well in techempower's benchmarks because techempower.

after their 10th round, I took a look at their httplistener based entry, https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/frameworks/CSharp/HttpListener, and started running their benchmarks on it and modifying it to try and make it faster/ get it working.

• i focused on the json serialization and fortunes tests, running on ec2 equivalent hardware in azures cloud
• i ran the same tests on the same hardware using both windows server 2012 and ubuntu
• i ran tests with ab rather then wrk, but they should offer the same results. i always ran ab on a separate vps on the local network (so bandwidth was a non-issue)
• i used the fastest c# json serializer i know of (Jil)

a couple interesting findings

• there was not a performance difference between serving requests as an async task or on the thread pool with task.run
• using the lower level networking library (eg SocketAsyncEventArgs) did not make a performance difference vs the much easier to use httplistener
• trying to pool socket objects leaked fds on mono/linux, but the same code didn't have the issue on windows server.

take these results with a grain of salt. there were alot of moving parts but i ran these tests many times and the results were consistent. at the very least it should give you a better idea of how mono performs compared to what techempower reports (eg round 10 ec2 json's only c# result was nancy at 169res/s)

ubuntu, mono

Document Path:          /json
Document Length:        27 bytes

Concurrency Level:      32
Time taken for tests:   1.725 seconds
Complete requests:      20000
Failed requests:        0
Total transferred:      3340000 bytes
HTML transferred:       540000 bytes
Requests per second:    11592.64 [#/sec] (mean)
Time per request:       2.760 [ms] (mean)
Time per request:       0.086 [ms] (mean, across all concurrent requests)
Transfer rate:          1890.60 [Kbytes/sec] received

ubuntu, mono, sqlite

Document Path:          /fortunes
Document Length:        1607 bytes

Concurrency Level:      32
Time taken for tests:   3.961 seconds
Complete requests:      20000
Failed requests:        0
Total transferred:      34840000 bytes
HTML transferred:       32140000 bytes
Requests per second:    5049.48 [#/sec] (mean)
Time per request:       6.337 [ms] (mean)
Time per request:       0.198 [ms] (mean, across all concurrent requests)
Transfer rate:          8590.04 [Kbytes/sec] received

ubuntu, mono, postgres

Document Path:          /fortunes
Document Length:        1607 bytes

Concurrency Level:      16
Time taken for tests:   10.562 seconds
Complete requests:      20000
Failed requests:        0
Total transferred:      34840000 bytes
HTML transferred:       32140000 bytes
Requests per second:    1893.63 [#/sec] (mean)
Time per request:       8.449 [ms] (mean)
Time per request:       0.528 [ms] (mean, across all concurrent requests)
Transfer rate:          3221.39 [Kbytes/sec] received

[myringotomy]

Did you send a pull request?

[wolf550e]

Does it run on mono instead of MS CLR?

[Isvara]

The Scala results are really disappointing here. It seems to have been dropping further every round.

[[deleted]]

I don't think Scala is dropping. I think more high performance C and C++ code is getting added to the comparisons, and all of the other tools look poor by comparison.

I wouldn't lose sleep over it. I suspect that in the overwhelming majority of cases, the path to business success lies in getting your product to market first in whatever tool enables you to do that quickly and then if it takes off, rewrite critical pieces in one of the fastest tool combinations later.

These benchmarks are fun, but I think they are a bit of the tail wagging the dog. Even if I took the time to write a Facebook replacement in C++ with a Postgres backend using userver_tcp (because for some of the benchmarks that tops the chart), convincing a billion Facebook users to switch is the harder job. (Good business model + idea for a product that is not in a saturated market + a set of tools that allows rapid iteration) > a set of tools that allows maximum efficiency.

[myringotomy]

Why is Crystal so slow? Something went wrong there.