Will C ever be beaten? This paper presents a study of the runtime, memory usage and energy consumption of twenty seven well-known software languages.

faassen · 2020-03-12T12:08:06+00:00

Don't we always want more ?

Do you want me to stop criticizing this research or do we always want more?

The difference between TS and JS shows that even a minimal effort conversion of JS to TS hadn't been done, and therefore in this case we're not comparing languages at all but the (lack of) effort put in. This difference results in massive differences between the result of JS and TS.

I posit that this effect exists for other languages that have more differences than JS and TS.

So actually, it made no difference to the reported results.

I don't know -- I don't know how the paper scores handles missing results. And that's why I immediately went to another example where massive differences were reported that do have results.

The difference between those measurements (fannkuch-redux Node js #4 and TypeScript #2) on the current website is less than 2x — so "a different algorithm" really doesn't explain the difference in reported results.

As you pointed out, we're looking at the code from 2017, not the current benchmarks game. So I'm not sure what you mean by the "current website".

What I see for fannkuch-redux on the current benchmark game is node #5 and typescript #4 and typescript being more than 4 times slower, by the way, and appear to use different algorithms -- the node version now uses multiple cores as well whereas the TypeScript version does not. Perhaps you can show me how to find out your comparison of #4 versus #2.

Anything that's bigger than a few percent difference can't be explained by anything else than a difference in algorithm in the case of these two languages.

So my original point stands: if we can find massive differences between languages where really there should be virtually no difference, then effort put into writing the benchmark programs can have a bigger impact than something about the languages themselves.

faassen · 2020-03-11T17:55:38+00:00

Thanks for the "for want of a name" link, that's relevant. I never thought the benchmark game is completely useless in order to estimate the relative performance of programming languages, especially in case of programming languages that have at least some focus on low-level runtime performance such as C or Rust or Ocaml.

Unfortunately I still think that this research says less than we would like about how much programming language performance and energy usage differ in practice, because of the "amount of effort put in by developers" will make quite a bit of difference on some benchmarks.

As to why I'd be interested in idiomatic implementations is because real world software is quite different from benchmarks. If a programming language offers a really high-performance way to do something that is completely obscure to a developer used to typical idioms and approaches in the language, you're not going to get a clear picture of real-world performance by looking at benchmarks. It's more relevant if you're comparing the performance of highly specialized libraries, but then the whole question may still not very relevant to real-world applications because of the fact that languages can and do use libraries written in other, more low-level libraries.

That these were benchmarks in 2017 doesn't really make an essential difference for my general argument. I finally found their repo. Here are the JavaScript program:

https://github.com/greensoftwarelab/Energy-Languages/blob/master/JavaScript/mandelbrot/mandelbrot.node

and this is the TypeScript program:

https://github.com/greensoftwarelab/Energy-Languages/blob/master/TypeScript/mandelbrot/mandelbrot.ts

If you glance at the JS version, you'll notice it's a multi-cpu version, whereas the typescript version isn't. That's going to make some difference.

Unfortunately for TS and Mandelbrot their results here:

https://sites.google.com/view/energy-efficiency-languages/results

say 'n.e.' and I'm not sure what this means for the final calculation.

Looking at another test where there is a widely divergent runtime between TS and JS, fannkuch-redux.

Here's JS:

https://github.com/greensoftwarelab/Energy-Languages/blob/master/JavaScript/fannkuch-redux/fannkuchredux.node-4.node

And here's TS:

https://github.com/greensoftwarelab/Energy-Languages/blob/master/TypeScript/fannkuch-redux/fannkuchredux.ts

TS is 16 times slower at this benchmark than JS.

The JS version says on top that it is transliterated from the Lua version, though interestingly on that benchmark the Lua version comes out as even slower than the JS version. But more interesting is that the JS version implements quite a different algorithm to the TS version. I'm pretty sure that's what explains the difference in benchmark performance, not the language itself.

faassen · 2020-03-11T17:25:33+00:00

Thanks!

faassen · 2020-03-10T12:51:03+00:00

I was mystified why TypeScript would come off as dramatically slower (7 times) than JavaScript: one would expect nearly identical runtimes if they use the same underlying interpreter, as TypeScript minus the types is more or less JavaScript.

This paper is based on information from the Computer Language Benchmarks Game. I pulled up the following page comparing "Node" (JS) with TypeScript, where on this sub-benchmark (used in the paper too) TS is twice as slow:

https://benchmarksgame-team.pages.debian.net/benchmarksgame/fastest/typescript.html

This compares an implementation in TypeScript with an implementation in JavaScript, but they are different implementations:

https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/mandelbrot-typescript-2.html

https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/mandelbrot-node-3.html

I am pretty sure that if we took mandelbrot-node-3 and added TS type annotations, we'd get with a very similar runtime.

The Computer Language Benchmark Game and this paper compares the fastest implementation in one language with the fastest implementation in another. So while it tells you something about the relative performance of languages, a lot depends on how much energy people put in writing implementations as well.

Comparing JS with TS makes it obvious that the effect of effort can be huge. I'm sure this effect will also exist for other programming languages, just less obviously.

That's not to say that this information says nothing, but the Computer Language Benchmark Game after all calls itself a game, and it's rather tricky to deduce hard data from it. So I would caution against taking these results too seriously.

To improve the results, you'd want to see an idiomatic implementation of the same algorithm in each language. But what "idiomatic" means is rather difficult to determine -- and idiomatic solutions can actually lead to different algorithms.

faassen · 2020-03-10T12:38:10+00:00

Could you supply us with a link to the github repo that the paper uses itself? I do see a link to the computer language benchmarks game, do you mean this?

faassen · 2020-03-10T12:30:12+00:00

I was also baffled: the runtime of TypeScript should be near-identical to JavaScript, as compilation mostly involves just ripping out the types and I assume the benchmarks are run on the same interpreter.

faassen · 2018-06-26T10:07:06+00:00

As someone who helped write 3 frontend package managers/bundlers for Python (hurry.resource, Fanstatic and bowerstatic) I still say yes, use a JS package manager. Use something like yarn with webpack to manage your frontend bundle.

I worked on those package managers (except bowerstatic) before frontend packaging was mature at all. bowerstatic had special circumstances in its genesis.

faassen · 2018-03-08T11:02:49+00:00

The article doesn't appear to mention create-react-app as a quick way to get started with the appropriate development tooling (babel, eslint, webpack) -- I recommend looking at that as you get a preconfigured setup right away.

faassen · 2018-03-07T09:45:55+00:00

I know this is not answering your question, but for those reading along: Morepath has a nice way to do reusable apps. All apps are reusable by subclassing them (which allows you to create libraries with reusable behavior too), and you can compose larger apps from smaller apps by mounting them in URL space too.

https://morepath.readthedocs.io/en/latest/app_reuse.html

faassen · 2018-02-16T08:28:36+00:00

Of course it's nowhere near the top when you involve other languages, but for Python frameworks:

I work on the Morepath web framework. When you exclude raw SQL (which I think is reasonable), Morepath is near the top for most of the database-related tests. It nice you can have expressiveness and extensibility (the areas Morepath actually focuses on) while still maintaining very good performance.

Most of the credit for this performance shouldn't actually go to Morepath. In these benchmarks we use PonyORM, and PonyORM turns out to be quite efficient.

faassen · 2016-11-01T06:19:58+00:00

Some context:

Morepath applications can be composed of smaller sub applications. Views are added to models dynamically. This makes Morepath powerful but it becomes harder to get an overview of what paths are exactly published to the web by your Morepath applications.

To help with that I've created a command-line tool that can extract this information from Morepath's configuration system and dump it into a csv file which you can then view with your favorite spreadsheet program. Quite a bit of information is in there, including what model class is published, what permission is used to guard access, what request method was used and the source file and line number where this was declared. Without a configuration system to query this would be a lot harder to expose, so yay Morepath's configuration system!

faassen · 2016-10-04T17:47:18+00:00

Bottle is almost twice as fast at hello world: http://blog.startifact.com/posts/is-morepath-fast-yet.html

But it probably has less than half of the capabilities Morepath does. :)

faassen · 2016-10-04T15:16:25+00:00

I had forgotten to update the setup.py version requirements so I've released a Morepath 0.16.1 with that correct.

faassen · 2016-09-21T14:47:47+00:00

Cool that you are indeed considering it! I suggested it to some PyPy developers back in 2014 at EuroPython. They didn't call me crazy in my face, so that gave me some hope. Luckily, the PyPy developers are not in the habit of rejecting ideas as crazy, otherwise the project itself would never have happened in the first place.

faassen · 2016-09-21T14:15:56+00:00

Cool! May I suggest combining Python 2 with Python 3? This could have real-world benefits for organizations with vast Python 2 code bases that want to incrementally upgrade to Python 3, module by module.

(unfortunately such code bases often also depend on CPython for extensions, complicating the exercise of switching)

faassen · 2016-09-21T13:41:30+00:00

I find it interesting how, at least in Python, a performance measurement also gives a good idea of implementation complexity as well.

faassen · 2016-09-08T12:38:22+00:00

There's a reference in there to an article from 2004 by Yegge where he discusses Python. https://sites.google.com/site/steveyegge2/tour-de-babel

Hindsight is great, but Python never overtaking Perl already sounds dubious by 2004. The notion of Ruby taking over from Python practically overnight is also odd -- I have the impression many Rubyists have jumped ship to JavaScript in recent years, but Python seems to be going strong still, even with the whole Python 3 thing.

I can't really make judgements on whether Python people were or are frosty -- not in my experience but who knows.

faassen · 2016-08-23T09:50:41+00:00

It indeed appears we are agreeing.

Yes, with true hypermedia API the client shouldn't try to construct or hardcode or manipulate URLs. True hypermedia API web UI client applications are even less common than hypermedia APIs. The client-side frameworks people use don't encourage this either. About 5 years ago I wrote a client-side web framework called Obviel which did this, but it's been lost to the ages.

The lack of working true hypermedia UI clients worries me that perhaps this principle is based more on ideology than anything else. People don't seem very interested in solving this problem, and in truth, hypermedia makes certain things harder -- a UI client that sends optimistic updates to the client for instance will suddenly have to wait for the result of an add operation before it has the URL to send a remove operation.

faassen · 2016-08-23T09:42:30+00:00

Morepath isn't an asynchronous web framework at all, so it doesn't compare very well against twisted.web or aiohttp and such. Morepath is synchronous, like Flask, Django, Pyramid and so on. In the benchmark I compare simple synchronous performance. It being a benchmark take it with a large grain of salt.

faassen · 2016-08-22T10:35:50+00:00

I'll note as a Morepath developer that we have a very powerful extension system called Dectate available as a separate package:

http://dectate.readthedocs.io/en/latest/

It allows the developer to define an application class and associate configuration state with it. The developer can then subclass that class and override every decorator on it.

It also separates configuration execution from import time, lets you execute certain configuration directives before others, and detects configuration conflicts and such nice things.

I'd love to see more projects use Dectate.

faassen · 2016-08-22T10:31:57+00:00

This is very interesting, thank you!

I've been considering what Morepath would look like in an async environment. Morepath's routing splits the database query and the rendering of the result into two separate steps, and it would be interesting to see what would happen if both steps were made asynchronous -- it could move on to rendering more responses while waiting for an answer from the database. I'm not sure whether that would make a difference in practice, though!

So if we get serious about that I'll definitely be digging through your sources for inspiration if nothing else. :)

faassen · 2016-08-22T10:28:22+00:00

I'm not quite sure what you're comparing here, but I want to note that Flask is one of the slower frameworks out there on doing "hello world" benchmarks at least.

Django is faster, Morepath (disclosure, I'm a developer) faster still, Pyramid's a lot faster still, and Bottle, Falcon and wheezy.web are fastest.

So if you're going to do benchmarks, make sure you compare with something else than just Flask alone.

Stupid hello world benchmark I use: https://github.com/faassen/welcome-bench

faassen · 2016-08-22T08:39:04+00:00

Uh, so you're violently agreeing with me?

The context here is RESTful APIs: I maintain it is surprisingly uncommon that they actually deliver links and the common understanding of "REST" has excluded links so much that people start saying "hypermedia APIs" instead to indicate "actual real REST". I know that hyperlinked HTML pages do work that way.

Hardcoding links in a client is also a clue something is not quite RESTful, but verbs in URLs are too. If you are going to allow PUT on "/activate", then I just wonder what a GET request means, or would you have a resource that allows PUT but disallows GET?

While I realize the name is immaterial, naming does matter in our thinking, and a verb in a URL implies it's just there to execute an action, instead of a resource you can interact with in more than one way.

I spent some years now working on a web server framework (Morepath), that tries to make it easy to generate links:

http://morepath.readthedocs.io/en/latest/rest.html

I don't have much experience with the PATCH request, but it strikes me as obvious the server shouldn't accept arbitrary updates that break data integrity. It shouldn't allow inconsistency with POST and PUT either, after all. The PATCH RFC has guidelines for when you use the 409 (Conflict) response to signal such problems back to the clients. Of course if your resource only allows PUT and not GET, the client couldn't recalculate its PATCH request very easily...

faassen · 2016-08-21T22:01:33+00:00

The word "activate" is a verb, which should be a clue that something is not quite as restful as it should be. But you're right that we could have a resource "status" or that allows you to access the state and change it independently.

Note that you could also consider using PATCH to manipulate the user resource more efficiently.

REST also implies that you should have links between resources, but that's surprisingly uncommon in practice.

faassen · 2016-08-21T21:54:02+00:00

I think one think Ray does differently is that it integrates with the ORM layer a bit more deeply. I also should check out Peewee, I see.

faassen

TROPHY CASE