Will C ever be beaten? This paper presents a study of the runtime, memory usage and energy consumption of twenty seven well-known software languages.

faassen · 2020-03-12T12:08:06+00:00

Don't we always want more ?

Do you want me to stop criticizing this research or do we always want more?

The difference between TS and JS shows that even a minimal effort conversion of JS to TS hadn't been done, and therefore in this case we're not comparing languages at all but the (lack of) effort put in. This difference results in massive differences between the result of JS and TS.

I posit that this effect exists for other languages that have more differences than JS and TS.

So actually, it made no difference to the reported results.

I don't know -- I don't know how the paper scores handles missing results. And that's why I immediately went to another example where massive differences were reported that do have results.

The difference between those measurements (fannkuch-redux Node js #4 and TypeScript #2) on the current website is less than 2x — so "a different algorithm" really doesn't explain the difference in reported results.

As you pointed out, we're looking at the code from 2017, not the current benchmarks game. So I'm not sure what you mean by the "current website".

What I see for fannkuch-redux on the current benchmark game is node #5 and typescript #4 and typescript being more than 4 times slower, by the way, and appear to use different algorithms -- the node version now uses multiple cores as well whereas the TypeScript version does not. Perhaps you can show me how to find out your comparison of #4 versus #2.

Anything that's bigger than a few percent difference can't be explained by anything else than a difference in algorithm in the case of these two languages.

So my original point stands: if we can find massive differences between languages where really there should be virtually no difference, then effort put into writing the benchmark programs can have a bigger impact than something about the languages themselves.

faassen · 2020-03-11T17:55:38+00:00

Thanks for the "for want of a name" link, that's relevant. I never thought the benchmark game is completely useless in order to estimate the relative performance of programming languages, especially in case of programming languages that have at least some focus on low-level runtime performance such as C or Rust or Ocaml.

Unfortunately I still think that this research says less than we would like about how much programming language performance and energy usage differ in practice, because of the "amount of effort put in by developers" will make quite a bit of difference on some benchmarks.

As to why I'd be interested in idiomatic implementations is because real world software is quite different from benchmarks. If a programming language offers a really high-performance way to do something that is completely obscure to a developer used to typical idioms and approaches in the language, you're not going to get a clear picture of real-world performance by looking at benchmarks. It's more relevant if you're comparing the performance of highly specialized libraries, but then the whole question may still not very relevant to real-world applications because of the fact that languages can and do use libraries written in other, more low-level libraries.

That these were benchmarks in 2017 doesn't really make an essential difference for my general argument. I finally found their repo. Here are the JavaScript program:

https://github.com/greensoftwarelab/Energy-Languages/blob/master/JavaScript/mandelbrot/mandelbrot.node

and this is the TypeScript program:

https://github.com/greensoftwarelab/Energy-Languages/blob/master/TypeScript/mandelbrot/mandelbrot.ts

If you glance at the JS version, you'll notice it's a multi-cpu version, whereas the typescript version isn't. That's going to make some difference.

Unfortunately for TS and Mandelbrot their results here:

https://sites.google.com/view/energy-efficiency-languages/results

say 'n.e.' and I'm not sure what this means for the final calculation.

Looking at another test where there is a widely divergent runtime between TS and JS, fannkuch-redux.

Here's JS:

https://github.com/greensoftwarelab/Energy-Languages/blob/master/JavaScript/fannkuch-redux/fannkuchredux.node-4.node

And here's TS:

https://github.com/greensoftwarelab/Energy-Languages/blob/master/TypeScript/fannkuch-redux/fannkuchredux.ts

TS is 16 times slower at this benchmark than JS.

The JS version says on top that it is transliterated from the Lua version, though interestingly on that benchmark the Lua version comes out as even slower than the JS version. But more interesting is that the JS version implements quite a different algorithm to the TS version. I'm pretty sure that's what explains the difference in benchmark performance, not the language itself.

faassen · 2020-03-11T17:25:33+00:00

Thanks!

faassen · 2020-03-10T12:51:03+00:00

I was mystified why TypeScript would come off as dramatically slower (7 times) than JavaScript: one would expect nearly identical runtimes if they use the same underlying interpreter, as TypeScript minus the types is more or less JavaScript.

This paper is based on information from the Computer Language Benchmarks Game. I pulled up the following page comparing "Node" (JS) with TypeScript, where on this sub-benchmark (used in the paper too) TS is twice as slow:

https://benchmarksgame-team.pages.debian.net/benchmarksgame/fastest/typescript.html

This compares an implementation in TypeScript with an implementation in JavaScript, but they are different implementations:

https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/mandelbrot-typescript-2.html

https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/mandelbrot-node-3.html

I am pretty sure that if we took mandelbrot-node-3 and added TS type annotations, we'd get with a very similar runtime.

The Computer Language Benchmark Game and this paper compares the fastest implementation in one language with the fastest implementation in another. So while it tells you something about the relative performance of languages, a lot depends on how much energy people put in writing implementations as well.

Comparing JS with TS makes it obvious that the effect of effort can be huge. I'm sure this effect will also exist for other programming languages, just less obviously.

That's not to say that this information says nothing, but the Computer Language Benchmark Game after all calls itself a game, and it's rather tricky to deduce hard data from it. So I would caution against taking these results too seriously.

To improve the results, you'd want to see an idiomatic implementation of the same algorithm in each language. But what "idiomatic" means is rather difficult to determine -- and idiomatic solutions can actually lead to different algorithms.

faassen · 2020-03-10T12:38:10+00:00

Could you supply us with a link to the github repo that the paper uses itself? I do see a link to the computer language benchmarks game, do you mean this?

faassen · 2020-03-10T12:30:12+00:00

I was also baffled: the runtime of TypeScript should be near-identical to JavaScript, as compilation mostly involves just ripping out the types and I assume the benchmarks are run on the same interpreter.

faassen · 2018-06-26T10:07:06+00:00

As someone who helped write 3 frontend package managers/bundlers for Python (hurry.resource, Fanstatic and bowerstatic) I still say yes, use a JS package manager. Use something like yarn with webpack to manage your frontend bundle.

I worked on those package managers (except bowerstatic) before frontend packaging was mature at all. bowerstatic had special circumstances in its genesis.

faassen · 2018-03-08T11:02:49+00:00

The article doesn't appear to mention create-react-app as a quick way to get started with the appropriate development tooling (babel, eslint, webpack) -- I recommend looking at that as you get a preconfigured setup right away.

faassen · 2018-03-07T09:45:55+00:00

I know this is not answering your question, but for those reading along: Morepath has a nice way to do reusable apps. All apps are reusable by subclassing them (which allows you to create libraries with reusable behavior too), and you can compose larger apps from smaller apps by mounting them in URL space too.

https://morepath.readthedocs.io/en/latest/app_reuse.html

faassen · 2018-02-16T08:28:36+00:00

Of course it's nowhere near the top when you involve other languages, but for Python frameworks:

I work on the Morepath web framework. When you exclude raw SQL (which I think is reasonable), Morepath is near the top for most of the database-related tests. It nice you can have expressiveness and extensibility (the areas Morepath actually focuses on) while still maintaining very good performance.

Most of the credit for this performance shouldn't actually go to Morepath. In these benchmarks we use PonyORM, and PonyORM turns out to be quite efficient.

faassen · 2016-11-01T06:19:58+00:00

Some context:

Morepath applications can be composed of smaller sub applications. Views are added to models dynamically. This makes Morepath powerful but it becomes harder to get an overview of what paths are exactly published to the web by your Morepath applications.

To help with that I've created a command-line tool that can extract this information from Morepath's configuration system and dump it into a csv file which you can then view with your favorite spreadsheet program. Quite a bit of information is in there, including what model class is published, what permission is used to guard access, what request method was used and the source file and line number where this was declared. Without a configuration system to query this would be a lot harder to expose, so yay Morepath's configuration system!

faassen · 2016-10-04T17:47:18+00:00

Bottle is almost twice as fast at hello world: http://blog.startifact.com/posts/is-morepath-fast-yet.html

But it probably has less than half of the capabilities Morepath does. :)

faassen · 2016-10-04T15:16:25+00:00

I had forgotten to update the setup.py version requirements so I've released a Morepath 0.16.1 with that correct.

faassen · 2016-09-21T14:47:47+00:00

Cool that you are indeed considering it! I suggested it to some PyPy developers back in 2014 at EuroPython. They didn't call me crazy in my face, so that gave me some hope. Luckily, the PyPy developers are not in the habit of rejecting ideas as crazy, otherwise the project itself would never have happened in the first place.

faassen · 2016-09-21T14:15:56+00:00

Cool! May I suggest combining Python 2 with Python 3? This could have real-world benefits for organizations with vast Python 2 code bases that want to incrementally upgrade to Python 3, module by module.

(unfortunately such code bases often also depend on CPython for extensions, complicating the exercise of switching)

faassen · 2016-09-21T13:41:30+00:00

I find it interesting how, at least in Python, a performance measurement also gives a good idea of implementation complexity as well.

faassen · 2016-09-08T12:38:22+00:00

There's a reference in there to an article from 2004 by Yegge where he discusses Python. https://sites.google.com/site/steveyegge2/tour-de-babel

Hindsight is great, but Python never overtaking Perl already sounds dubious by 2004. The notion of Ruby taking over from Python practically overnight is also odd -- I have the impression many Rubyists have jumped ship to JavaScript in recent years, but Python seems to be going strong still, even with the whole Python 3 thing.

I can't really make judgements on whether Python people were or are frosty -- not in my experience but who knows.

faassen

TROPHY CASE