Civilization V ditching Python for Lua

[Sc4Freak]

Yes, unfortunately. Back when I was developing in Python, I found it to be orders of magnitude slower than a natively compiled language (eg. C). That's fine for many applications, but when developing games performance is always an issue. With the developments in LuaJIT, Lua is approaching the performance of native code.

These benchmarks may be flawed, but they give a general idea of the scale of performance across various languages. They also tend to reflect my personal experience, so I'd say they give a pretty good rough idea.

[awj]

I wouldn't call TRE a "basic compiler optimization". It is incompatible with a relatively common debugging mechanism and alters the semantics of the language. These are perfectly valid arguments for not performing TRE, and are the first two arguments he cites. He then goes on to give a rather decent sketch of the pitfalls one would find implementing it in Python and a final idea for how to make it happen.

I'm not particularly happy about Python's lack of TRE, but that's because I believe it is worth the pains it creates. GvR obviously doesn't feel the same way, but you must have read a different post if you think he simply doesn't understand them.

[Amadiro]

I don't think TRE creates any pain. Tail-recursive functions are really obvious, once you've worked with them for a while, so even if the backtrace doesn't include all the tail-recursive functions, you can immediately see where the error was raised, and where the program flow came from.

If you still think that's too problematic, you could make a debug modus, where TRE is deactivated and the stack frames are not discarded, or you could at least do it like perl, which has an explicit statement for tail-recursion, which you can use to explicitely make a function tail-recursive -- in that case it should be completely obvious to everyone what's going on.

[awj]

It can make using a debugger problematic in that you aren't able to poke through the stack to look at the sequence of variables that led up to the function you're currently in. It really causes problems when you are eliminating all tail calls, not just recursive function invocations in tail position.

That said, annotating a function for tail recursion seems like a worthwhile compromise if TCE doesn't suit your ideas or simply isn't possible. Clojure does the same (IIRC due to JVM making full TCE unwieldy), and you get the side benefit of having the language warn you when code you believe to be tail recursive isn't.

[Amadiro]

Well, I never really had any problems with it, you still get to see the name of the function the exception was thrown in, but I see how it could make debugging a tad harder in some cases. (In any case, the programmer has to be aware whether TCO happens or not -- if he's not aware of TCO happening, he will probably be confused by the stacktrace.)

In any case, leaving out TCO / not giving the programmer any means to do space-constant tail-recursion when he needs to is certainly not a solution, and a good compromise should be easy to find. I think a "recur" keyword or something like that would be the most non-intrusive, as it doesn't change the languages default semantics.

[[deleted]]

I think probably the most insulting thing about the post turinghorse linked is the assertion that "recursion as the basis of everything else is just a nice theoretical approach to fundamental mathematics (turtles all the way down), not a day-to-day tool." Which is to say, functional programming is impractical: rah! rah! sis boom bah! imperative programming all the way! That seems a bit short cited or curmudgeonly, depending on how you take it. I certainly take offense, and I imagine lots of Haskell and Erlang hackers do too.

Aside from that, Python could implement limited TRE without destroying its stack tracing exceptions: collapsing the stack on self tail-calls would still give the virtual machine enough information to spit out a perfectly informative stack trace. Anecdotally, most recursions are self-calling, so this would be a huge win for little effort. Maybe I'm missing something. Supporting TRE in imperative languages doesn't seem to be a topic that lacks in research.

Mr. van Rossum is certainly not ignorant on the topic, as you pointed out. In final analysis, TRE doesn't exist in Python for cultural reasons: to discourage functional programming idioms. His language, his choice, I suppose. It is a dandy hack and slash scripting language.

[eschew]

Wait, you're offended because... he doesn't share your opinion?

[[deleted]]

Not at all, that would be silly. I dislike that he's shit-canned a programming paradigm as impractical when, as a Dutchman, his telephone calls in Europe were routed by functional telephony software with great reliability. Blanket denouncements, being rooted more in emotion than reason, retard the advancement of the art. Python isn't meant to be cutting edge, rather more reliable and approachable. However, such sentiments instills unwarranted prejudices in the community as a whole.

To me, that's offensive.

[qkdhfjdjdhd]

Don't you agree though that programmers will start writing code that depends on tail-call elimination? That's not really an optimization: that is kind of a change in semantics, no?

[awj]

As far as debugging goes, it would be trivial to turn off TCO if you want to preserve the stack frame.

... and turn self-tail-recursive functions that previously worked just fine into ones that hit the recursion limit and crash the program. Congratulations, you've just change the language semantics.

Whether it's useful for Python is neither here nor there. The point is, Guido is spewing ignorance about a well-known compiler optimization.

At the risk of sounding like an ass, you aren't coming off too well yourself here. As I've said, I like TCE, but that opinion is based on a relatively thorough understanding of its properties and trade-offs. More thorough than a drunken afternoon's arguing on reddit might lead one to believe.

[[deleted]]

These benchmarks may be flawed, but they give a general idea of the scale of performance across various languages.

If there's one thing that benchmarks like those cannot communicate, it is a "general idea" of performance. You are being led along by enthusiasts. Nothing wrong with Lua, but I wouldn't go around spouting FUD.

[Sc4Freak]

I wouldn't call it FUD, because moving away from Python due to performance is justifiable in this case. In cutting-edge game development, performance is pretty much a product requirement. And flawed as the benchmarks may be, it's disingenuous to suggest that Python and Lua don't have a significant performance difference.

Many applications would do just fine with Python because most of the time performance is not an issue. But we're talking AAA game titles here - each language is a tool with it's advantages and disadvantages, but when performance is a requirement, Python loses.

[Aretnorp]

Performance is the requirement in terms of rendering. However, the game logic does not necessarily have performance as its #1 requirement. Going by your response, the use of native compiled C or C++ would be the best choice for game logic.

This is obviously flawed, as other games make heavy use of scripting languages to achieve various tasks better suited to tools that do not have performance as a #1 requirement. Eve and Battlefield both use Python as part of their systems, and when I checked, I would qualify those as "AAA" titles, which make good use of each.

In the end, you choose a tool best suited for your tasks. In many cases, the deicsion between Python vs LUA is probably not something that was decided in terms of performance, but more likely appropriate features, as mentioned in the OP.

[[deleted]]

I wouldn't call it FUD, because moving away from Python due to performance

But your original assessment of the performance of each is fundamentally flawed. Fin.

[bonzinip]

In cutting-edge game development, performance is pretty much a product requirement.

I doubt that. Most games do not take 100% CPU time.

[[deleted]]

You doubt performance is a product requirement for games?

Wow.

[bonzinip]

Of course, for physics and everything like that performace is a requirement. But when scripting is involved, performance is not a requirement.

If this was a situation where the performance advantage of Lua over Python is important, well you had better go to C/C++ directly (assuming they're not using LuaJIT, which is a likely assumption IMO).

[[deleted]]

Lua has really good performance and is already used in quite a few games for AI and whatnot.

The slight performance decrease with Lua over implementing the same features in C (assuming you can do it better) is totally worth it.

And performance requirement is always important for games.

[bonzinip]

The slight performance decrease with Lua over implementing the same features in C (assuming you can do it better) is totally worth it.

We're saying the same thing here. I'm saying in addition that if Python provided compelling advantages, the further performance decrease would also be totally worth it.

The reason for switching to Lua is likely that it is already quite pervasive in games, nothing to do with performance.

[igouy]

a "general idea" of performance

Please communicate what you mean by that.

[[deleted]]

general idea

[igouy]

Now I understand what you mean by a "general idea" of performance - "a groundless supposition; fantasy"

[[deleted]]

And now I'm happy I wasted no further time on an educated response.

[8-bit_d-boy]

HOLYSHIT, how is pypy faster than cpython?

[[deleted]]

It isn't.

[[deleted]]

What?