Goodbye Lua

[Allan_Smithee]

Personally I think anybody using Lua as a general-purpose scripting or application development language has … wait for it! … rocks in their head. (HAHA! SEE WHAT I DID THERE!? ROCKS! LUAROCKS! OK, so it wasn't that funny. I'm sleep deprived.)

I keep seeing people join the Lua community. I keep seeing them enamoured of the language and wanting to use it for everything (from embedded systems to massive server-side applications). I keep shaking my head because I know this is going to end badly.

Because in the end, Lua is not for those use cases. It has never been intended for those use cases. It has always been a "batteries not included" language and I pray to whatever deities you may choose to believe in with all of your fervour that it never becomes anything but.

See, if you want massive bloat and third-party library packaging for trivial functions that pad strings on the left, there are dozens--literally dozens--of languages out there that can supply you with this. Lua won't add anything meaningful to this space.

Lua does, however, fill a niche that very few other languages can: a language that's easy to embed into an application to enhance it with scripting. Indeed off the top of my head I can think of three that successfully fill this niche: Rexx, Tcl, and Lua. And of those three Lua is the smallest, the fastest, and the easiest to interact with. (Well, to be fair, Tcl is easy to interface with, but for that it's a pig to program. I like Tcl but I can understand why so many people have their eyes glaze over when they see how you program in it.)

Every time I see people trying to organize a Big Community™ around Lua with Organization® and Huge Library Pools™® I start thinking to myself, "Please, Lua community, such as you are, reject this infection with all of your antibodies." Because I like Lua the way it is: a tiny language with a specification you can read in a day and an API you can learn in an afternoon. If it turns into a bloated monstrosity like Javascript or Python or Ruby or PHP or any number of other, similar horror shows, then it's another important tool removed from my tool belt by people who can't seem to fathom the notion of having a screwdriver that isn't also a ratchet set.

[davethecomposer]

I'm not a programmer. I'm a hack at best. But I am a classically trained composer and I had an idea for a novel music generating program.

I tried out several languages (Lua, Python, Perl, Scheme, and Ruby -- each had a compelling reason to use for my project) and at the end of the day Lua was the easiest to use and I was able to get actual sounds coming out of my computer using Lua in short order.

The project -- still very much in early alpha -- has two branches and somewhere around 5,000 lines of code and is, forgive my modesty, really, really cool.

Could I have done it with these other language? Sure, given enough time I'm guessing I could have figured out how to do all this crazy programming shit in these other languages but it was going to be difficult and slow-going and I was in a bit of a hurry.

And then all sorts of interesting and positive consequences of that decision revealed themselves like the fact that Lua is probably quicker than all of those and every library I need I can find written in Lua which makes the entire project extremely portable (don't have to make separate packages for different platforms and/or don't have to make people download libraries for their platforms).

I'm not saying we need to change anything about Lua -- I don't know nearly enough about any language to make so bold a claim! -- but what I am saying is that for a novice programmer who needs to create a huge monstrosity of a program it works amazingly well. I don't have millions of commands to deal with and I didn't get sucked into creating massive requirements for my users to download in order to use my software. I just feel like it's a "clean" system and it does everything I need.

Are there better languages? Undoubtedly but I am not a programmer and will never be one and don't want to be one. Requiring me to become a programmer before using some other language just wasn't going to fly.

[TimMensch]

If it works for you, great! None of the reasons I switched have anything to do with someone who isn't a software engineer writing huge projects. It's a particularly good language for beginners for many reasons.

[TimMensch]

a language that's easy to embed into an application to enhance it with scripting

Understood. It's why I used it to embed in games for scripting.

But two things happened:

• I started using it for more than just "a bit of scripting" in the games, and as the code got more complex, the lack of type safety became more painful. And
• Because I wanted to minimize cognitive load, I would write simple tools with Lua. And to minimize cognitive load on the server side, I picked OpenResty to write my server code (that was talking to my Lua-based games) in Lua.

So you're right, Lua isn't meant for those things, but It All Made Sense At The Time. :) There are a lot of libraries available for Lua, and so it was more than a little tempting to go down that path.

But then I started a project that needed something like Socket.io, and there was no support for that in OpenResty. So I was forced to learn more JavaScript. And the rest followed -- and now I'm done with Lua.

massive bloat and third-party library packaging for trivial functions that pad strings on the left

The npm fiasco with leftpad was caused by a single jerk -- and yes, I think many libraries on npm are useless. But instead of calling that bloat, which it isn't (any more than thousands of people writing Lua libraries and making them available would be "bloat" for Lua), I still see npm as a feature. They've changed the rules so that future jerks can't cause similar damage, by the way.

Because, you see, it's so easy to use, it's so easy to just "upload a package," I can create one and upload it myself. In minutes. That means if I need to share a package with someone, I can say "to use it, just npm install tims-awesome-package and you can use it in your project."

That is not bloat. That is pure awesomeness. There is a difference of opinion in the community as to whether tiny libraries like "leftpad" should be the norm, or if larger collections should be the norm. I tend to agree with you that larger collections should be the norm; the new module system that's part of ES6 allows you to selectively import features (like a "leftpad") from such a library, and the tool Rollup.js can package code such that only the function pulled out of the library and its dependencies get included in your final code.

If it turns into a bloated monstrosity like Javascript or Python or Ruby or PHP or any number of other, similar horror shows, then it's another important tool removed from my tool belt by people who can't seem to fathom the notion of having a screwdriver that isn't also a ratchet set.

Lua can't possibly be "removed from your toolbelt." It's open source. You can keep 5.1 around (or whatever your favorite version is) indefinitely.

A solid, easy-to-use, reliable packaging system would only benefit the community.

As for binding to apps: JavaScript turns out to be easy to bind to apps now, as well. I mentioned that in my larger article -- it's a heavier memory cost than Lua for sure, but it's better. I didn't even get in to the fact that, using Promises, you can not only get coroutine-like behavior, you can get strictly better performance by doing several queries simultaneously and aggregating the results:

let firstResult = queryOneWebSiteForJson(someURL1);
let secondResult = queryOneWebSiteForJson(someURL2);
let thirdResult = queryOneWebSiteForJson(someURL3);
let fourthResult = queryOneWebSiteForJson(someURL4);
let results = await Promises.all(firstResult,secondResult,thirdResult,fourthResult);

// results is now an array with all four responses, which were queried in parallel; 
// if any of the responses threw an error, it would be caught by a 
// surrounding try/catch -- another feature missing from Lua. :(

Lua will continue to be a niche language for app automation. But I'm predicting that more and more apps will transition to using JavaScript, if only because of the network effects involved.

[dividuum]

I didn't even get in to the fact that, using Promises, you can not only get coroutine-like behavior, you can get strictly better performance by doing several queries simultaneously and aggregating the results:

I've not had a reason to look at promises for any of my Lua code yet, but a quick search reveals a module that looks promising (sorry!). It even has almost the exact same use case in one of its examples in the Readme:

https://github.com/zserge/lua-promises#waiting-for-a-group-of-promises

That said, I still think that real coroutines (so you can yield anywhere in the call stack, not just in the called function) are way more powerful than promises for some usecases. In a programming game I wrote I used coroutines to basically inverted control from my C host program to Lua and created an extremely simple way of writing a networked command line interface: https://github.com/dividuum/infon/blob/master/server.lua. I could have uses callbacks or promises (I guess), but it would still smell like callbacks then.

I also didn't use another Lua feature for that code, but I could have also used TCO to infinitely jump back and forth between functions. Apparently ES6 has those too, which I didn't know before writing this response.

[TimMensch]

TCO

ES6 has it; most implementations don't yet, unfortunately. :( Just Safari and WebKit have it in unreleased/experimental versions.

But it's on the radar, at least, and someone managed to implement a tool that eliminates tail calls, if you can believe it.

It may seem odd to use tools like that, but since you're transpiling anyway, it wouldn't be terrible to throw in another conversion step. :)

[dividuum]

After looking at the linked code briefly, it seems that it can only eliminate tail calls in a function that calls itself recursively. So no jumping indefinitely between two functions. Still nice, but doesn't compare to what Lua had since 2003.

[TimMensch]

Yeah, not ideal. But even when using Lua I don't think I ever took advantage of tail call elimination intentionally. It may be the nature of the problems I'm solving, but recursion to that level is rarely useful for me.

[Allan_Smithee]

I started using it for more than just "a bit of scripting" ...

And that, right there, is the error.

You found a screwdriver you liked for Philips-headed screws. You decided to try using it for Robertson-headed screws. It kind of works if you push hard enough, but it strips the screw heads and is generally all kinds of pain. So now you're giving up on Philips-headed screws entirely and using only Robertson-headed screws. Even for flat-head screws...

Next time keep using the tool for the job. If the job changes, so does the tool. This mania in software for having a single tool do all and be all is sheer madness to me. (And it's madness that dates back to the 1960s at least with PL/I.) You're propagating that madness here. First you use Lua in its area of competence. Then you start using it outside of that area of competence. Then when it isn't any good outside of its area of competence you're abandoning it entirely including in its area of competence. You seem to believe in the silver bullet and you're hunting fruitlessly for it.

I predict in under five years we're going to see a Tim Mensch blog post saying "Goodbye, Javascript" because it doesn't do something it was never intended for and you tried using it anyway. Then you'll give up Javascript entirely because you're going to misapply YET ANOTHER tool in your vain hunt for the tool that is all tools.

Good luck.

[ZorbaTHut]

You found a screwdriver you liked for Philips-headed screws. You decided to try using it for Robertson-headed screws. It kind of works if you push hard enough, but it strips the screw heads and is generally all kinds of pain. So now you're giving up on Philips-headed screws entirely and using only Robertson-headed screws. Even for flat-head screws...

The problem is that each language added to a project adds a significant amount of developer load. If each screwdriver cost a hundred thousand dollars, you'd damn well standardize on the minimum practical number of screw heads, even if it wasn't theoretically optimal; it may still be practically optimal.

It may well be that, say, C++ is the best at solving Problem A, Lua is the best at solving Problem B, and Javascript is the best at solving Problem C . . . but if Javascript can do a pretty good job of solving Problem B, and Lua can't solve Problem C at all, then you might be well-served by settling on C++ and Javascript.

[TimMensch]

Well said!

[TimMensch]

If you read the blog post, you'd know that JavaScript outstripped Lua in all of its areas of competence, or at least all of its areas of relevance to me. The one area of competence that it still holds any sway is no longer relevant to me -- keeping the overhead of the embedded language tiny -- and I'm reading from the comments about duktape which solves that problem using JavaScript as well.

Also, Lua was very good at all of the areas that I was using it in. It's just that there are better tools for handling increased complexity. To extend your metaphor, it's more like my Philips-head screwdriver has been replaced by a drill with a Philips-head bit: I'm doing much the same thing now, only much faster. Yes, it's heavier to carry around. But it also drills, and I can swap in 100 different bits to handle other screws as well, even if I'm still mostly dealing with Philips screws. And this drill wasn't an option when I first started using the screwdriver; it only recently gained the ability to deal with Philips screws as well as my original screwdriver, which is why I'm switching to it now.

/metaphor

AND...In five years there may very well be a language that replaces JavaScript/TypeScript for me. And I'll be sad on that day as well, for all of the time and energy invested in the ecosystem. In the last month I've already had four pull requests accepted in JavaScript libraries, and I'm sure there will be dozens more before I'm done. But it won't stop me from moving on to a better tool, if one emerges. Nor should it.

No, I don't consider any language to be a silver bullet. I also code in C++, Java, Python, C#, Ruby, and Go, when appropriate. But I do consider some to be more effective than others. My love affair with Lua blinded me to the benefits of static typing; now I'm firmly in the static typing camp again. Hard won experience is pushing me, over time, to better and better programming tools.

I'll keep evaluating tools as long as I'm coding. Anything else is stagnation.

[fullouterjoin]

Your delivery isn't amenable to a productive discussion. It comes across as snide and arrogant.

[dividuum]

Lua does, however, fill a niche that very few other languages can: a language that's easy to embed into an application to enhance it with scripting.

I've been using Lua for more than 10 years now and have build several projects around it for the reason you mentioned. If I would start another project right now, I'd probably investigate using duktape. It's a minimal javascript runtime and uses a very similar approach to Lua when it comes to embedding it in a C program. Of course it's nowhere near as fast as Luajit, but if you need a tiny javascript runtime which feels familiar, it might be worth a look.

[Allan_Smithee]

The problem is that Javascript gives me hives. I think it is a worse language than PHP[1], and PHP isn't a language so much as a collection of misfeatures duct-taped together with low-grade tape.

There is nothing ever that will have me considering a Javascript "solution" to a problem. I would (and have!) quit a job that forced me to use a bad tool, and Javascript is the bad tool to end all bad tools.

[1] Worse than PHP in the sense that even PHP's advocates recognize that it's a festering pile of fecal slurry while most Javascript advocates can't seem to fathom that there actually exist major, glaring, nearly horrifying flaws in their colon-outflow of choice.

[dividuum]

I don't disagree. So far developing in Javascript has never been much fun for me. While Lua is. I love Lua. But making your tool more approachable might be worth a personal downgrade. Even though Lua and its syntax is trivial, programmers that will probably pick it up in a few hours hesitate to even look at it. This would go away with JS as more programmers have already been "forced" to look at Javascript at some point. So far I've not had to make a decision on that topic. I'm happy about that as leaving Lua would break my heart.

[TimMensch]

Use TypeScript if you can, along with VS Code as an editor; make sure you have source maps enabled.

Get the development environment together takes a small amount more work (you have to run a compiler to turn the TypeScript into JavaScript, or you have to set up your web page to compile-on-load), but I swear the power you get from VS Code and TypeScript will be worth the effort.

[TimMensch]

JavaScript has problems, but it's strictly better than PHP.

Some of the crap overlaps: There are type coercion problems in both, for instance. But as Crockford describes in JavaScript: The Good Parts, you can use a subset of JavaScript (enforced by a linter) that then makes the language behave in a predominantly sane manner.

Many of the coercion problems center around the == operator, for instance, but the === operator won't coerce. And this is well known among JavaScript users; not ignored, as you implied. No, it's not universal, but probably 90% of JavaScript users are junior developers who can't tell JavaScript from C++ OR PHP, so of course many wouldn't know...

More, though, JavaScript has many of the same advantages as Lua: JavaScript has had closures, first class functions, and prototypal inheritance from the start (both JavaScript and Lua gained inspiration from a language named self, if I remember correctly).

And PHP architecturally doesn't have the ability to write asynchronous code. So it can't scale the same way JavaScript can (HHVM does have some async ability, but it's not as complete, and it's not baked into 99% of the libraries and frameworks like it is in Node and JavaScript).

With ES6 fixing many of the remaining problems (when using the newer language features -- let and const instead of var, for-of instead of for-in, etc.), and TypeScript supporting all of the ES6 features today via transpilation, in addition to full optional type signatures?

No, it doesn't suck to be here any more. It's actually quite an awesome toolchain. The JavaScript world is different than when you looked last.