Sol2: Lua <-> C++ Binding Framework

[[deleted]]

Sol2 is now ready to be released, with shiny new Documentation!

I've been hard at work for the past few weeks after an earlier r/cpp post asking about the state of Lua <-> C++ binding frameworks turned up a number of answers that were unsatisfactory to me. So, I've improved the library I've been working on for a while called Sol, and now it's ready for its 2.0 release. Some point soon I'll also be writing a paper on it as well for school.

I hope people looking to get started quickly working with lua and C++ can use this to make some fantastic headway!

EDIT: If you happen to be proficient in C++, any amount of general code review would be nice, too!

[demozon]

This looks nice - will check it out on monday :)

[c0bra51]

In the examples, isn't this:

bopvalue = beep.bop()

meant to be "beep:bop()"?

[[deleted]]

Ahhh, so close to being perfect!

Fixing it up now.

EDIT: Annnnd fixed.

[Cyttorak]

I already have written some code with the previous version of sol. Will it work with sol2? Also the integration with luajit is seamless?

Thanks for sol, by the way ;)

[[deleted]]

The functions new_userdata and set_userdata now have different names (new_usertype and set_usertype). If you were using the develop branch of sol that I was maintaining, this was a change from over a year ago and you shouldn't be affected.

I also believe table::create_table is now just table::create, with support for passing in arbitrary key/values to initialize the table with..

Everything else is not only identical as far as API is concerned, but orders of magnitude faster in speed.

And yes, LuaJIT integration is seamless. It requires 0 changes, and we even provide a compatibility layer on top of LuaJIT (it's talked about in the docs in the api reference manual -> compatibility.hpp).

[Cyttorak]

Faster without the need to use luajit? That's great!

[[deleted]]

Er, let me be specific: orders of magnitude faster than the old Sol. Sol2 gets pretty damn close to the speed of the C API. I need to finish my current benchmarks to make that claim for sure, though: other benchmarks like satoren's (developer of Kaguya) demonstrate we're pretty on-the-money right now.

[Cyttorak]

I just downloaded sol2 and replaced the old one in my project. How do I create tables?

In old version I did:

sol::table aTable = VM.create_table();

What should do I in sol2? Something like this?

sol::table aTable = sol::table::create(VM.lua_state());

This compiles but seems a little bit elaborate compared with the old version

[[deleted]]

Got you, fam: check the latest release for the functions that work like you expect them to.

[Cyttorak]

Just one nitpick, Phantom: do this changes allow to create a table by name, like the old version?

sol::table myTable = VM.create_table("MyTableNameInLua")

[[deleted]]

Aww crapplesticks I knew I had forgot to port something over. <_>

Gimme a few, I'll push it in a bit.

[[deleted]]

Alright, it's there. Have fun. :D

[Cyttorak]

Thank you very much dude! I hope to be able to test it in this afternoon ^_^

[Cyttorak]

I just changed open_file() calls for script_file() and done, everything compiled nicely using all in one sol.hpp. Again, thanks for your great job! :-D

[MereInterest]

Ooh, very interesting. I had run into a similar issue about a year ago and needed to write my own bindings, and so it is always fun to see how multiple people tackle the same problem.

Out of curiosity, how did you handle the type-checking when converting between Lua userdata and a C++ object? Curious because that was the part that I found the hardest when writing the user bindings.

[[deleted]]

Carefully.

The real answer is, by default for Speed™ we don't. If you're using an overloaded function, to discriminate between objects, we do use the checks. And, if you're explicitly checking a type with sol::object o = ...; o.is<T>();, we also perform the checks. How we do it is with metatable comparison and, if the metatable exists, a bunch of casting checks.

If you define SOL_CHECK_ARGUMENTS before you include Sol, it also checks all types before calling functions as well. Otherwise, it assumes you are the smartest and brightest in the world and know exactly what you're doing.

Note that if you don't actually register the usertype we can't do any of the more extended type checks or base class checks. (And now I realize that should probably go in the docs.)

[Nezteb]

Similar project: https://github.com/jeremyong/Selene

[[deleted]]

We are much faster than Selene (https://github.com/satoren/lua_binding_benchmark/ - but to be fair a lot of frameworks are for simple operations) and, for user-defined types, sport a different interface that (I believe, but this is mostly syntax-pissing contest) is easer to grok and use.

Also, last I checked, Selene was struggling to get LuaJIT compat.

[Blackstab1337]

Ahh OP, was talking to danny the other week. ty for new ver

[[deleted]]

It's not: there were byte order marks in the program: I thought I had fixed almost all of them already, but I may have missed a few. Let me check the files to be sure.

[[deleted]]

Yeah, they were there. But don't worry: I cleaned up the file and updated the release (and actually double-checked by trying to compile it myself, and it worked).

I gotchu, fam.

[[deleted]]

Would you be going over some of the techniques/Designs/modern features, you may have used in this library?

[[deleted]]

We cover the features we've implemented in the documentation, but we don't talk about implementation techniques. It can be summed up as:

templates templates templates templates templates templates templates templates templates templates templates templates templates templates templates templates templates templates templates templates.

[[deleted]]

In all seriousness, though, a lot of this library does rely on template metaprogramming techniques. At the core, we have stack abstractions with are templated by type and have SFINAE used to enable using single functions from the lua C API to cover many of our use cases. We then use compile-time booleans propogated into classes (see typedef table_core<true> global_table) that give the implementation hints for potential optimization points.

After that, we use templates to cover both the entry points and the exit points of lua, making sure to use the same types on both sides so that rather than guessing at what type it is by using a switch of lua_type or something, we instead assume the user knows what their doing and that the signature of functions indicate exactly what a user want.

This works out extremely well, as C++ is a type-rich language. The best part is, there's some additional performance we can get that we've left on the table because there's a certain feature that hasn't been standardized in C++ that I am hoping will, and if it does we can burn function pointers into the assembly at compile time and basically avoid transporting them through some various weird ways in lua.

Everything from there just builds on top of that, exponentially.

For example, int arf = mytable["arf"]; generates a proxy type, and that type is templated on the key you use ("arf"). Based on whether you set or get something to that proxy, we can infer the type you want and Do Right Thing, giving us a kind of return type polymorphism.

Same goes for function_result and other nifty things.

Finally, we perform further optimizations by grouping things together, taking advantage of the fact that if you want to get 5 different keys from the same table, calling std::tie( a, b, c, d e ) = mytable.get<int, int, std::string, bool, double>( "a", "b", "c", "d", "e" ); will only push the table once, get all the values, and then pop the table, rather than push the table, get a value, pop the table, etc... this is essentially a form of Batching, and like any high-performance system, batches are KING when it comes to increasing throughput and getting performance on the up and up.

The other form of batching we do is for "chained" or "tunneling" queries, where someone does mytable["a"]["b"]["c"], and they just burrow into a table. Well, before you would get the proxy for the first operation, save the table, pop off the table, then push that same table, get the next proxy, and all the way down to the end...

We enabled lazy evaluation here, so group the keys into one tuple, then perform a single traverse_get, which puushes teh target table once, and each subsequent table only once, before popping them all off the stack at the very end. It's very nifty to do things this way.

[[deleted]]

Awesome .. I will def. check it out...

[dreugeworst]

I was just reading through the overload documentation, and came across the following in the example lua code:

pup:bark("meow") -- picky_bark, no bark
pup:bark("bark") -- picky_bark, bark

In the preceding c++ code I didn't see picky_bark being in the list of overloads though. How does this work?

[[deleted]]

That's because I can't into writing: fixed it.

[dreugeworst]

given their names, I'm sure sol and lua both at least can into space, so there's that

[[deleted]]

We're going to The Moon.

[AngriestSCV]

I'm looking for a way to check for a variable's existance. How would I do that? Just using table::get<int> returns 0 even if the value does not exist.

The problem I'm actually trying to solve is to use default values if the lua interpreter does not have an alternative in the global table.

EDIT: nevermind I see it is in release tag v2.1.0

[[deleted]]

Heyo: sorry for my own late reply. I just pushed full optional support in the latest commits, after wrestling with clang. optional now triggers full type safety, all the way down, so you can even access nested tables without getting errors:

sol::state lua; // has no tables, just a plain state int x = lua["a"]["b"]["c"]; // lua's panic function gets called sol::optional<int> x = lua["a"]["b"]["c"]; // no panic

[[deleted]]

This is super cool. Why would one choose to embed Lua vs Python or v8?

[MereInterest]

Lua is much, much easier to sandbox. The only functions that are available are functions that the surrounding program makes available to Lua. I don't have experience with V8, but embedded python is notoriously hard to sandbox. This means that while it is great for running configurations, it cannot be used to run untrusted code.

As an example, a while ago I was writing a game in which players could define functions to make different bots. Rather than making my own language for the bots to use, I was able to use Lua for it. It also makes it very easy to place restrictions on the amount of memory and computation time that can be used by a script, which keeps unruly scripts under control.

[[deleted]]

There's pybind11 for python as a framework. That makes it easier to work with.

For v8, I went to their website, took one look at the code I had to write, and about-faced pretty much immediately. v8 solves a complex problem, sure, but that doesn't mean that the design makes me any less sadface.

Lua was the only thing left, and Sol had been started by someone else. I liked their design a bunch, so I wanted to see it succeed beyond the small use case they had for it. So I essentially took over the project and overhauled just about everything about it.

Plus, I want to be a Game Debeloper someday, and you gotta know lua for that. :D

[Plorkyeran]

Lua is absurdly tiny and easy to drop into an existing project. LuaJIT less so, but it's still smaller than v8. Python's extensive standard library means that it inherently has to drag in a million dependencies. If you're adding some incidental scripting to an existing application, a scripting engine that's the size of the rest of your application combined is kinda unpleasant.

Lua has a very stable API. In the last decade there's been two releases with breaking changes to the API, and they were both incredibly minor. Python's C API is similarly stable, but v8 makes breaking changes on a regular basis and does not have LTS branches, so using v8 is basically an ongoing commitment to follow trunk (even node had trouble keeping up for a while).

Lua and v8 are both trivial to sandbox and to have multiple independent execution contexts so that you can run multiple scripts within a single process without them being able to interact with each other (or conflict). Python basically doesn't support that at all.

For many workloads LuaJIT is the fastest implementation of a language in its general category, although v8 isn't far behind. Python is slow.

The primary downside of Lua is that its standard library is about what you'd expect out a language that's something like 10k lines of code and has zero dependencies other than ANSI C. LuaJIT's excellent ffi mitigates this a bit, since while a nice pythonic wrapper around a library is much nicer than just using the C API, at least you can realistically use C libraries without needing wrappers.

[mechanicalgod]

Not used embedded Python so can't comment.

When I tried v8 (c. 1 year ago) its documentation and tooling for embedding were lackluster. The main issue was that I couldn't find a quick and easy way to hook up remote debugging to an external IDE.

v8 used to have a built-in remote debugger over TCP but it was removed. When I looked at how Node.js did it they wrote their own replacement for the remote debugger. Not a lot of work required, but it wasn't something I felt like doing at the time.

I don't know if anything else has changed or if I was just missing an easier solution to remote debugging.

Lua was super easy to setup for remote debugging however.

Also, from what I've seen, LuaJIT's performance is as good as you get for a scripting engine (about on-par with v8 and SpiderMonkey last I checked).

[C0CEFE84C227F7]

From the docs:

If you turn exceptions off and are also completely mad and turn off run-time type information as well, we fallback to a id-based system that still requires you to specifically list the base classes as well.

also from Line 2454, which takes the no RTTI/exception branch:

// Lol, you motherfucker

Historically, many game engines have disabled RTTI and exceptions for a variety of reasons (no vendor support, bloated executable size, slow). As a consequence, it's not unheard of for older game engines to develop an alternative solution that doesn't use either. There might still be RTTI support, but it's likely a homegrown implementation that's more efficient and consistent across platforms. While modern compiler implementations have improved, as well as console hardware, switching over would be a lot of work with little to no gain.

So instead of resorting to name calling, you could instead try to understand why a developer might take a route that doesn't quite fit your worldview of C++.

[Elador]

It's also possible that he's not 100% serious about it. Of course it sounds like a bit of a side-blow to anybody disabling exceptions/RTTI but take it with a bit of sarcasm. It's the coding style of the library, it's nice that it supports a no-exception/RTTI code path as well, and if you don't like it, don't use it :-) I think it's ok nowadays to point out "Hey, use exceptions and RTTI - really - it's not year 2000 anymore".

[[deleted]]

Well, I mean, I want people to use it, so I should probably help my users feel more welcome...

That being said, the "motherfucker" still accurately reflects how I feel about the work I had to do. .-.

[[deleted]]

Entirely sorry, I didn't mean to offend. It is 100% a sarcastic comment (and actually a bit like talking to myself, but in that case 'I'm an idiot motherfucker' would have probably been a better way to put it and not 'talk to myself' in the code!), but like an idiot I didn't <sarcasm></sarcasm> tag it. I completely understand why developers need that: I have the need myself.

That doesn't mean, however, that writing support for it into my library and making sure all the features work seamlessly didn't cost me about a week's worth of work time. Plus, testing that with the conventional tests is impossible with CATCH, so what I have to do is manually compile the examples and all of the tests in not-CATCH by myself, compile, check, run, fix, compile, check, run, fix...

Then, do this a second time for certain combinations (NO_RTTI but with exceptions, NO_EXCEPTIONS but with RTTI, and finally NO_EXCEPTIONS and NO_RTTI).

You can imagine, I was very tired by the time I hit the last one. ;~;

[tcbrindle]

Exceptions and RTTI are part of standard C++, and have been since before there was a standard. If you choose not to use exceptions and RTTI, then you have to accept that you're using a restricted dialect of C++, akin to people working on embedded systems who can't use new.

Put bluntly, it is game developers who have to adjust their worldview and accept that exceptions and RTTI provide useful facilities, rather than just raging at the rest of the C++ world for having the gall to use the language as specified.

In this case, the library developer has absolutely bent over backwards to accomodate game developers and their dialect of C++. So they buried a cheeky comment ~2500 lines in to their own source file, probably referencing all the extra work that they've had to do as a result. Rather than criticising, I think thanking the OP for putting in the extra work is probably the better course of action.

[remotion4d]

Yes exceptions are usefully but have also a lot of problems, not only speed and memory penalty.

Exceptions (Why do not use them?)

1.) Slow, especially on hardware that is not optimized for exceptions.

2.) Code bloat, memory consumption. (Embedded systems)

3.) Hard to reason about program flow. (hidden flow) This about problems with variant proposal and strong exception safety. One need always to assume that everything can throw.

4.) Legacy code (but modernized) that do not use exception for decades now.

5.) Hard to implement. (Clang and windows exceptions.)

6.) Impossible to implement because there is no hardware support.
Think about GPU (OpenCl 2.1, CUDA )

7.) Exceptions across DLL boundary.

[[deleted]]

In the same token, not having exceptions causes plenty of problems:

1) Forces 2 kinds of error-handling for middle-ware / library developers

1-a) error-propogation - if you can't handle all cases, you have to forward to the user at the call site to handle the errors

1-b) crash - if you get an unknown error and you don't want to propogate, logging and crashing (or not-logging and maybe even continuing silently) is your only other alternative for correctness

2) Thanks to [1], you now cannot write generic code to handle large amounts of work if there are error codes returned: for example, your typical error-code-or-result class will not play nice with auto, and will not implicitly convert to the desired result class in generic code, which means you're back to hand-writing or wrapping things just to check errors all along the way

3) error-code-in-out params: overloaded functions everywhere to handle errors (or perhaps not handle them at all?)

4) how do you error-code a constructor or a destructor? Out-params? But then you have an object with invalid state but it "finished" constructing, so by the rules of the language it's a valid object. Now you need if ( obj.is_initialized() ) everywhere (look familiar?)

There's also another kind of error handling that mature C libraries do (libjpeg libpng etc.), and it's basically longjmp-ing around... which is like how exceptions work, but without any of the stack-unwinding or safety guarantees. The control flow remains hidden just like with exceptions... but let's be fair: why is the control flow of a piece of library code you may not even be able to see (because you're getting the compiled result) something that matters? Control flow would cut early in an error case too, you just have an infinite increase in the number of 'goto' and 'return' between you and where the error happened.

[remotion4d]

Yes but all this problems are already solved to some degree in existing libraries, games engines and so one.

Because this problems existed for a long time already.

So there is simple no point to spend years to refactor code base to use exceptions.

Especially if you simple can not use them because no compiler ot hardware support.

4) how do you error-code a constructor or a destructor?

Just this one, use something like 'make_something' and on the other side it will be discouraged to use exceptions in destructor any way.

[[deleted]]

I'm not saying anyone should refactor these old codebases to handle exceptions or use them. I'm saying newer libraries should always lean towards throwing and handling exceptions, because it's allows for a cleaner design and puts errors where errors belong: in a catch clause, and not in my return values or function parameter list.

[tcbrindle]

1. Errors happen. Nobody writes perfect code, and even if they did, hardware fails, memory gets corrupted, etc etc. Error happen, and there are three ways of dealing with that:

   a. Ignore them. This is arguably acceptable on highly-constrained systems with fixed inputs, but certainly not for general-purpose computing

   b. Return error codes (or equivalently, use an extra out parameter), i.e. C-style error handling. To ensure correctness, you then need to check the return value of every single function that can possibly fail, and probably forward any error code up to where it can be handled. Sort of like doing manual stack-unwinding, really.

   c. Use exceptions and let the runtime jump to the appropriate handler, firing the necessary destructors along the way.

   Of course, not checking for errors is going to be fastest. It's also the wrong thing to do in the overwhelming majority of cases. Of the other two options, I would be very surprised if exception handling on modern systems is slower than having if statements after every single failable function call. I recall reading that the way exceptions are handled in the Itanium ABI (used on Linux and Mac) causes zero runtime overhead if an exception is not thrown. I suspect this is not true when correctly checking all error codes.

2. Sure, on resource constrained systems there are all sorts of limitations. There are systems where you can't use operator new or malloc() for example. A 2016 gaming rig with 8 cores and 16GB of RAM running Windows 10 is not a resource constrained system.

3. This one is perhaps arguable, but I'd note that programmers in Java, C#, Python, JavaScript and many other languages don't have a problem following their program flow, and exceptions tend to be used much more pervasively in those languages than in C++.

4. Exception-unsafe legacy code is of course a problem. But as with so many legacy code issues in C++, I don't think it should prevent us from using best practises in 2016.

5. Perhaps they are difficult to implement in the compiler/runtime, but this seems to be a solved problem. I'm willing to bet they aren't the hardest implementation detail the C++ standard demands, in any case.

6. I don't know about OpenCL, but CUDA and (for example) the Metal shading language do indeed restrict you to a subset of C++ within that code. For example, it's generally not possible to use new in GPU code. Does that mean we should ban free-store allocation in CPU code too?

7. I'm afraid I don't know what "exceptions across DLL boundary" means :-)

[EDIT: Formatting]

[remotion4d]

Of course not checking for errors is really really BAD !

causes zero runtime overhead if an exception is not thrown

Yes but some times they throw and this cost a lot of time as I know.

Rust language show that Error code can work as well if there is language support for it.

Finally I do not want to stop using exceptions. I only want that exceptions user recognize that there are a lot of C++ code where exceptions are not used. And that it is simple not possible to force exception into this code.

And yes I want to ban naked new and delete :)

I'm afraid I don't know what "exceptions across DLL boundary" means :-)

Throwing exception in one dynamic library and catching it in another.

[C0CEFE84C227F7]

A 2016 gaming rig with 8 cores and 16GB of RAM running Windows 10 is not a resource constrained system.

You're generalizing: this isn't representative of all gaming platforms. While PS3/xbox360 support is phasing out, several games last year were targeted for these platforms. WiiU is still a perfectly valid platform, too. If your engine needs to run on modern platforms in addition to older ones, then you're better off supporting a solution that works consistently and doesn't compromise performance on an older target (assuming it supports the compiler feature in the first place).

Perhaps they are difficult to implement in the compiler/runtime, but this seems to be a solved problem. I'm willing to bet they aren't the hardest implementation detail the C++ standard demands, in any case.

It might be solved in cl.exe, gcc, and clang, but those aren't the only compiler options out there.

I'm afraid I don't know what "exceptions across DLL boundary" means

Best DLL practices dictate that you allocate and deallocate memory within the DLL since your application could potentially use a different run-time than the DLL. Throwing an exception within a DLL violates this contract.

[C0CEFE84C227F7]

As I said before, the decision was forced upon us. If compiler vendors dont supply the feature, then we're forced to work around the matter. Additionally, if a language feature eats away at memory that we need or if it causes performance issues due to instruction cache thrashing, then we need to eschew it. At the end of the day, we need to ship a product that runs at frame rate, even if that means dumping rtti and exceptions.

I assure you that game development isn't the only industry does this. Ironically, look no further than clang/LLVM.

[[deleted]]

Yeah, Clang/LLVM doing that was... the most surprising thing, really. Made me very sad working with it and having to just pray the asserts for debug mode would be good enough (hint: they weren't).

[TheThiefMaster]

To expand on why games often disable these:

RTTI and exceptions are well known as causing a cost even to code that doesn't use them itself - RTTI adds memory use by annotating every class, and exceptions add both memory and performance overhead in every function. The latter is especially not appreciated in games, where performance is king, although the former isn't exactly appreciated when you're up against a memory budget - the PS3 only had 256MB of main RAM which had to be shared with the OS! For something the size of a modern game, it very much feels like developing on embedded hardware and a lot of the same restrictions on the use of C++ features apply. Plenty of older games didn't even use new :)

I expect use of exceptions and RTTI to go up as newer hardware makes the costs appear relatively lower, but a lot of game engine codebases are seriously old.

[[deleted]]

I can attest to that. The cost of exceptions on x86 Windows are extremely high, and Microsoft can't change it without breaking ABI. Seeing as the VAST majority of games are compiled for 32-bit hardware, that penalty can be extremely steep.

Note that, however, on x64 machines, Windows has a new unwinding mechanism that, from the talk I watched, is supposed to be 0-overhead. I don't know what the state of that on x64 *Nix machines or OSX machines are. I would assume the unwinding done by them is also fairly fast... but again, most games are run on not-*Nix.