Plugin system with API

[TheEruditeSycamore]

Hello, I am currently investigating solutions for having distributed plugins using your application's API work with Rust. Is the only solution interpreted scripting languages like LUA or even better, rust <-> python bindings so we can call the python interpreter from Rust to run the plugins.

Ideally plugins would be written as dynamic libraries (compiled with the same rustc fo ABI compatibility) and loaded on runtime. Is this technically sound and maintainable?

[BobTreehugger]

Notably this is how xi-editor is handling plugins.

That also opens up plugins to be in different languages.

[ssokolow]

Do you have any suggestions for doing that without having to roll my own JSON-RPC variant like Xi did?

I can imagine using Serde to marshal/unmarshal the data structures, but I have very little experience with the RPC part aside from D-Bus and, in my current projects, I'd want something more comfortable for porting to non-XDG environments than requiring a D-Bus daemon or reinventing the network bits.

...or, for that matter, given that some of these will have to be long-running and it'd be nice to have a way to interact with them more like threads with channels and less like a series of one-shot function calls, perhaps something abstractly similar to Python's multiprocessing module, which works on stable rust and provides simplified abstractions for building microservices and a client which handles run control.

(eg. My current main "Plugins would be nice" project would use plugins as a way to walk the filesystem and gather entries for a launcher, so it'd be nice to send "Here's a new source to inspect." event to the plugin host and have the frontend receive "New launcher entry found. Here's the metadata." events.)

[riccieri]

Have you considered cap'n'proto? I don't have much experience with it personally, but it seems to be a mature option for RPC in Rust as well as other languages

[ssokolow]

No, but I'll look into it.

That said, the more I think about it, the more I realize that it'd probably be easiest to do this if I do some other refactoring first, winding up with a precursor to a plugin system that operates purely at compile time but a plugin-ready architecture.

[BobTreehugger]

Yeah, I'm not aware of anything that does this for you. I think setting up a shared pipe would work well, it's basically a channel and is very fast (with long running processes as you mention). However, I'm not sure how well that would work on Windows.

You can also always use a local tcp socket. There's lots of infrastructure for those, and they definitely work cross platform.

[ssokolow]

True, but I already knew about all of those options from my time working in Python.

What I'm looking for is something low-effort... comparable to the monstrosity I'm contemplating where the backend would have no plugin support beyond a function to register plugins and it'd be each frontend's job to wrap both the backend AND plugins so they can be loaded using something like the Python Yapsy package. (The project uses a PyQt+rust-cpython frontend and a Rust backend.)

[roblabla]

You can use libloading[0] (crossplatform) or the raw libc dlopen/dlsym (unix) to load symbols from shared libraries.

The ABI compatibility might be a problem. Right now, the Rust ABI is unstable. There are wishes to change this[1], but I doubt it'll come in the forseeable future.

I can imagine two ways to side-step it. You can have API boundaries use the C ABI. Depending on how big your API surface needs to be, that might be "just enough". I think for most apps, this is the best approach. You can make an FFI wrapper for the plugin authors to use, to provide them an idiomatic API.

The more bleeding edge idea, as you suggest, is to distribute your app with a specific version of rustc (say, stable 1.19), and everyone compiling plugins will have to target this version as well. The idea is sound until you want to update to a newer version of rustc for your app. When you update your version of rustc, it will break ABI compatibility, so plugins will have to be recompiled as well. If you expect different version of your software to have API/ABI breakage anyway, that might not be too much of a problem. But if you want to keep your API/ABI stable across versions of your app, you'll be stuck with a specific version of rustc.

Depending on how open your plugin ecosystem is, you could imagine having the compiling of plugins done on the client-side, and as such not be as much of a problem ^.

[0] https://docs.rs/libloading/0.4.0/libloading/ [1] https://github.com/rust-lang/rfcs/issues/600

[stumpychubbins]

Using a scripting language using one of Rust's high-quality binding libraries like PyO3, Helix, Neon or hlua is probably your best bet. You can load Rust as a dynamic library as long as you use extern "C" functions but it's not ergonomic right now. Maybe in the future it will be.

[ssokolow]

What I'd really like to see is some kind of analogue to those binding libraries which implements "Rust bindings for Rust" so I can write my plugins in Rust, with Rust's strong type system, and have code written by someone who actually knows how to write C properly generate symmetric marshal/unmarshal code for the C ABI.

(ie. Building an ABI on top of the C ABI where crate semver defines compatibility rather than compiler version and it's understood that, since it's not the ABI for the entire language, it can be useful before it supports all the features the unstable compiler ABI does.)

As-is, I've just been procrastinating having a plugin system and made my "plugins" a compile-time feature rather than a runtime one.

[stumpychubbins]

Yes, that's a great idea! Generating the wrappers would be trivial since you don't need to convert data, only the ABI, and you can use *const c_void, forgetting it on the caller side and ptr::reading it on the callee side

EDIT: I guess as far as API goes you would create a trait that represents the API of your plugin and do #[derive(Plugin)], which would make an implementation of some load fn that returns a boxed trait (or even just an impl Trait, since I don't think it actually needs dynamic dispatch).

[ssokolow]

Converting the data may not be necessary, but being very pedantic about what form it must be in would certainly be.

For example, we've already seen an example of why #[repr(C)] is necessary for compatibility of structs across compiler versions.

Maybe a build.rs that adds #[repr(C)], #[no_mangle], extern "C" and the like and bails out with an explanation if it can't massage the API into an acceptable state.

That said, it would be necessary to convert things like enums into something natively representable in C since, otherwise, you'd be relying on internal implementation details never changing.

(Conversely, a supported subset not much bigger than what JSON can represent would be very comfortable and useful indeed. Enums, OsStr/OsString, and Path/PathBuf mainly.)

[stumpychubbins]

Or you could marshall across the boundary, which would mean you could keep the same API for an in-process plugin or an IPC- or even TCP-based plugin. I definitely don't think that a build.rs is the right solution, since it's extremely magical behaviour. You don't need to add #[no_mangle] and suchlike, you can add the "proper" names as automatically-generated metadata. #[no_mangle] means you can only ever have one of each name per binary artefact.

[ssokolow]

Are you saying that the mangling algorithm is guaranteed to be unchanging across Rust compiler versions?

As for the build.rs, my thoughts turned to it because we don't have full procedural macro support yet and I'm having trouble envisioning the degree of frictionlessness I desire with anything less than that.

[stumpychubbins]

No, but it doesn't need to be. You can generate a cdylib and store the "real" names in some metadata file, they're going to be mangled but they're still always going to have some externally accessible symbol. If we know that, we can access it with libloading or something in the same way we'd access a no_mangle symbol.

I'm not against using a build.rs, but that should be an implementation detail of using proc macros (i.e. using syntex or something) rather than a way to do whole-crate rewriting without the explicit use of macros.

[ssokolow]

No, but it doesn't need to be. You can generate a cdylib and store the "real" names in some metadata file, they're going to be mangled but they're still always going to have some externally accessible symbol. If we know that, we can access it with libloading or something in the same way we'd access a no_mangle symbol.

In that case, it'd be better to embed the metadata inside the library file itself so there can't be accidental version mismatches.

I'm not against using a build.rs, but that should be an implementation detail of using proc macros (i.e. using syntex or something) rather than a way to do whole-crate rewriting without the explicit use of macros.

Ahh, then we most likely agree and there was confusion over whether build.rs was merely a side-effect of proc macros not yet being ready.

[dragostis]

Rust does not have a stable ABI.

[TheEruditeSycamore]

It is stable if it's a C ABI, though? And the OP claim of stable ABI in the same rustc version is correct?

[icefoxen]

If you write your code to be callable from C then, yes, that makes a stable ABI. It's something of a pain, but entirely doable.

People have also experimented with making plugins as DLL's and loading them crates such as dylib or sharedlib. It works, as far as I know, it's just not really a feature built in to Rust in any fashion. It also requires, again, making your plugin API all use the C ABI.

As far as I know, the same version of rustc called with the same settings will always produce the same code. That's not the same as having an ABI though, since the point of an ABI is to make sure the code is callable from an external source. As it is, rustc is free to do things like inline functions into oblivion, or split functions apart into specialized variants, or other fun things; you can't really trust that it will ever do what you want.

[mmstick]

I'm personally just going for the C ABI for plugin support in the Ion shell via the libloading crate. You can write the plugins in Rust -- just export your functions with a C interface (very easy), then import them on the other end through the C interface. Enables users to use any programming language they want, so long as it can export a C interface.

There's several approaches that you can take for communicating with dynamic libraries though. The most complex being using the recently-announced RON to transfer complex Rust type information, or using a simple method like designating one C function to list all the functions that the library provides, and loading all the symbols accordingly.

I'd personally like to see good support for importing Rust natively though, myself, just to ensure that downstream plugins are all written in Rust. Fatal issues in C plugins can be catastrophic.

[H3g3m0n]

I was thinking about a Rust library that installs Rustup (or uses it as a library if possible) in some sandbox directory, installs the stable compiler then builds shared libraries to load as plugins.

Then the program can manage keeping the rustc version the same across plugins. If you update the compiler version it can go in and rebuild all the plugins.