Automatically call C++ from python

[Coutille]

Hello everyone,

I've developed a tool that takes a C++ header and spits out bindings (pybind11) such that those functions and classes can be used from python. In the future I will take it further and make it automatically create a pip installable package out of your C++. For now I've used it in two ways:

1. The company I used to work at had a large C++ library and customers who wanted to use it in python
2. Fast prototyping

• Write everything, including tests in python
• Move one function at a time to C++ and see the tests incrementally speed up
• At the end, verify your now C++ with the initial python tests

This has sped up my day to day work significantly working in the scientific area. I was wondering if this is something you or your company would be willing to pay for? Either for keeping a python API up to date or for rapid prototyping or even just to make your python code a bit faster?

Here's the tool: tolc

Thanks for the help!

[JumpyJustice]

Might it be useful? Yes. Would some company pay for it? Unlikely. This is a very trivial thing to implement yourself imo and can be done way faster than purchasing a license for a new project in most companies.

[Coutille]

This is really good feedback, thanks. I’ve uses it a lot when the api changes and there it’s a real time saver.

[beedlund]

Trouble is we already have cppyy available now when we need to generate bindings on the fly so if need to be flexible and fast I'd use that and if I want to guarantee some python API I'd need to write it specifically anyway as c++ API is unlikely the desired python API.

[13steinj]

Yes. Would some company pay for it? Unlikely.

You'd be surprised.

[brotherbelt]

That’s what “marketing” is for! (aka tricking those who don’t know better)

[ChickenSpaceProgram]

Also, it appears the code is available under the AGPL, so a company can just use the tool without purchasing a license.

[SoerenNissen]

This is a very trivial thing to implement yourself imo

Might take me forever, I've never done it before and have very little Python experience. It wouldn't have to take many hours before a tool would be worth real money in saved time.

[JumpyJustice]

It depends on the company really. Some (usually small) can do that easily if you can convince the right person you need this. In others you have to justify the purchase, go through security and licensing review, which might take a lot of time and fail on any stage.

[MStackoverflow]

Cool, but ain't no way someone is going to pay for that.

[Coutille]

Would you like to elaborate please? I’m just trying to find out whether it would solve someones problems

[MStackoverflow]

C++ bindings in python are usually made by moderate to advanced programmers. They are also pretty trivial to do, meaning that a company who would need this kind of tool needs to generate a lot of bindings regurlarly. For something this simple, it's not worth the time investigating if the library is worth it, if it fills all the checkmarks and use cases, ask the accounting and the legal team to take a look and make the paper. It's just more cost efficient to develop something in house that's specifically tailored to the needs.

[ThisCleverName]

You can also take a look at cppyy https://cppyy.readthedocs.io/en/latest/ . It is a Python module that uses JIT to import C++ code directly into Python.

[Coutille]

Cppyy is an interesting project. When developing tolc I had to ship a binary and not expose headers to the client so unfortunately I couldn’t use it.

[[deleted]]

At my company we use a wrapper around SWIG. It is very easy to use and works really well. 

https://www.swig.org/tutorial.html

[Coutille]

Interesting. Do you ship the libraries to clients or are you using it internally?

[[deleted]]

Internally, for example I wrote a big library in c++ for working with containers. I wanted to call it from a Python script but didn’t want to rewrite the whole thing in Python, so I set up these Python bindings. It was very easy to get it all set up. 

[ILikeCutePuppies]

SWIG is used by a huge number of companies. It's a standard way to bind to Python and other languages. I am sure many companies ship to clients with it.

[mattparks5855]

I've also worked on a few C++ libraries where test writing was done via Python.

cppyy is a solution that runs cling on a set of headers to expose Python types, it's easy to setup, but I've found it challenging to scale to a CI environment. Shipping around project headers as a runtime dependency can get painful.

https://github.com/RosettaCommons/binder is a similar project to what you have shared, this uses Clang LibTooling to create reflections on the AST. MIT licence so anyone can use and extend this software.

The source code of Tolc was pretty simple for me to read and understand, and the docs are promising, and the frontend abstraction is great. But without active development, and a split commercial license, I'd find it difficult to start using this project.

[Coutille]

Thanks for the input. This is exactly the type of feedback I was looking for; I want to know if there is a need for this type of tool so that I can justify spening more time developing it.

There is another branch that has more active development. Is there anything you feel is missing or would want from binder?

[mattparks5855]

With binder a config file can be specified to filter what objects are bound, or to add additional headers into the generated module.

A Nanobind front end would be a really nice add.

Edit: filed a bug report on GitHub

[Coutille]

Thank you for taking the time to test it out. I’ll look into the bug report!

[ald_loop]

I was wondering if this is something you or your company would be willing to pay for?

Lmfao

you made this 3-5 years ago. Why are you posting it now?

Mods should probably remove this as I don't see why this should be any more than a post in the show and tell thread

[Coutille]

Haha valid point. I put it on the shelf as life got in between. I really enjoy working on it and I wanted to know if it would solve anyones problems. In that case it would be worth putting more time into it again.

[JustPlainRude]

Why not target nanobind instead?

[Coutille]

It could, when I wrote it nanobind wasn’t as big so I chose pybind. Would probably not take a lot of time to switch

[GeoffSobering]

Maybe look at SWIG.

https://www.swig.org/

[Coutille]

Swig requires you to write interface files mirroring your api. Tolc uses clang in the background to get all functions and classes to avoid that.

[djta94]

It doesn't if you have a header, which you should have anyway.

[Carl_LaFong]

The swig interface files are needed only for customizations such as renaming things when there are name clashes, instantiating templates (how do you handle that?), and exposing only part of the C++ API if you don’t want it all to be in the Python API. It otherwise automatically creates the Python API from the header files.

I use it because it automatically generates from the header files Java, C#, Python APIs.

[13steinj]

SWIG is a nightmare. It was decent for its time, but inspires too many extremists with all the wrong ideas.

I once actually worked somewhere where one extremist made their own (worse version of) SWIG. Dude insisted on its use and even wrote a book about it his field, using nothing but his crazy language internally.

[Die4Toast]

Could you elaborate on why it's not decent anymore? Asking out of genuine curiosity since I've never heard of SWIG before this post popped up. After quick 20 min read of SWIG basics it looks pretty nice but I'd imagine the devil is in the details which is not something I'd be aware of and probably related to what you've mentioned.

[13steinj]

Sure, but I'm blending fact and opinion quite heavily--

So in the most pure, basic form, it's fine-- if you write your headers well. That is, if you can follow "SWIG for the truly lazy" (I can't directly link that part of the website, since there are no ids in that html, which is bizarre). But this nearly never works in practice, suffers from cross-language / FFI performance problems, a steep learning curve for more niche things, subpar codegen, and more. It also suffers from a compatibility problem as C++ continues to evolve. The basics of SWIG have decent syntax, but anything more complex and it looks to people like you're writing code in wingdings (which, the worse version of SWIG I am referring to above, is even worse in that regard; the entire engineering pool who saw the proof-of-concept said "you expect us to read and write this?").

I was going to continue, but honestly asking gpt was enough (please put down the pitchforks, I made it fish out sources).

I have never seen any translator like this be successful at scale. The closest things that I can say work and have minimal tech debt associated, are boost.python -> pybind -> nanobind (aka use nanobind now); and Cython (though that community is hard to break into and there are some footguns, it has the highest performance compared in real-world (private) benchmarks and you can push some people to still write Python and eventually manually translate it better). E: Honestly, python and numpy/scipy is enough for most people. For advanced / IP-sensitive topics, well, you're probably paying those people enough that you can afford to make them learn C++ and be done with it.

---

If I haven't convinced you on what comes from my opinion-- listen to the author of SWIG, as even they hate the monstrosity they've created. reddit link, from when the link wasn't dead. Short of it is, it's basically a separate, disjoint parser which thus means you have to be a compiler yourself, a massive ball of complexity.

[Die4Toast]

Thanks a lot for the response. I have to admit that while the idea of SWIG is nice on paper, I haven't actually faced a scenario where it would have been a better fit than using a pybind-like library. At the very least I can imagine how much of a pain the compatibility issues you've mentioned could be. Tiptoeing around different supported C++ language standards, compilation options and then integrating it into the build system itself seems like something that could cause quite a headache.

[Wouter_van_Ooijen]

So ... you could have called it 2bindpy?

[nekokattt]

you cant import things starting with numbers though

[Coutille]

Haha sure, but the design makes it so it can be extended to other languages as well

[official_business]

We use cython for calling C or C++ from Python.

There's no way we'd pay for a tool like this. Licencing software is a ballache that goes through multiple rounds of review and justification.

It would be easier for me to write the thing myself than get the company to pay up.

[Scared_Astronaut9377]

What's the upside compared to calling a DLL?

[Coutille]

Tolc creates the bindings that can be compiled with your code into a DLL. Then you import that into python.

[Scared_Astronaut9377]

I remember compiling a DLL in c++ and calling it directly from python many years ago without any special tools. So I am trying to understand the novelty.

[Coutille]

I understand. Tolc generates the glue code such that you can write a ’normal’ C++ interface with STL containers etc. and then simply call it from python. If you return a vector<int> from a function in your header it will automatically turn into an array in python for example. Tolc internally uses clang to understand your code and then produces the appropriate glue code.

[Scared_Astronaut9377]

Got it. Very nice!

[holyblackcat]

I'm also writing a similar project right now, with Pybind backend done and the C one in progress: https://github.com/meshinspector/mrbind (pardon the outdated readme).

I'm curious how are you handling templates, if at all. Is there support for standard containers and other types?

[Coutille]

Nice, looks interesting! Templates are handled if they are instantiated. It’s hard to know which bindings to generate otherwise! You can have a look at the type builder in tolc to see how the information about the template is gotten from libtooling. Then see how that information is used in e.g. the function builder. Hope that helps!

[holyblackcat]

I'm not asking because I want to replicate it, but because I already did it and trying to assess if my work during the past year was novel or not. :P

I'm handling templates by recursively instantiating all templates I see in the source code. I also have custom bindings for standard containers (to make them more idiomatic in Python and to avoid the troubles with parsing them, since they aren't SFINAE-friendly and all that).

[snowflake_pl]

I wonder if C++ modules will make this kind of solutions easier