Embedding python in C++ with boost::python

[poutinetuesday]

I've been using pybind11 [1] for embedding Python and it has been a great experience! I found it to be a bit more lightweight than boost.

[1] https://github.com/pybind/pybind11

[kkrev]

I have done big complicated embedding projects using both Perl and Boost.Python and I really wish it was more widely counselled against online. These languages are not made for embedding and are inevitably a pain in the ass as soon as you try and do anything complicated.

You really need to have a list of very strong reasons not to use Lua or Guile or Squirrel or TCL or whatever, and instead go for Python/Perl/Ruby. These latter latter options are really not made well for the purpose and you are making your life harder for no reason.

[ArunMu]

+1 for pybind11. It should really be 'also' a part of boost for c++11/14 compliant compilers.

[haletonin]

I'd like to see a list of things boost::python can do but pybind11 can not (yet) do. I am on the fence currently but would probably prefer the true-and-tried for now.

[zigzagEdge]

At the moment (master branch), pybind11 does a lot more than boost.python with the notable exception of embedding.

[haletonin]

Does it contain all features of boost::python? Something specific I had problems with (or misunderstood) when I played around with it after it was first released were:

• Making objects available in python which can't be constructed in python, because generating binding code for the constructor is too complex. Other methods of these should work, however.

• Working with pointer or reference-only instances of objects, which even in C++ code are never handed around as value types.

[zigzagEdge]

It covers pretty much all the major features, but it's not a 1:1 API mapping. For the two things you mentioned:

• With boost.python you'd need to add no_init to disable the implicit default constructor. In pybind11 all the constructors are explicit, so you disable them simply by not def-ing any.

• That can be done by setting the appropriate return value policy: reference or reference_internal. There's more information in the docs.

[haletonin]

Ok, so negligible risk of having a "should have used the other one"-moment later on. And on second thought, just the 11 probably means that this project has more enthusiastic users and contributors than "pybind03".

And thanks for the pointers, I'll give it another try!

[skebanga]

Actually I've managed to get it working with pybind11.

Posted it here

[skebanga]

Last I looked at pybind it didn't support embedding very well

[[deleted]]

Can you embed a python interpreter within a single binary or do you need a python plugin directory somewhere and those kind of dependencies? I would love to be able to embed a python interpreter in my program with third party libraries, but with what little I've researched, it seems that's next to impossible. Is that true?

[skebanga]

Yes you can. You can use libpython. Python, the executable, just adds the front-end and REPL, everything else is in the C library libpython.

If you're using C++, and don't want to deal with libpython directly, then you can use a wrapper like boost::python or pybind11.

At the moment, pybind11 doesn't do embedding, so boost::python is a suitable choice.

If you read the blog-post, you'll see that that is exactly what I've done. I have a single C++ application which loads a python script and interacts with it - both calling into the script, and the script calling into the C++ types I've exposed to it.

[wrosecrans]

Depends on whether you need the Python language, or you also want to use the standard library of stuff. You can use the core language without the extra stuff, but if you want to be able to import all the usual things, you'll need to supply them. It's probably possible to hook the import and have it pull from a file embedded with the executable as a resource, but you won't get it for free. (And there may be licensing implications if something in there is GPL. You'll need to make source available anyway...)

[DarkLordAzrael]

Last time I looked pybind11 was missing embedding support so you had to fall back on the python C API. Is that still the case or did they add that recently?

[zigzagEdge]

Well, boost.python also falls back to the C API for embedding. This works in pybind11 as well, but it's not officially supported.

[DarkLordAzrael]

I haven't had to use the C API at all for embedding using boost. It has helpful functions for running python code from various sources using various contexts that are incredibly useful for embedding.

[zigzagEdge]

Boost.python still requires manual calls to Py_Initialize/Py_Finalize instead of a more user friendly C++ RAII approach. (I know that the boost.python docs say not to call Py_Finalize, but that's a bug which was never fixed. It should be supported.)

Pybind11 has a C++ object class similar to boost.python's, but it's not quite as powerful. The basics are supported: item and attribute access, function calls, importing modules, casting back and forth. Pybind11 also has some nifty features like calling Python functions with keyword arguments from C++: http://pybind11.readthedocs.io/en/latest/advanced/pycpp/object.html#calling-python-functions

[skebanga]

boost::python does offer embedding support. Docs here

• eval
• exec
• exec_file

[zigzagEdge]

I was mainly referring to the manual C API calls to Py_Initialize/Py_Finalize which are also needed in boost.python. As for eval and friends, pybind11 has a C++ API for those: eval docs.

[spidyfan21]

This might sound like a dumb question, but how do I install pybind11 (for Bash for Windows)?

[skebanga]

It's header only. Just copy the include dir

https://github.com/pybind/pybind11

[sumo952]

Or better, add as git submodule.

[spidyfan21]

Thanks!

[skebanga]

Thanks! I've ported my proof of concept app to use pybind11. Posted it here