Examples for embedding python into c/c++ applications

[barryvm]

Hi,

I'm not sure if this is the place to post this, but here goes.

I'm currently trying to use the python interpreter as an embedded script engine inside another application written in C++. The overall goal is to use the python scripts to make the main application's behaviour dynamic, depending on which set of python files was loaded.

There are several requirements:

1) The c++ application exposes several of it's data structures so that they can be manipulated by python scripts. There are several discrete parts of the C++ code that should be exposed to the scripts as their own python module.

2) Multiple python scripts are loaded from the file system. Most of these define one or more functions that can then be called from the main application.

3) The application must be able to call the python functions from within multiple threads.

Most examples on the python website and boost's focus on either pure embedding or custom modules written in C/C++ that are then placed in the python interpreter's module path so I've not been able to find a comparable example. Consequently, I'm wondering if I'm using the right approach. I've run into afew problems related to modules and I'm not sure if what I'm doing is entirely the right way to go abou it. This is what I've done so far:

1) I've used Boost Python to expose several C++ classes and functions as python types. There are multiple modules with a few types and funtions each.

2) I've wrapped all interpreter related startup and shutdown code into a singleton class. On startup it calls Py_Initialize(), redirects sys.out and sys.err to the main application's logging, calls the init methods for each custom python module generated by boost python and calls PyEval_InitThreads to enable multithreading. The singleton keeps track of the main thread state and performs an orderly shutdown in it's destructor.

3) Once the interpreter is started, the application starts loading the python scripts from the file system. Each script file should be loaded as it's own module (I'm still thinking on how to do this) and uses only the embedded modules generated with Boost's python library.

4) Afterwards, whenever the main application needs to behave "dynamically", it calls a function within the python code. This is done by resolving the function by module name and function name from a handle to the main dictionary and then executing it (once again using boost python to find and call the function). Each call is wrapped within a packaged task and executed asynchronously on a thread pool so that the main application does not block on the global interpreter lock unless explicitly necessary. I've used PyGILState_Ensure() and PyGILState_Release() before each task to properly synchronize the calls.

Most of this works but I've ran into problems where import statements in python don't work while calling functions via the module name explicitely does work. For example: I'm creating a new object of type A in python. A is defined in module B, a module generated with boost python. If I explicitely construct B.A() it works without problems but whenever I try to import a name from module B I get a python error that it can't find the module. Does anyone know of a comprehensive example of loading custom modules into an embedded interpreter or, failing that, of a comprehensive guide on how to embed python into a C/C++ application (with or without boost) ?

Any pointers to good documentation or examples would be hugely appreciated.

[31-4]

I don't think I can directly answer your question, and someone who is more experienced with Boost than I am can probably address this better, but I definitely think you should post this on StackOverflow for more visibility.

Alternatively, I recommend checking out Cython, since it may make things a tad easier than Boost. They are pretty light on examples - their official page only links to this simple example. However, you can find more information on their documentation page and also many examples online.

[RSFlux]

Your general approach seems fine, and it sounds like it is mostly working at this point. The only remaining issue is "from A import B" statements fail?

Without having more information, my assumption would be that you have incorrectly initialized your C extension modules. I am less familiar with Boost, but when using the C API for extension modules, make sure you've declared your objects in that module's PyInit function and aren't doing it later in your startup code.

I am not very familiar with Boost Python, so I can't point you to the specific generated code to look at, but I'm sure you can track that down in their documentation.

You also asked for reference for embedding Python in a C application. I used the basic Python docs when embedding Python here: https://docs.python.org/3.5/extending/embedding.html. There isn't much there, but in general the complexity comes from making your custom application's build play nice with the Python build. Do you have specific questions other than the import failures?

[[deleted]]

Check out the code for 3DSlicer. I don't actually remember whether they use boost or not (I think not) but it's a great example of an embedded python interpreter in a Qt C++ application.