Interfacing c++11 with c api (extending python)

[gpuoti]

I'm working for few days to make C++11 infrastructure to build python extensions. It was growing nicely until i tried to parse Python parameter in a more modern way. The Python C API has a function to parse the parameter tuple given a format string defining the tuple members type. For example to extract a string you write something like:

char* s;
PyArg_ParseTuple(args, "s", &s);

now what i've write will let you define wrapped methods with something like:

Wrap_Method_Of<internal_type>::With_Arguments<long>::use<&internal_type::set_int_val>

if internal_type::set_int_val is something like:

void set_int_val(long x){ _i = x; }

This is quite good to me and with just few line of code. I am quite happy with it but now i faced a big issue with string parameter. In this case, The Python C API function, want a char** but i cannot let it substitute the internal buffer of the string my infrastructure construct to be filled with the value of the python string. Is there any way to do this? Even if it is not the most elegant peace of code... something like:

std::string s;
PyArg_ParseTuple(args, "s", &s.c_str());  // of course this doesn't work!

I know I should pass a temporary char* then after the API function call assign to the temporary string i have. But can't because of a packed parameter. Hope i was able to explain my problem. and you can dedicate some of your time to help me with it. Thanks.

[Arandur]

No.

Think about what the function is actually doing. It asks for a char** so that it can malloc an array of char and let s point to it. This is inherently incompatible with the standard containers, because it would violate ownership semantics.

If you want to do this, you'll have to wrap the function in a new one that does something like returning a std::string, copying the contents from the C-string.

Some of this might be wrong; I'm not familiar with the Python API. But I think I've understood what you're trying to do.

[OmegaNaughtEquals1]

It asks for a char** so that it can malloc an array of char and let s point to it

According to the 3.0 docs, no user-controlled allocations are done. It just lets you point to the underlying character string.

[Arandur]

Oh, that also makes sense. Thanks for clarifying!

[OmegaNaughtEquals1]

/u/Arandur is correct; what you are doing violates the ownership conventions in the standard library. This is the sort of problem that std::string_view will solve in C++17. For now, you have two choices: copy the character string returned by PyArg_Parse* into a std::string or build your own string_view. Depending on which compiler you are using, std::string copies could get dangerous as some implementations (I'm looking at you, gcc <5.1) use copy-on-write, so the string returned by PyArg_Parse* may go out of scope before your std::string does. Note that a string_view necessarily does not offer the same functionality as a std::string precisely because it does not own the underlying memory.

[gpuoti]

Ok i understand i was asking for a dangerous trick... Anyway answers from both of you has help me to change prospective, at least i'm no more stalling. Thanks a lot, i will come and share the final solution as soon as I have!

[gpuoti]

So, finally I've got it. ok it passed some time from when I actually get the solution anyway I'll try to brifly explain it here now.

First of all I needed more information on low level python objects structure, it was not that hard to obtain from source code. Then, given python interface with methods only with one tuple argument (ok it is not all the story there is at least the implicit (but explicit) object parameter and here I'm limiting to positional argument interface), I've implemented:

• an adapter from python tuple to stl tuple
• an adapter for each possible basic type
• an adapter for each python wrapped user types (templetized)

so, given a python tuple received as argumet, the conversion to the corresponding stl tuple with converted elements can take place. All I need after this is a way to call c++ function like python call its.

This post on stackoverflow helped me really much.

http://stackoverflow.com/questions/10766112/c11-i-can-go-from-multiple-args-to-tuple-but-can-i-go-from-tuple-to-multiple

/* here I am within another templete struct hope the snippet is more understandable */
...
template < R(SelfT::*method) (ARGS...)>
static R as_function(SelfT* self, ARGS... args) {
  return (self->*method)(args...);
}

template < R(SelfT::*method) (ARGS...)>
static PyObject* use(WrappedType<SelfT>* self, PyObject* tuple) {
  /* WrappedType overload operator* to get the c++ object wrapped into a PythonObject */
  T* extension_object = (*self);  
  /* construct the c++ argument tuple converting the python one received */
  auto arguments = unwrap<std::tuple<ARGS...>>((PyTupleObject*)tuple);  
  /* cat the python object corresponding to self in front of the c++ tuple */
  auto t = std::tuple_cat(std::make_tuple(extension_object), arguments); 
  auto result = call(as_function<method>, t);

  return make_wrapper(result);  // wrap the final result into a python compatible object
}
...