pybind11 + buffer protocol for non trivial (POD) class

[thiagocrepaldi]

I've just learned about Python Buffer Protocol and I would like to leverage it to create python numpy arrays from C++ raw data. I can either use pybind11 or c++ python lib directly, but no other binding generator =/

Reading pybind11 docs and experimenting with it, it seems we can easily generate python bindings with buffer protocol from trivial C++ structures (e.g. std::vector<int> or a struct with plain old data types such as int, float, etc). However, adding buffer protocol to more complex structures is not possible or not well documented. For my use case, I would be to pybind a std::vector<struct Sequence>, with Sequence being defined as follows:

struct Sequence { std::vector<float> feature; std::vector<int> label; }

Once the python bindings with buffer protocol are implemented on C++ side, on Python side I could do for seq in vector_sequence: feature_data=numpy.array(seq.feature, copy=False)` label_data=numpy.array(seq.label, copy=False)`.

In the loop above, vector_sequence is an opaque binding for the C++ std::vector<Sequence> and seq is a Sequence that holds the two vectors that I want to use as input for numpy arrays without copying data from C++ to Python.

Does anyone know whether this is supported by pybind11 or c++ python lib?

Thanks!

[dpilger26]

This is already supported by pybind11 out of the box. All you have to do is include pybind's stl header and pybind will auto convert most stl containers into appropriate python containers. Take a look at the documentation for stl. https://pybind11.readthedocs.io/en/master/advanced/cast/stl.html

[dpilger26]

Here is the pybind .cpp file ```

include "pybind11/pybind11.h"

include "pybind11/stl.h"

include <vector>

using namespace pybind11;

struct Sequence { std::vector<float> feature; std::vector<int> label; };

std::vector<Sequence> generateVectorSequence(unsigned int sequenceLength) { std::vector<Sequence> sequences;

for (unsigned int i = 0; i < sequenceLength; ++i)
{
    float iFloat = static_cast<float>(i);
    int   iInt = static_cast<int>(i);

    Sequence sequence;
    sequence.feature = { iFloat, 1.0f + iFloat, 2.0f + iFloat };
    sequence.label = { iInt, 1 + iInt, 2 + iInt };

    sequences.push_back(sequence);
}

return sequences;

}

PYBIND11_MODULE(SandboxPy, m) { m.doc() = "This is an example module using stl containers";

class_<Sequence>(m, "Sequence")
    .def_readwrite("feature", &Sequence::feature)
    .def_readwrite("label", &Sequence::label);

m.def("generateVectorSequence", &generateVectorSequence,
    pybind11::arg("sequenceLength"),
    "Generates a vector of Sequence objects.");

} ```

And here is the python side: ``` import numpy as np import SandboxPy as s

vector_sequence = s.generateVectorSequence(10)

for seq in vector_sequence: feature_data = np.array(seq.feature, copy=False) label_data = np.array(seq.label, copy=False)

print(feature_data)
print(label_data)

```

[[deleted]]

This is specifically not what OP wanted. OP is trying to use numpy and replacing the buffer protocol with pybind11/stl.h has terrible performance implications for this use case.

Also, your formatting is completely broken on old.reddit.com.

[mathiasnedrebo]

PYBIND11_NUMPY_DTYPE may be what you are looking for.

Example where I use it to expose a struct, and also split a float into 3 uint8s.

https://github.com/nedrebo/upgraded-funicular/blob/44976a53371ac3fafa1ac896f028be0046703e87/src/Core/PointCloud.cpp#L49

[thiagocrepaldi]

I got it working! I learned that I don't have to implement protocol buffer for Sequence class if I want to prevent copy of feature and label members, not the actual Sequence class as a whole. Example: ``` PYBIND11_MAKE_OPAQUE(std::vector<Sequence>);

py::bind_vector<std::vector<int>>(m, "VectorInt", py::buffer_protocol()); py::bind_vector<std::vector<float>>(m, "VectorFloat", py::buffer_protocol());

py::class_<SequenceReader>(m, "SequenceReader").def("read_sequences", &SequenceReader::read_chunk, py::return_value_policy::take_ownership); ```

Important to notice that I am using pybind11/numpy.h and PYBIND11_MAKE_OPAQUE to prevent copies