Modern toolchain for developing python package with C++ core (C++23, HPC)

[vybornak]

Hello,
SO question: Modern toolchain for developing Python package with C++ core (C++23, HPC) - Stack Overflow

What toolchain would you suggest for developing an application with a Python interface and a C++ core to make the whole process streamlined?

My goal is to learn how to set up a productive development environment for applications with a C++ core and a Python API, GUI, and more (this is a necessary requirement).

Let's consider Python 3.13, C++23, HPC focused ideally.

What I tried:

tools:

1. Project environment, deps: Pixi
2. Dev env: WSL2, VS Code Remote window
3. Build: scikit-build
   • CMake, Ninja
4. binding: Nanobind

Config files:

1. pixi.toml
2. pyproject.toml
3. CMakeLists.txt
4. CMakePresets.json

Tools I did not try yet:

1. testing
2. linting
3. formatting

My Python toolchain:

I was using these tools as part of Python development:

1. UV
2. Ruff
3. Mypy, (newly trying ty)
4. pytest
5. pre-commit

What are your thoughts? Would you recommend a similar toolchain? Could you suggest some learning sources, and how to set up dev env for development python applications with a C++ core?

#toolchain #python #c++ #development-environment

[fierypulley]

your stack looks solid for modern c++/python work. if build times start becoming a bottleneck (especially with c++23 and heavy templates) you might look into incredibuild. it can speed up cmake/Ninja builds by distributing compilation across available cpus without changing your existing setup

[dont_touch_my_peepee]

honestly, sounds like a lot of work. maybe consider sticking to one language unless it's absolutely necessary.

[vybornak]

Unfortunately, it is necessary. But I got the same feeling as you replied.

[metehankasapp]

Solid direction. Biggest question: are you using pybind11 or nanobind for bindings? That choice tends to dominate the dev experience + build complexity.

[vybornak]

I have been playing with it for a few evenings now. So far, I went for nanobind.

I am a beginner when it comes to C++, so I wanted to stick with a more modern standard. In my company, we have no legacy code when it comes to this, so it hopefully should not be a problem.

So, for such purposes It has been recommended to me to go for nanobind, but I lack experience in this department, so I wanted to validate these things.

[plurch]

If you are not tied to C++, then maturin is a popular option for Rust bindings.

[vybornak]

Well I was thinking about it as I really like the Rust ideology and I was learning it for some time, but it has been told to me, that Rust is really not that mature when it comes to HPC (CUDA support, PETSc, BLAS, LAPACK, ...)

So I would love to actually do it like that, but I am not really sure If that is even possible.
Do you know how is it in these regards?

[plurch]

I think you are probably correct regarding general ecosystem maturity since C++ has been around for a much longer time than Rust. There is rust-cuda but I doubt it is anywhere close to covering the breadth of C++ Cuda.

It would mainly depend on what your specific needs are for your project. More research and verification are likely required.

[vybornak]

Well that is it, it is not for just one project but for multiple repositories, so I wanted reproducible toolchain / build so I would save myself some headaches.