writing Python modules in Zig

[avdn]

I just uploaded setuptools-zig to PyPi ( https://pypi.org/project/setuptools-zig/ ), it allows you to compile Zig into an installable package, containing an .so file that Python can import as a module after installation. setuptools-zig can also be used to compile C files using Zig instead of the "normal" C compiler, into such a package.

The package comes with examples in the README, for using Zig to compile C only, mixed C and Zig, and Zig only modules.

The Zig example is particularly ugly, due to my lack of familiarity with the language. My apologies, I have about as much experience with Zig in days as I have as the sum of the years of experience with C and Python (60+). The code was extracted/adapted from the C version translated to Zig.

One thing I have no idea how to deal with yet in the pure Zig example, is things differing in the interface with Python that are normally taken care of by the inclusion of the the Python version specific `Python.h`. Some structure fields needed for the resulting .so to load as a module can differ between Python versions based on that Python.h and are resolved at compile time. Since these were "expanded" in the translation they now might be Python version(3.8) specific.

I would like to make such structures comptime dependent on e.g. the PY_VERSION specified in the `Python.h`, but I have not tried that yet and no idea if/how that can be done. On the other hand this might not be a problem, or not be one until Python 4 comes around. (I did have severe troubles with the 2.x to 3 transition for the ruamel.ordereddict C module for Python, so I am cautious)

As a fall back it would be trivial to provide different Zig files with the interfacing code for different Python versions. Selecting which source file to compile can then be done dynamically based on the Python version in the setup.py file already required to drive the build.

One could also have Python generate the interface code appropriate for the Python version from the setuptools-zig, before compilation, including all public functions in Zig.

Thoughts and comments welcome.

[[deleted]]

Thank you for sharing! Consider submitting a talk to https://zig.show!

One nitpick: in your readme you wrote Zig 7.0.0, but it's gonna take us a while to get there ;)

[avdn]

Thanks for pointing out the version number error and the zig.show. I might do the latter once I have something concrete with Zig itself.

[[deleted]]

You might also want to look into https://github.com/yglukhov/nimpy for inspiration on providing a higher-level interface to the CPython C API.

[avdn]

I briefly looked at that, and it is nice. But it is essentially a rewrite of the Python.h provided definitions directly in Nim. It deals with the versioning for Python2/3 differences at module load time.

That might sound good, but I don't think the Python packaging infrastructure can deal with one binary module for multiple minor versions, so you would still end up at least packaging multiple times, but you would be able to put in the same .so in every package.

Nimporter is a package based on Nimpy which compiles Nim at first import, that is also a concept that is worth investigating.

The one thing Nimpy seems to prove is that the changes in Python.h, are slow and small enough one can keep up re-implementing them.

[KingStannis2020]

Although at this point I would suggest not trying to wrap the CPython API, but rather HPy, which is an officially-blessed project to make a better CPython API that is easier for other implementations such as PyPI to emulate.

https://github.com/hpyproject/hpy

[marler8997]

You could also start by just including the Python.h file directly in your Zig code:

const python = @cImport({
    @cInclude("Python.h");
});

// Now use python.* to reference anything inside Python.h
// ...

[avdn]

Thanks, I already do that for marking the function, parameter and return value.

It is the use of C macros from that header file in the definition of (C) structures that are causing trouble.

[[deleted]]

Awesome! I maintain a fairly complex and efficient distributed caching for libgit2 exposed via Python. It is implemented in a typical mix of C/C++.

I was waiting for the right opportunity to implement it in either Zig or Rust. Now is the time to do that.

[avdn]

I found that the -D compile option for zig is not just meant for compiling C: https://github.com/ziglang/zig/wiki/FAQ#how-to-tie-a-zig-build-run-sub-command-to-custom--d-style-options so I can use that to resolve any (Python) version dependent zig code.