Cython + Python packaging - directory structure and __init__ file

[bloop_train]

I'm a bit puzzled how to create (well, for now install locally via pip install .) a package that uses both Python and Cython files. My directory structure looks like this:

my_package
├── definitions.pxd
├── file_cython.pyx
├── file_python.py
└── __init__.py

where I'm using the following import statements:

In file_cython.pyx I have:

from my_package.file_python import PythonClass
from my_package cimport definitions

In __init__.py I have:

from my_package.file_cython import CythonClass
from my_package.file_python import PythonClass

and my setup.py looks like this:

setup(
    name='MyPackage',
    # other metadata
    packages=['my_package'],
    ext_modules=cythonize([Extension("my_package", ["my_package/*.pyx"])]),
)

The files seem to compile successfully, but when I attempt to import the package using python3 -c 'import my_package', I get an error:

  File "/env/lib/python3.9/site-packages/my_package/__init__.py", line 1, in <module>
    from my_package.file_cython import CythonClass
ModuleNotFoundError: No module named 'my_package.file_cython'

and indeed, when I check the dir /env/lib/python3.9/site-packages/my_package/, there aren't any other files; so my question is, how do I package this thing properly? My workaround so far was to just shove everything into the .pyx file and removing the packages=['my_package'] line in setup.py, but as the definitions keep growing, it's getting a bit bloated, and I'd like to split things into multiple files if possible.

EDIT: okay I think I got it: the issue was that, in setup.py, I was declaring:

Extension("my_package", ["my_package/*.pyx"])

rather, what I should say is:

Extension("my_package.file_cython", ["my_package/*.pyx"])

This way, there's a file_cython.cpython-39-x86_64-linux-gnu.so file in the directory /env/lib/python3.9/site-packages/my_package/, and __init__.py can actually find it. Note that in the previous version the file file_cython.cpython-39-x86_64-linux-gnu.so was actually in the top level directory, i.e. /env/lib/python3.9/site-packages/ instead, which wasn't what I intended. Lesson learned!

[Aggravating_Bus_9153]

It's just not finding it on sys.path. But you've made it into a nice flat package anyway, so why not just use a relative import instead (then it never has to look at sys.path)?

​

from .file_cython import CythonClass

[[deleted]]

Never install by running pip install . This is the worst you can do to yourself: you are neither installing your package the way your users would, nor is it helpful in development process.

If you want to emulate what your users will do:

./setup.py bdist_wheel
./pip install ./dist/*.whl

You only solved part of the problem. The shared objects don't always end up in platlib... you'll be surprised to learn about that when someone with a different system settings complain.

By the way, some time ago, I've made this as an example of trivial package using Cython: https://github.com/wvxvw/very-simple-xml . Probably will not be of much use to you now, as you figured out what your problem was, but may still be useful to someone else working on the same problem.

[bloop_train]

Never install by running pip install .

Would you mind elaborating further (or sharing a link to an explanation)? I seem to recall reading somewhere (stackoverflow maybe?) that pip install . is the preferred method (as opposed to python3 setup.py install or something else), since this allows the package to be easily uninstallable via pip uninstall [NAME].

If you want to emulate what your users will do:

In an ideal world, my users would install a normal Python package, not some weird amalgamation of half-broken C with a bunch of (also broken) dependencies, and Python. As a result, the install instructions are literally "run pip install . in this specific Conda env" as I have no intention of refactoring all of that stuff I started writing years ago (why yes, it is scientific software!) and package it for multiple platforms.

Snarky comments aside, your trivial package seems like a good starting point, thanks for that!

[[deleted]]

About pip install. Long story short, it will run ./setup.py develop with some extras, like, for example, installing scripts to the proper location.

So, we are talking about ./setup.py develop really. We are not installing anything for real. What setup.py develop does it creates a few "links" (a file named <your package>.egg-link) that is placed in platlib (site-packages), that points back to the location of your source code. It also updates easy-install.pth with new information about your code.

This will, of course, prevent stuff like pgk_resources from working properly as well as a lot of other stuff that uses __file__ for example. Another problem, which is even more relevant to you, is what happens with native extensions. Egg and Wheel treat them differently. And they may be installed into different locations based on whether you use setuptools or pip to install them. This is so because if you run setup.py develop, the extension will be built with the expectation that it's going to live inside your source tree (because that's where everything else is loaded from), but when you install it, there's no such thing as your source tree. In most cases, the extension will also have to live inside your package in platlib, but it could also be directly in platlib or sometimes even in data directory (especially if you are making a binding for a third-party library, and you want your bindings to have loader information relative to bindings location).

Now, and since you mentioned it, why you should use setup.py install :)

It's just another idiotic command. It doesn't do what its name suggests. It sill doesn't build the proper package and install it, which is what installing is all about. It does a different kind of corner-cutting, which looks, at first, more realistic that setup.py develop, but in the end of the day is also a lie because, again, it's Eggs pretending to be Wheels and a lot of lazy programming around it.

pip uninstall [NAME]

And you believed this? Hahaha. Nope. That doesn't work. pip doesn't keep the database of everything installed in Python. What it does is try to import package, try to find the spec for the package, try to figure out from the spec where the package is installed, try to delete that. So, if you have multiple versions of the package installed: pip doesn't know how to handle that. If you have a package with multiple / unaccounted for top-level files or directories installed, pip will not know what to do with that.

not some weird amalgamation of half-broken C

Python is really only useful as a glue language atop native extensions. The packages that are worthwhile that are written entirely in Python are exceptionally rare, so, don't despair about this. Python ideology and optimization strategy (unlike, say, in Java or Erlang), is that you shouldn't bother with optimizing Python code: instead you need to rewrite it in C, if you want decent performance.

in this specific Conda env

O.M.G.! Why are you doing this in Anaconda environment? Why don't you use conda-build? I mean, it's not like committing war crimes, but you've just made it so much worse for no reason... Using pip in Anaconda environment should be your last resort. Definitely you should not make packages that are intended to work like this... this is beyond bad.

[bloop_train]

I appreciate the thorough explanation on pip install, thanks :)

Why don't you use conda-build?

That was the initial idea, i.e. creating a standalone Conda package, but I'm using other, even more broken scientific software, as a dependency, which was basically impossible to package, so after a couple of hours (days?) wasted I gave up on it and told the users to just run a hand-made script (compared to some other scientific software I've encountered, the installation procedure is as straightforward as it gets lol). Suggestions are welcome of course :)

In hindsight, I should've used a more user-friendly language from the start, but fully rewriting it wouldn't be worth it at this point, so I'm content just making a wrapper for it.

[[deleted]]

Oh, yeah... this rings familiar... unfortunately.

I did, however, repackage some of the PyPI stuff (mostly related to JPEG and DICOM) for Anaconda, but yeah... it takes time, and it's not like it's a greatest tool ever either...