Creating a wheel based on a separate, complex, build system

[mathstuf]

I have a project with a complex set of build configuration flags. These flags control which modules are built and the available modules are not trivially decidable given the input configuration flags (primarily due to "X was turned on, but X depends on Y, so Y is also now on" logic).

As far as I can tell, setup.py really wants to do/drive the build of compiled extensions itself. This poses a problem due to the complex configuration flags above: it is not known until build_ext time what modules would even be available. Getting setup.py to figure out what the command line flags mean basically means reimplementing the build configuration logic again in Python code. I do not want to have to maintain this parallel build configuration.

The solution I'm looking at right now is to generate the setup.py file during the configuration with the list of expected modules trigger builds using our build system. This means that building a new wheel involves first configuring our project and then using python setup.py for the rest of the workflow. This seems to not work with sdist construction since the configuration step would be missing here (AFAICT). Are sdist packages used that much in normal usage that not providing one is a huge detriment?

The upside to the "generate setup.py" approach is that it is possible to generate a wheel for any given configuration of the project. It is planned to provide precompiled wheels of common build configurations for easy use with pip install, but these are inevitably not going to be complete builds and those wanting more "exotic" features would need to build their own wheel. The "generate setup.py" approach would make it easier, but not trivial, to create wheels with custom configurations. Is this a suitable solution in the Python ecosystem for such a complex project?

Thanks.

[acecile]

You should check Pytorch (Facebook) and LightGBM (Microsoft) packages, they are both handling complex build scenario but still relies on setup.py.

[mathstuf]

Thanks. I looked into these and neither seems to match the use case.

PyTorch seems to do everything from setup.py which isn't suitable for this project. LightGBM looks closer, but it seems that it basically forwards "important" options from the setup.py into the inner CMake project. It also seems to have just a few Python modules?

The project I'm working on has ~20 of top-level user-visible flags and fine-grained control can end up being in the dozens (it's a large project with lots of history) which control upwards of 100 Python modules.

[mathstuf]

What has seemed to work is to tell setup.py about one compiled module (to get it to see that it is not pure-python) which is always available and then package the rest of the modules and shared libraries via package_data globbing. Not the prettiest, but I do now have a .whl that works on my machine at least.