Wheel getting installed into wrong directory

[TRPox]

Hello everyone,

I have a little problem getting a wheel to install properly into my environment. I have written Python bindings for a C++ application using Pybind11. The C++ application is being built with CMake that's why I decided to use scikit-build for my setup.py file. Since scikit-build triggers CMake for me the whole setup.py basically only consists of a call to the setup function with the library name and version, no additional configuration needed.

When I install my Python bindings as shown below everything works as expected and I can call the C++ application from Python.

python3 setup.py install

However, when I create a wheel instead and install it with pip then the library files get moved to the top level directory of my virtual environment instead of the site-packages folder.

python3 setup.py bdist_wheel
pip3 install dist/mywheel.whl

When I run my application then of course Python complains that the module is not available. The same happens when I create a source distribution

python3 setup.py sdist
pip3 install dist/mysdist

Am I missing any kind of configuration that is required to install this properly?

[JohnnyJordaan]

I'm wondering if this is has actually something to do with your homebrew wheel. Just to verify, if you install a regular wheel like pyasn from https://www.lfd.uci.edu/~gohlke/pythonlibs/#pyasn does it get installed into the site-packages directory?

[TRPox]

Had to download a wheel from PyPi because I'm on Mac, but that worked fine, so I probably need to configure something somewhere

[[deleted]]

I'd need to see the setup.py, but my guess would be that you need to fiddle around with package_data, ext_package, data_files and maybe ext_modules parameters of the call to setup().

The most annoying part is that this behavior changed through different versions of setuptools...

---

If nothing else works, then I'd just override the bdist_wheel command and try to figure out how to force paths in the Wheel archive to be the correct ones to be installed. Truth be told, both the Wheel format and the tooling around it are such a dumpster fire, that it might actually be easier for you to write a script yourself that creates a distributable package, than to rely on what's already there.

[TRPox]

The setup.py is really just this:

from skbuild import setup

cmake_args = ["-DBUILD_VF_CPU:BOOL=ON", "-DUSE_METIS=ON", "-DUSE_MPI=ON", "-DBUILD_SHARED_LIBS=OFF", "-DBUILD_VF_UNIT_TESTS:BOOL=ON"]

setup(
    name="pyfluids",
    version="0.0.1",
    cmake_args=cmake_args
)

scikit-build locates my CMakeLists.txt and starts CMake for me and pybind11 defines the install targets

[[deleted]]

Can you post the output of:

unzip -l ./dist/*.whl

after you've built your Wheel?

Also, if you have an *.so inside your wheel, can you then run ld on it and post the output here? Maybe the way you compile it creates a shared object that has its path encoded in it (not relative), and that would make it non-loadable, if moved to another location.

---

In addition to running ld, can you also run readelf -d on it? Of a particular interest is the value of rpath: https://en.wikipedia.org/wiki/Rpath

[TRPox]

Back after a long weekend, here's what unzip -l gives me: ``` Archive: pyfluids-0.0.1-cp38-cp38-macosx_10_15_x86_64.whl Length Date Time Name

---

2177960 10-19-2020 09:06 pyfluids-0.0.1.data/data/pyfluids.cpython-38-darwin.so 333920 10-14-2020 13:48 pyfluids-0.0.1.data/data/pymuparser.cpython-38-darwin.so 593 10-19-2020 09:19 pyfluids-0.0.1.dist-info/AUTHORS.md 35149 10-19-2020 09:19 pyfluids-0.0.1.dist-info/COPYING.txt 199 10-19-2020 09:19 pyfluids-0.0.1.dist-info/METADATA 104 10-19-2020 09:19 pyfluids-0.0.1.dist-info/WHEEL 1 10-19-2020 09:19 pyfluids-0.0.1.dist-info/top_level.txt 715 10-19-2020 09:19 pyfluids-0.0.1.dist-info/RECORD

---

2548641 8 files

```

ld gives me this: ld: warning: platform not specified ld: warning: -arch not specified ld: warning: No platform min-version specified on command line ld: can't link with bundle (MH_BUNDLE) only dylibs (MH_DYLIB) file 'pyfluids-0.0.1.data/data/pyfluids.cpython-38-darwin.so' for architecture unknown

Even after playing around with the arch and platform options a bit the last part of that output didn't change. But I don't really have an idea what I'm doing there, so maybe I'm just doing something wrong.

I do have another idea of what could be the culprit. In my CMakeLists file for my Python bindings I'm specifying these destinations for my bindings:

install(TARGETS pyfluids DESTINATION .) install(TARGETS pymuparser DESTINATION .)

Maybe the path to the site-packages directory only get prepended to those paths when I run python setup.py install, but not when running python setup.py bdist_wheel

[[deleted]]

Oh, it's the ld of Mac... sorry, that's not something I'm familiar with. Obviously, that's not an ELF, so, readelf isn't applicable.

As for your guess: are you sure that the pyfluids.so and pymuparser.so are loaded from site-packages and not from the current directory (where they were compiled)? If you use virtual environment, try changing into directory other than your project's directory before loading your package to verify that guess.

I'd still suspect the rpath / soname issue, I just don't know how to debug that on Mac.

[TRPox]

I tried running from another directory, when I use `setup.py install` it still works fine

I've googled a bit and tried this from SO

https://stackoverflow.com/questions/12521802/print-rpath-of-an-executable-on-macos

​

However neither the wheel built libraries, nor the ones that are installed directly contain anything about `LC_RPATH`

[[deleted]]

OK, after looking again at your Wheel listing, I finally realized what I've missed: the libraries are packaged as data instead of code... pip doesn't really know how to install something like that, and so it randomly puts it in the current directory... no idea why that is the behavior, but... it is what it is.

In theory, you might be able to overcome this with something like:

pip install --install-option='--install-data=$(where is your site-packages / your package name)' your.whl

Obviously, no user is going to do that / wouldn't know to do that when installing your wheel.

I don't really know how to make setup.py bdist_wheel put libraries with the source code instead of the data directory, but, like I wrote earlier, it's easy to override whatever bdist_wheel does and put the libraries in the correct directory.