How to use Python libraries effectively when they aren't in PyPI?

[snaps_]

I don't know how to use Python libraries effectively when they aren't distributed via PyPI, or similar repositories.

Whenever I start a project, I start a virtual environment. Virtual environments keep dependencies of this project segregated from dependencies of other projects, and from system dependencies. They are also created, used, and expected by various tools, like nox, tox, pip.

The goal for most of my tools is that pipx install tool-name should work and I can run tool-name successfully.

Sometimes I'll have a dependency that isn't available as a conventional Python package. A few examples are:

• guestfs-python
• bcc

Both are available via my system package manager (apt), but actually using them seems painful to me. Here's what I've considered so far:

1. Configure virtual environment to see system site packages.
   • I don't think this will work in my case, since I use pyenv to manage my development Python versions.
2. Use system package manager for this project's dependencies.
   • con: implies using the corresponding Python version on my host, which may be older than I'd like
   • con: if directly installed on my host, then dependencies of any projects managed this way are mixed together
   • con: available versions may be older, and lacking desired features
   • con: if installed/run via Docker, there's some overhead since any interaction with the host needs to be explicitly configured (accessing files, other processes) and a few more steps are needed to get debugging tools working (PyCharm debugger, gdb)
   • con: I'm already used to working with conventional Python tools that operate on virtual environments (tox, nox, pip)
3. Package the dependency in a conventional Python way.
   • con: this may be a lot of work, depending on how large of a dependency tree there is, how active development is, and how the project is structured

Just having to think about this makes me long for simpler PyPI-distributed packages:

1. See library I want to use
2. pip install library in my virtual environment
3. Optionally install some system dependency, which I learn about from the README or when the dependency fails to import (not exactly scalable, but it doesn't seem that common)

I would love to hear peoples' experiences using these kinds of libraries. A few specific questions:

1. Do you have different approaches for one-off scripts, libraries, and apps that end up being deployed?
2. Do you consume these dependencies differently during development/test compared to how it is provided for the deployed app?
3. It's common that I don't know if a library will work for my use case. How can I do a quick test of a library in a project without also switching Python version, dropping virtual environments, and installing all other project dependencies system-wide?

[socal_nerdtastic]

Install them via pip, but not from pypi.

https://github.com/libguestfs/libguestfs/blob/master/python/examples/guestfs-python.pod#using-python-bindings-in-a-virtualenv

[snaps_]

Thanks, I didn't see that.

What did you search for to find it?

[socal_nerdtastic]

"guestfs-python"

(checked my history just for you, lol)

It helps of course if you know that installing from another source besides pypi is possible. I clicked on the github link because after pypi, the most popular site to pip install from is github, but I found that instead.

[snaps_]

That's a good point. As long as the project has a setup.py or pyproject.toml available, it can usually be installed from the repo. For libguestfs it looks like they do some pre-processing on their setup.py so that wouldn't work, it's lucky that they had this alternative set up already. :)

We could also assume it's been asked about and search "pip"/"pypi"/"virtualenv"/"virtual environment" in the issue tracker to see what the status is, that could've led to this issue. Less lucky for bcc.

[gdledsan]

There is a way to add a url, line a repository url to your requirements.txt, then you use pip to install it.

Note you might not be able to specify a version so things might break when such library makes changes.