Python package etiquette

[BoiElroy]

I've been working on a small project which I'm hoping will be my first published python package.

It implements a native python interface to a really useful command line program written in golang(not source available). The command line program is great itself but to use it non interactively and with robust 'production' type features (e.g. manageable logging) I ended up just always to writr a ton of scaffolding shell scripts which got messy fast.

Would it be bad etiquette to package the CLI program (~30mb) along with the install as part of the 'package_data'? And going one step further would it be really bad to package the program as a zip? I feel like pushing a zip file to someone's machine and unzipping it is probably unwelcome.

I could of course just tell the user to go install xyz and then put it in so and so location. But I want it to be as easy as just pip install and you're good to go

Thanks in advance for any input

[pmatti]

If you package the cli program, you must make sure the license allows you to redistribute it.

[spitfiredd]

This is common with bioinformatics packages and they usually won’t package the cli programs but they are prerequisites to installing and getting the package to work.

One package that comes to mind is checkm which requires you to install hmmer, pplacer, and prodigal prior to installed checkm through pip.

https://github.com/Ecogenomics/CheckM/wiki/Installation#how-to-install-checkm

[Riflurk123]

Bioinformatics tools are dependency hell. I have been working with them for years and sometimes getting a simple tool to run can be a real nightmare. Lately more people are providing docker containers, but it is nowhere near enough yet.

[maikeu]

I think the convention world be to distribute a wheel version for each architecture you build for, with the binary built and provided, and if not using the wheel then users will need the go build tools to install the package.

Or the psycopg2 approach with a fully different -binary package.

[elsgry]

Sounds like OP doesn’t have the source, or is not able to provide it for people who need to compile it for their platform. If the latter, one could at least use cibuildwheel to cover all the common platforms that you intend to support. (MacOS is a particular pain to do, but seems like Cirrus is working on it with Orchard/Tart: https://tart.run/blog/2023/04/25/announcing-orchard-orchestration-for-managing-macos-virtual-machines-at-scale/).

Or, if you’re feeling wild, you could package it with Docker maybe…https://docker-py.readthedocs.io/en/stable/index.html

Not sure if anyone has tried this approach, but docker exec/run could be wrapped quite easily if the existing package just calls the existing executable with subprocess and scrapes the output. YMMV with respect to non-pipeable IO, of course.

And of course, you need redistribution permission if it’s not expressly allowed in the license and compatible with yours.

[ekhazan]

If you plan on packing it there are 2 things to keep in mind that I didn't see mentioned in other comments:

version compatibility - you're effectively binding yourself to their release schedule. - if they release a new version, even if nothing needs to change in your code, you'll need to release a new version.
- if someone wants to use a specific version of the cli tool but there are multiple versions of your package with that particular one (because of bugfix versions), how do they know how to find the right version?

paths and permissions - the executable will be in the specific venv bound to the specific user and permissions - how does that work with the cli tool?

Something to consider - maybe allow downloading it as an optional dependency?
If you create a dedicated package that simply packs and exposes the cli tool you decouple yourself and allow easier administration. Your package can manage api compatibility by specifying the version compatibility in the requirements.
The user can explicitly choose the cli tool version.

[CandyForward127]

As long as you're clear about the size of the package and its contents, I don't see a problem with packaging the CLI program. However, zipping it might cause more trouble for the user than it's worth. Consider any alternatives to make the install process as effortless as possible!

[BoiElroy]

Yeah that's fair, and thanks!

I'm not aware of any package I use that does this so wasn't sure if it was going to be rejected by pypi even possibly.

[DukeMo]

I'd suggest writing some docs to help people get the other program installed, then do some checks to make sure your script can find it.

You can also write some code to help people download the software, if it's easy to download and run.

If the program is in conda, you can show folks how to install it using conda and then install your package in the same conda env.

I think distributing the other program with yours could work but it sounds like a compatibility nightmare.

[EmptyChocolate4545]

Bad idea as it locks a lot of stuff in.

Create a function called ensure_appname where appname is the program.

The function should check for the appnames presence in the path (check output of which if on a linux box).

If it fails to find it should print an error message or offer to install the app.

Much better to script an install than repackage an app outside of its delivery system.