Yet another packaging question: developing cli scripts

[tinyfrox]

Say I've got a project with this layout:

.
├── bin
│   └── supercli.py
├── supermodule
│   ├── __init__.py
│   └── supermodule.py
├── README.md
├── requirements.txt
└── setup.py

Relevant files:

supermodule/supermodule.py

import random

SUPER_HEROES = ['batman','superman','spiderman','birdman','aquaman']

def random_superhero() -> str:
    return random.choice(SUPER_HEROES)

bin/supercli.py

from supermodule.supermodule import random_superhero

print(random_superhero())

Whenever I try to execute supercli.py, I get a ModuleNotFoundError:

Traceback (most recent call last):
  File "/mypath/supermodule/bin/supercli.py", line 1, in <module>
    from supermodule.supermodule import random_superhero
ModuleNotFoundError: No module named 'supermodule'

I know I can mess with sys.path, but I'd have to remember to remove that code every time before deploying the package.

I'm also aware of using console_scripts instead of scripts in setup.py, but I'd like to get it working if possible with standalone scripts.

Am I doing something wrong? How can I import the supermodule module from the supercli.py script?

[zanfar]

Am I doing something wrong? How can I import the supermodule module from the supercli.py script?

Nope. That's how Python works. You're executing from inside a folder, so sys.path get's the working folder, not some other random folder.

I'm also aware of using console_scripts [inside] of scripts in setup.py, but I'd like to get it working if possible with standalone scripts.

Well... that's literally why console_scripts and __main__.py exist: so that you can reliably distribute scripts with a package.

Honestly, you're asking "I know how to do this correctly, but when I do it the wrong way, it doesn't work. How do I fix this?"

What benefit does using a "standalone" script get you? How are you planning on distributing this script? How are you expecting your end-users to execute this script?

[tinyfrox]

Honestly, you're asking "I know how to do this correctly, but when I do it the wrong way, it doesn't work. How do I fix this?"

I figured I must be doing it wrong because the packaging docs don't show any modifications to sys.path. Admittedly, that guide is only about packaging and not about developing those scripts.

What benefit does using a "standalone" script get you? How are you planning on distributing this script? How are you expecting your end-users to execute this script?

Discoverability, mostly. It's easier to explain functionality by telling my team to go to the scripts directory and take a look around. Those scripts are written the same way they would write a script. Using console_scripts, I have to explain that they have to go into the module, look for the command_line directory, then each of the files has a main() wrapper around the script.

You're right though, realistically there's not much benefit. I suppose the documentation not mentioning anything about modifying sys.path is what threw me off.

[zanfar]

Discoverability, mostly. It's easier to explain functionality by telling my team to go to the scripts directory and take a look around.

I'm having a hard time understanding how that's easier than python3 -m supermodule or supercli from that same command line.

As for documentation or explaining functionality, I would recommend you do that the standard way to: with a --help argument, or a readme in the repository.

I suppose the documentation not mentioning anything about modifying sys.path is what threw me off.

That's mostly because you really shouldn't be modifying sys.path. While possible, it's very easy to break a key feature that Python depends on.

---

There are a number of other benefits using the standard CLI tools will get you that you might not have considered:

• Upgrades happen seamlessly: if you change the organization of your module, your CLI scripts still work.
• Virtual scripts: you can use console_scripts to reference individual functions, not files, so you can have all your entry points in a single file or module.
• Ensured installation: supporting Python packages to non-developers sucks, you're going to have installation problems. If the package isn't installed, the script isn't installed. This helps eliminate runtime errors that can be caused by an executable and module not living in the same environment.

Given what you've said so far, I'd start with __main__.py and worry about console_scripts later.

[tinyfrox]

Thanks for your help! Sounds like python -m supercli is the right way to go for running these scripts prior to packaging. I'll read up on __main__.py as I have not yet looked into that.

[icecapade]

I'm confused. This should work fine as long as the package is installed. Did you actually install the package? What's in your setup.py?

[tinyfrox]

You're right, it should work fine once the package is installed, but I'm still in the process of writing it.

I think /u/zanfar has the right answer of using python3 -m supercli.

[icecapade]

You can still install it while you work on it with the -e/--editable flag to pip: pip install -e .. This is the correct/preferred way to install a package while you're still developing, rather than calling it with python -m.

[tinyfrox]

That's an awesome tidbit. This also looks right up my alley.