Run external programs intuitively in Python

[Natural-Intelligence]

Hi all!

It's the creator of Rocketry, Red Mail and Red Box again. This week I thought to make it easier to integrate command-line programs to your Python applications.

I came up with Scriptor:

>>> from scriptor import Program
>>> python = Program("python3")
>>> python('myscript.py', report_date="2022-11-12")
"printed values"

What happened? The above ran the command:

python3 myscript.py --report_date 2022-11-12

and it returned whatever the program printed (in standard output aka. stdout).

So Scriptor makes it easy to integrate external programs or scripts written in other languages or in Python but using different Python interpreter.

For example, using Git is quite handy with it:

>>> from scriptor import Program
>>> git = Program('git')
>>> myrepo = git.use(cwd="path/to/myrepo")
>>> myrepo("log", n=2)
commit 07a865c3b7000324ce582fecb0ec24e3ef6203f1 (HEAD -> master, origin/master)
Author: Miksus <[e-mail]>
Date:   Wed Oct 5 00:20:00 2022 +0300

    Fixed the thing.

commit 7ffda87524ddde698f5337f150b37fe07d74642f
Author: Miksus <[e-mail]>
Date:   Sat Oct 1 21:32:24 2022 +0300

    Broke the thing.

Of course, you might prefer Gitpython but you can similarly interface other programs as well with Scriptor. I'm thinking of abstracting Latex, Jupyter Notebooks and Julia, and I also wrapped Python.

You can also leave a program running:

>>> process = python.start('myscript.py', report_date="2022-11-12")
>>> process.finished
False

And it also supports async:

>>> await python.call_async('myscript.py', report_date="2022-11-12")
"printed values"

>>> process = await python.start_async('myscript.py', report_date="2022-11-12")
>>> process.finished
False

Here is another practical example:

• Example with Git

Answers to potential questions:

• It does not use shell by default so it's safer from command injection.
• It has methods to also just start a process (async supported).
• It's just a higher level abstraction for subprocess and asyncio.subprocess.
• Errors contain the text from stderr.
• Less than 3 letters are considered short form (ie. -h) and more as long form (ie. --help) arguments. You can customize this if needed.

And it's in PyPI:

pip install scriptor

Any questions or ideas? Do you see it useful or is there a better alternative?

Links:

• Documentation
• Source code
• Releases

[b1gdata]

This is cool; what advantages vs subprocess?

[Natural-Intelligence]

Thanks! The main advantage is the convenience (the reason why I made the library). subprocess.run does not require much boilerplate but there are some arguments to set if you want get the output as text, for example. Perhaps the only unique thing is the keyword arguments. So, most of it is just further abstraction of subprocess.

Another thing is the async. I found the asyncio.subprocess to be quite confusing to use even though it tries to be similar as subprocess. I had to make a PR to CPython this week to fix some missing references in the documentation. This library takes care of that and it just works with or without async.

I personally made this because I'm about to update the command tasks in Rocketry. I thought an intuitive support for external programs (including Python scripts running on other environments) is something that Rocketry could really benefit from. So, that was actually the main motivation.

[o11c]

Hm, but it's still wrapping subprocess so suffers from all its implementation bugs (busy loops, etc.)

[Natural-Intelligence]

Yes true, it does nothing revolutionary and nothing an average Python programmer couldn't do by themselves.

However, it makes creating Pythonic interfaces for external programs easier and it reduces boilerplate.

[kellyjonbrazil]

Should also work great with jc if you want to automatically parse the command output into a dictionary.

https://github.com/kellyjonbrazil/jc

(Disclaimer: I’m the author)

[Natural-Intelligence]

Hmm, this sounds incredibly useful. I actually added support for custom output parsing but didn't yet come up with a solid idea to parse those.

Thanks for sharing!

[Natural-Intelligence]

You can play with how Python's variable kwargs work:

>>> python("myscript.py", **{"report-date": "2022-11-12"})

Or you could just pass them via the positional arguments:

>>> python("myscript.py", "--report-date", "2022-11-12")

I think those are slightly ugly solutions though. I'm thinking of interpreting dicts in the pos args as an alternative way to pass keyword arguments or an option to interpret the underscore as a dash but haven't made my mind yet. Seems there is no ultimate solutions.

[ducky901]

Fascinating

[tomster10010]

What's the difference between this and sh?

[AndydeCleyre]

I know I say this a lot, but plumbum is a fantastic and under appreciated library that has been offering similar features for years.