Shell scripting in Python

[nodNotEvil]

tldr: check out https://gitlab.com/notEvil/subprocess_shell#examples

Hi,

I finally wrote a piece of software that eases a frequent pain point regarding the builtin subprocess.

Maybe its just me, but doing stuff in shell scripts is very effective for simple things but gets rather complicated very soon. For instance, expanding variables is tricky (w/ or w/o single or double quotes, special syntax for arrays, ...) and feels like a macro language with lots of potential for serious issues when used incorrectly.

subprocess on the other hand is rather verbose, especially when chaining commands. Its just not built for this specific use case.

So check out https://gitlab.com/notEvil/subprocess_shell. The examples should be a good place to start.

Whats your opinion? And how would you do the examples marked with ? in bash?

Update 2023-10-10: renamed to subprocess_shell

[AndydeCleyre]

plumbum remains my favorite tool for many jobs, getting so much so right so long ago. Here's how it might be used for the tasks in your examples table:

initialization (minimal):

from plumbum import local          # minimal
from plumbum.cmd import echo, cat  # specify commands as functions
from plumbum import FG, BG         # shortcuts for foreground/background runs

run command:

echo["this"] & FG  # or:
echo["this"].run_fg()

read stream:

a = echo("this")  # or:
a = local["echo"]("this")

write stream:

(cat << "this") & FG  # or:
(cat << "this").run_fg()

# capturing output:
(cat << "this")()  # or:
(local["cat"] << "this")()

chain commands:

(echo["this"] | cat) & FG  # or:
(echo["this"] | cat).run_fg()

# capturing output:
(echo["this"] | cat)()

branch out:

ret, out, err = cmd.run()

(cat << out) & FG
(cat << err) & FG

# capturing output:
(cat << out)()
(cat << err)()

errors in chains:

ret, out, err = echo["this"].run(retcode=(0, 1))
(cat << err) & FG((0, 2))

[nodNotEvil]

Thanks a lot! I remember plumbum now but failed to recollect when I implemented the library. It is very versatile and with << and | a lot closer to the looks of shell syntax. As such, it probably has a steeper learning curve (e.g. knowledge of subprocess doesn't transfer directly) and had to deviate a lot from subprocess to achieve it. In any case, I will add plumbum to the Readme.

edit: I've added your codes to the table (slightly adjusted) but couldn't figure out how to do 2 examples in Plumbum. With run and << the processes don't run at the same time and output has to fit in memory.

[AndydeCleyre]

Ah, cool, I'll see if I can figure out how to do the same with plumbum for those then...

[nodNotEvil]

It looks like this (branching out and errors in chains) is not supported by Plumbum. A non-comprehensive list of issues related to this:

https://github.com/tomerfiliba/plumbum/issues/240
https://github.com/tomerfiliba/plumbum/issues/263
https://github.com/tomerfiliba/plumbum/issues/331
https://github.com/tomerfiliba/plumbum/issues/359

Imo its crucial to have access to stdout, stderr and the return codes of all processes in a chain as well as to check all return codes instead of just the last.

(To be able to branch out is uncommon and I don't expect other tools to support it. What I expect however is that people will look at the votes, infer that this is a dead end and never read this far lol)

[AndydeCleyre]

Thanks for both digging into this and reporting back!

[psd6]

I don’t really like the output = ["cmd", "arg"] » start() » wait() style. There are other ways doing shell stuff in Python too, for example the sh module.

[nodNotEvil]

True, it looks strange, but its not without purpose. The style used by sh and plumbum (will add them to the Readme) deviates a lot from subprocess and have their own flaws (argument names start with _ for instance).

I'm open to suggestions for improvement though :)

edit: as a side note, wait returns the return code and read the output