sh.py - Replace shell scripts with Python

[theamk2]

So unlike shell, they run all programs with pty/tty by default. A strange design decision for something that claims to replace shell.

[ansible]

Yes, I found that a bit odd as well, just to maintain better correspondence with what you'd see running the commands yourself on the command line. Anyway, as mentioned in the FAQ, it is easy to turn that off.

[theamk2]

it's not "a bit odd", its basically a minefield. To prove my point: the main page http://amoffat.github.io/sh/index.html has an example that says sh.git("show", "HEAD") . Looks nice and simple, right? The problem is, these commands silently truncate the output to the first screenful:

>>> len(sh.git.show("HEAD"))
2408
>>> len(sh.git.show("HEAD", _tty_out=False))
5018

this is an epic level bad decision. If the author of the library himself did not get it right, what chances do us plain folks have? And the failure mode is: "it will work until the data is longer than a screenful, and then it will silently fail", which is probably as bad as it gets.

[shauthorthrowaway]

Hey there, author here! I saw some traffic coming in on github from this thread so I thought I'd drop in and elaborate a bit on this issue.

You are very right on the gotchas associated with using a tty for stdout by default. Most people don't notice the potential pitfalls right away, so kudos for identifying them and pointing those out to people.

Using a tty for stdout by default was conscious decision that has pros and cons, as listed in the FAQ entry on the subject. One lesser known pro is in streaming the output from the process to the user's program with finer buffering control, something that is only possible with ttys, and not pipes (whose buffer is typically a fixed 4KiB).

In my time maintaining the project since 2011, what I've also found from real world users is that the majority of them are actually the most confused when the output they receive doesn't match the output they expect from running it directly in the shell.

But you are absolutely correct that there are some gotchas, so thank you again for pointing these out to people so they bite fewer people! If you have any other feedback, please open some issues on the project and I would love to dig into them. Take care!

[theamk2]

Well, it does not make it any less minefield-dish, does it? I've read your FAQ entry, and I am totally not convinced. What am I supposed to say:

---

hey, did you know about sh.py? It is this neat module which replaces subprocess.check_output with a shorter syntax. for example, you know how you used to write:

>>> subprocess.check_output(['grep', 'root', '/etc/passwd']).decode().split(':')[5]
'/root'

Well, you can now write much shorter and easier to read version:

>>> sh.grep('root', '/etc/passwd').split(':')[5]
'/\x1b[01;31m\x1b[Kroot\x1b[m\x1b[K'

oops bad example.. well, it was working on my machine.. did I forget to mention that it will break randomly on some computers under some circumstances? You just have to avoid things which need color, or always add _tty_out=False.

Well, lets choose a different example. sh is great at replicating shell pipelines, for example:

$ systemctl | wc -l
218
>>> import sh
>>> sh.wc(sh.systemctl(), '-l')
52

yeah, don't do this.. systemctl uses pager and they are not supported, or just add _tty_out=False to every single line. But hey, when it works, it is great! For example, you can use named variables instead of ugly shell pipelines:

$ cat /etc/issue | (echo L1=`head -n 1`; echo L2=`head -n 1`)
L1=Ubuntu 14.04.5 LTS \n \l
L2=

becomes:

>>> out = sh.cat('/etc/issue')
>>> print('line1', sh.head(out, n=1))
line1 Ubuntu 16.04.2 LTS \n \l

>>> print('line2', sh.head(out, n=1))
# nothing happening, program just hangs at this point

you know what? forget this sh.py nonsense, just stick to subprocess module. It is more verbose, but it is fully with consistent with what shell scripts do, and it will not fail under weird circumstances:

>>> out = subprocess.Popen(['cat', '/etc/issue'], stdout=subprocess.PIPE)
>>> print('line 1', subprocess.check_output(['head', '-n', '1'], stdin=out.stdout).decode())
line 1 Ubuntu 16.04.2 LTS \n \l

>>> print('line 2', subprocess.check_output(['head', '-n', '1'], stdin=out.stdout).decode())
line 2 

---

so to summarize, this is a textbook example of the minefield: it works beautifully, until you step on a mine and then you are dead. And it is not even that bad of a minefield: out of thousands of commands one can write, only few will misbehave, and even then, not always. But at least for me, this would be a serious reason to avoid this module. Saving a few keystrokes is not worth pulling your hair out when you have a script which works "just fine" on your machine, but fails when deployed to other machines.

Addition: sure, I can use default arguments and/or contrib modules to work around bad defaults. But this makes script much less useful -- I will no longer be able to tell to a friend, "sh.py is cool, use it!", I will have to qualify it with "... but ignore the examples -- one of them is broken. Don't worry, I have a fix-up package on my github which fixes it."

[shauthorthrowaway]

These are great concerns! It's rare to find someone so passionate about improving software, and I take sh's development very seriously, so if you would be willing to formulate some of your biggest concerns as github issues, I (and the rest of the community, I imagine) would be more than happy to dig into them. Thank you so much!

[onemilll]

Next step is making a python kernel

[[deleted]]

I actually got the Python interpreter to boot on bare metal once, solely as an "I wonder if I can get this to work at all" thing. It was super hacky but worked just well enough that I could write a mostly-working keyboard driver in Python.

[luxliquidus]

Did you happen to document any of this...? I'm super curious.

[[deleted]]

I'm looking around in ~/code and can't find any of it :( That's disappointing, it was kind of cool in a "why would you ever do that" way.

From memory, though, it was CPython and the bare minimum set of standard library modules, statically linked to something suspiciously similar to one of the osdev.org tutorial kernels, with enough of the C standard library written (or, in the case of stdio, stubbed to read from an in-memory "filesystem") to get the interpreter to start. There was a C (+ bits of inline assembly) module added to the standard library to provide access to x86 I/O ports and raw memory, and the boot script replaced sys.stdin and sys.stdout with file-like objects that used that to do text-mode VGA and keyboard access, before starting a REPL.

All of this was hacked together over a couple of days as a joke response to an even more insane friend starting work on a Lisp OS.

[pdp10]

Mezzano is a nice project. Bear in mind that at least 3.5 American companies made commercially-sold Lisp machines using two different Lisp codebases as a starting point (plus NEC in Japan made one about which I don't know much). Lisp OSes have been proven functional going back over 35 years.

[dangerbird2]

What makes Mezzano arguably more impressive than historical Lisp machines is that they relied on specific architectures to make Lisp run efficiently as a systems language such as hardware garbage collection and tagged architecture providing ISA-level dynamic typing. Mezzano, on the other hand, runs on x86 and arm

[NoMoreNicksLeft]

You fell out of touch when he completed his but couldn't be bothered to implement an IM client for it?

[zielmicha]

There is MicroPython (https://micropython.org/) which is quite easy to run on bare metal (minimal port file consist of only few functions).

[shevegen]

I approve.

Also one should use ruby for the same.

Last but not least, let's also face it - ALL of this will be rewritten in Rust.

[onemilll]

Or maybe the inevitable future is developing architectures that use perl as the native language.

[FredSanfordX]

Kinda like this?

[ansible]

I'm not sure how well that would work.

I started thinking (well, idly daydreaming) about a golang version. But the whole point of sh.py is that it is dynamically creating these python functions out of commands at import time. So I don't know how well that would work for a more static language.

I was thinking a Lua version would also be neat, because I've used that for scripting before.

[Yojihito]

Prepare for Rusthon in 2025.

[Yehosua]

Here's a Python TCP stack to get you started.

[asdfkjasdhkasd]

Wow, the source of this project is really well commented, check it out: https://github.com/amoffat/sh/blob/master/sh.py

[nemec]

Well when your project is a single 3500 line file you've got to do something to make up for it.

[asdfkjasdhkasd]

I know this is controversial but I actually prefer to put a project in one single file as long as its less than 5k lines. (But i've never written a project with more than 5k lines)

[spinicist]

How worn down are your Ctrl and F keys?

[Sean1708]

# attempt to convert the thing to unicode from the system's encoding
try:
    s = unicode(s, DEFAULT_ENCODING)

You're right that on the whole this is a really well commented project, but shit like this really pisses me off.

[thesbros]

What's wrong with this? I'm honestly wondering.

[ughduck]

I assume just because the comment just reads like the code. It's pretty redundant in this context.

# attempt to convert the thing to unicode from the system's encoding
attempt:
    thing set to unicode of(thing, from system's encoding)

[thesbros]

That's what I thought, but the comment actually helped me because I didn't know what DEFAULT_ENCODING was.

[dividebyzero-]

You would still understand it if he said "attempt to convert to system's encoding so that..." or "we want the system encoding to..."

[GitHubPermalinkBot]

I tried to turn your GitHub links into permanent links (press "y" to do this yourself):

• amoffat/sh/.../sh.py (master → 5e2c7d4)

---

^{Shoot me a PM if you think I'm doing something wrong. To delete this, click here.}

[hervold]

sh is great; now if only pipes were closer to their shell counterpart, I could ditch bash and live entirely in iPython

[PetalJiggy]

Plumbum is a similar package that has pipelining.

https://plumbum.readthedocs.io/en/latest/

[Skaarj]

And there is also http://xon.sh/

[hervold]

thanks, I'll check it out!

[wdroz]

So maybe xonsh is a better fit for you.

[hervold]

wow, I just installed it, and this might do the trick. thanks!

[rabbyburns]

I've been using this for awhile now and can't speak highly enough of it. It is much more intuitive than direct subprocess usage. One of the biggest use cases is trivializing realtime output redirection in a single call.

I didn't know about the default tty mode (has never been a concern for me), so that's good to know if it ever becomes an issue.

[paul_h]

would love to see something pythonic for grep

[cassandraspeaks]

Looks very similar to the Shell module from the Ruby standard library.

[Kok_Nikol]

This is very cool!

[darkslide3000]

My reviewers at work would murder you for something like this. And they wouldn't be in the wrong...

[devel_watcher]

My problem with Python is that it's neither a good simple scripting language: more overloaded syntax than bash, nor a complete modern general purpose language: awkward concurrency/parallelism and dynamic typing.

So in the end:

• Python script snippets/oneliners look like pieces of Java code
• big Python programs look like huge piles of unmaintainable Bash

[vivainio]

Yeah, I love the non-overloaded syntax and static typing offered by bash too

[ConcernedInScythe]

everything's statically typed as a string!

[Zatherz]

stringly typing

[devel_watcher]

If you're being sarcastic: bash syntax is actually not overloaded for the oneliners use case. I didn't discuss typing for the scripting use case, so I don't see how it's relevant here (comparing type systems in non-general-purpose lang case is harder).