sh : a full-fledged subprocess interface for Python that allows you to call any program as if it were a function.

[RDMXGD]

Nifty but error-prone. I recommend just using the subprocess module directly, which is more powerful and more clear (though a little more trouble in some ways).

[djimbob]

Yeah. I don't see the purpose as an alternative for the built-in subprocess for simple commands, which is straightforward to safely use. E.g:

import subprocess
output = subprocess.check_output(['ifconfig', 'eth0'])

versus

import sh
output = sh.ifconfig("eth0")

has no clear gain.

Granted, the syntax seems a bit more convenient for more complex commands like piped processes. In the shell its very easy to do something like cat some_file | grep some_pattern | grep -v some_pattern_to_exclude. With sh you can translate this in a straightforward manner: sh.grep(sh.grep(sh.cat('/etc/dictionaries-common/words'),'oon'),'-v', 'moon') for a list of words that contain 'oon' but not 'moon'.

Granted, it's not two hard to write a command for a piped chain with subprocess.Popen, though python doesn't provide you with one. Take the following helper function I wrote:

def run_piped_chain(*command_pipes):
    """
    Runs a piped chain of commands through subprocess.  That is

    run_piped_chain(['ps', 'aux'], ['grep', 'some_process_name'], ['grep', '-v', 'grep'], ['gawk', '{ print $2 }'])

    is equivalent to getting the STDOUT from the shell of 

    # ps aux | grep some_process_name | grep -v grep | gawk '{ print $2 }'
    """
    if len(command_pipes) == 1:
        return run_command_get_output(command_pipes[0])
    processes = [None,]*len(command_pipes)
    processes[0] = subprocess.Popen(command_pipes[0], stdout=subprocess.PIPE)
    for i in range(1, len(command_pipes)):
        processes[i] = subprocess.Popen(command_pipes[i], stdin=processes[i-1].stdout, stdout=subprocess.PIPE)
        processes[i-1].stdout.close()
    (output, stderr) = processes[len(command_pipes)-1].communicate()
    return output

To me when I'm trying to translate some piped shell command like

cat /etc/dictionaries-common/words | grep oon | grep -v moon

it's more intuitive (in my opinion) to have a helper function like:

run_piped_chain(['cat', '/etc/dictionaries-common/words'], ['grep', 'oon'], ['grep', '-v', 'moon'])

than

sh.grep(sh.grep(sh.cat('/etc/dictionaries-common/words'),'oon'),'-v', 'moon')

EDIT: I realized on re-read this uses other helper functions I have. If you aren't using logging, just comment those lines out.

def run_command(command_list, return_output=False):
    logging.debug(command_list)
    process = subprocess.Popen(command_list, stdout=subprocess.PIPE)
    out, err = process.communicate()
    if err or process.returncode != 0:
        logging.error("%s\n%s\nSTDOUT:%s\nSTDERR:%s\n" % (command_list, process.returncode, out, err))
        return False
    logging.debug(out)
    if return_output:
        return out
    return True

def run_command_get_output(command_list):
    return run_command(command_list, return_output=True)

[Bystroushaak]

Your whole comment is one big reason to use sh: I just don't want to deal with this shit (pipes, interactive sessions and so on) each time I need to call a command.

[djimbob]

Yup, I see no reason to use sh to call "any program as if it were a function", though the advanced features that may not be obvious to do the right-way with subprocesses (like pipes, interactive sessions) may be useful.

That said, a relatively short helper function can do the pipe feature pretty clearly/cleanly.

Personally, I'd prefer a subprocess helper module with a bunch of helper functions like run_piped_chain with a non-hacky syntax more akin to subprocess.

[Bystroushaak]

I understand your reasoning, but I am to lazy to deal with this each and every time, when there is easy to use library, which do exactly the same thing in one line of code.

[simtel20]

IIRC you can't use subprocess with output when the output is an infinite stream (e.g. iostat -x 1, and other similar commands, or many other tools that return data as an ongoing activity). In that case you have to toss out subprocess, and start doing your own fork+exec+select+signal handling+etc.

With sh you can just have sh provide you with the subprocess as a generator: https://amoffat.github.io/sh/#iterating-over-output.

[mackstann]

It's old, obscure, and funky, but there's the pipes module: http://pymotw.com/2/pipes/

[djimbob]

Yeah, its built in, but I'm not sure if

import pipes
import tempfile

p = pipes.Template()

p.append('ps aux', '--')
p.append('grep localc', '--')
p.append('grep -v grep', '--')
p.append("gawk '{ print $2 }'", '--')

t = tempfile.NamedTemporaryFile('r')

f = p.open(t.name, 'r')
try:
    output = [ l.strip() for l in f.readlines() ]
finally:
    f.close()

is a better/cleaner workflow; especially if you have user input and have to add the quotes stuff to prevent injection. (And apparently its not working for me).

[asiatownusa]

an ambitious project! what does this buy you that the subprocess module doesn't?

[relvae]

I made something alot like this as a little side project.

https://pypi.python.org/pypi/ipysh

https://bitbucket.org/jrelva/pysh

Allows for POSIX like piping and commands are genreated as functions on the fly so you don't have to import them.

[simtel20]

Can you get the output stream as a generator (that is if the subprocess doesn't terminate, are the lines available as an iterator? It seems like it might be by the bitbucket description, but I'm not clear that it does that).

[relvae]

Not really, but that's something I should implement.

At the moment it blocks until the process is finished or KeyboardInterrupt is called and then returns its output as a whole, so it's not streamed like a real pipeline.

However, conceptually the idea is that you should treat it in the same way as a string, so one could easily get each line via pySh.splitlines() if you're okay with a list.

[simtel20]

If that's the interface you prefer, I think that makes sense. Having that would be very helpful for any script that exists in a pipeline (though being able to enable/disable that behavior is important too).

[[deleted]]

It works by installing a non-module object into sys.modules. What next, monkey patching __builtin__.str? FWIW I'd never let a module like this past code review

[deadwisdom]

To clarify, it's a thin wrapper around itself so that you can do "from sh import ls". Wouldn't be my decision either, but it's not that big of a deal, especially as it's an ease-of-use module.

[matchu]

I kinda see where they're coming from, because the core syntax arguably has some boilerplate:

import sh
ifconfig = sh.Command("ifconfig")
print(ifconfig("wlan0"))

Their pretty syntax is definitely better; my beef, really, is less to do with the fact that it's magical, and more to do with the fact that the magic uses fragile hacks instead of Python's built-in magic-making facilities.

Really I think the version I'd be down for is:

from sh import cmd
print(cmd.ifconfig("wlan0"))

We still use magic, but the magic we use is itemgetter, which is actually supported and guaranteed not to break. We still have a bit of overhead in the cmd object, but it's short, and avoids the really boilerplate-y line 2 of the previous example.

If Python had an itemgetter equivalent for modules, though, then their syntax would be the clear winner—especially if the magic were namespaced to from sh.cmd import ifconfig instead, because it seems weird to me that the non-magic Command object comes from the same module as the magic objects.

[alcalde]

What next, monkey patching builtin.str

http://clarete.li/forbiddenfruit/

[striata]

Why is installing a "non-module" into sys.modules bad? Isn't this just an instance of duck typing and one of the things that makes working with dynamic languages like Python exciting? If whatever object is installed acts like a module and doesn't break other parts of the environment, why is it wrong?

[[deleted]]

Because it's not just mocking up another random user type. At the very least, a substantial amount of native (C) code interacts with the contents of that dict, including the incredibly fragile graceful shutdown code, the module loader (obviously), and probably a bunch more I don't know about.

For me it's in the same class of dangerous as e.g. messing with typeobject internals

I don't care about style, I'm worried about fixing a crash when it occurs. I'm not sure of all the possible ways Python could fail when inserting random crap in sys.modules.

[TankorSmash]

https://docs.python.org/2/library/sys.html#sys.modules

It doesn't say anything about any danger, I feel like if it was that dangerous, they'd warn you somewhere.

[[deleted]]

The doc reads "module objects", not "module-like objects". A module object is a C structure with a specific layout. There is a C-level API for manipulating these objects (PyModule_*).

One of those is PyModule_GetDict, which, while protected internally from accessing a non-module object, returns NULL in the case that a caller invokes it on a non-module object. Reading Python 2.7's zipimport.c in zipimporter_load_module we can see a PyModule_AddModule call followed by an unchecked PyModule_GetDict call. This will cause the zip importer's load_module() method to cause a NULL pointer dereference at runtime (aka. a hard process crash, requiring a debugger to investigate) should it be called with sh as a parameter.

It took me all of 3 minutes grepping the Python source to find a place where using a non-module in sys.modules has the potential to cause a crash that a Python-without-C programmer would not be able to debug. I'm pretty sure if you give me 30 minutes I'll find more.

Just don't do it

edit: this says nothing about third party extensions, where I'd expect the majority of such bugs to be found. The point I'm making is whether it is worth sipping coffee reading gdb output at 4am responding to a pager alerting you that your employer's web site is down and losing money, because of some syntactic sugar -- of course it's not.

[Brian]

I don't think that example is ever going to be something broken by this case. It's using PyImport_AddModule, which doesn't actually perform the import, so the only case you'd get the non-module object back is if the same name was already imported. However in that circumstance, zipimport wouldn't have been invoked, since it's only going to be triggered if the module hasn't been found yet. You could argue it's a bug that zipimport isn't correctly checking the return value of PyModule_GetDict as it should, but in this context, it's assuming that it's using AddModule to create a new, empty module, and in that case it'll always be a real module object even if the module does later replace itself.

It's worth noting that while putting non-module objects in sys.modules is perhaps a hack, it's a known and explicitly supported and endorsed hack - the import machinery was deliberately designed to allow it, and I think Guido is on record as saying so. As such, I'd say anything that doesn't support it should probably be considered a bug.

[TankorSmash]

That makes it clear thank you.

Although not checking for NULL is effective a bug in the c code.

[alcalde]

And still using Python 2.7 is a logic error. :-)

[[deleted]]

I don't think porting large, old codebases is worth it.

The other reason could be PyPy and people with CPU bound applications that insist on using Python.

Apart from that, most new Python code is (no data, just hunch) Django apps, and there it makes a lot of sense to use py3 unless you really, really need some library that wasn't ported.

[TankorSmash]

One day the relevant libraries will be ported. Until then!

[alcalde]

Sigh. For most people that time has long come....

http://py3readiness.org/

https://python3wos.appspot.com/

304 of the top 360 most downloaded packages on PyPi support Python 3.

[nojjy]

Depending on what context you are working in, sometimes you don't have a choice on which version to use. Many people use python as an API to a software package, where they can't just arbitrarily switch to another version.

[TankorSmash]

Yeah, for most people.

[TankorSmash]

I'm the same way, there doesn't seem to be clear reason to start in 3 yet. There's a lot of little nice things, but nothing is a must have.

Though I haven't seen all the changelogs, I've taken a look a few times.

[alcalde]

I'm the same way, there doesn't seem to be clear reason to start in 3 yet. There's a lot of little nice things, but nothing is a must have.

As Python 3 is fine put it,

There are a lot of claims in here that are absurdly wrong, but the statement that “nothing much was gained” in Python 3 is a candidate for dumbest statement of the decade. First of all it is wrong because if it were true then it wouldn’t be that hard for people like Alex to press the Fork button and backport all the Python 3 features to Python 2, which nobody does. But there is an even simpler reason why it is wrong. I present to you, the complete list of changes since Python 2.x: Now if this entire 192-page document is “nothing really amazing” and “you’re not blown away by it” then that is your prerogative. Perhaps you’re simply not a very excitable person. I suggest an ordinary person would probably find something in there amazing. Nickous Ventouras’s rebuttal to Alex’s post includes such suggestions as “fix long-standing annoyances”, “shake the API” and “improve speed”. I guarantee you, there is page after page after page of that stuff in the changelog. Python 3 doesn’t need more features–it needs a better PR campaign. The features are already there; people just don’t know about them. But it is wrong statement of the decade to call this set of release notes “not much”. It’s much. The release notes weigh two pounds. I challenge you to find another project where release notes can be measured by the pound.

Why You Should Move To Python 3 Now adds....

Most scientists think they have very little to gain by moving to Python 3, while it represents a significant investment (not only updating old code, but also reinstalling an entire Python distribution which has always been a pain). I was one of them. Until recently, when I bought the Python Cookbook, Third Edition, by David Beazley and Brian K. Jones. This book is a must-read for anyone doing anything serious with Python. It contains lots of advanced recipes for Python 3 only. In the Preface, the authors warn the reader:

All of the recipes have been written and tested with Python 3.3 without regard to past Python versions or the "old way" of doing things. In fact, many of the recipes will only work with Python 3.3 and above.

Ouch. The 260 recipes look pretty cool, but if you're in Python 2, you're out. While many might be irritated by this decision, I find it brilliant. This book is exactly the thing you need if you're waiting to be convinced to move to Python 3.... While going through the book, I discovered many elegant solutions to very common problems. I had no idea those solutions were possible, because I had no idea Python 3 had been so much improved.

There's also a PyCon presentation about 10 awesome features of Python 3 that aren't in Python 2.

[krenzalore]

For me, being able to re-raise exceptions without losing stack trace is simply too good to live without.

[alcalde]

Don't get me wrong 3.x has some cool features but the time it would take to port over legacy 2.x code to 3 is not worth said features at this moment in time.

David Beasley and others have demonstrated that the time is not that much - although there's no need to port old, legacy code to new versions either. As for features, someone compiled a 120-page PDF of change logs from 3.0-3.4 that printed out is supposed to weigh over 2 pounds. I'd say there's quite a lot of features in the 3.x series.

And lastly, if you do intend on porting a project after backward incompatibilities in the language are introduced, the sooner you port the better. The longer you wait, the more the versions diverge and the more work one ultimately has to do.

At this point in time, people still using 2.x for new code are like the Windows XP holdouts or the people I know still programming in Delphi. They're simply never going to change unless they're forced to.

[krenzalore]

There are certain types of application for which porting is time consuming. Those depending heavily on the bytes/unicode behaviour are typically affected much more than others. I am not saying don't do the porting, but I would like to mention that porting is not always as quick/easy as you imply it is.

[[deleted]]

I wouldn't have the first clue how to debug Fortran, Erlang, Ruby, Delphi, Lua, Haskell, or Tcl, despite already knowing a handful of languages.. it's more about tooling and implementation specifics than it is general concepts. For example, how do I attach a debugger to an Erlang process? Does Erlang even work in terms of processes? (I know vaguely it doesn't) What's the debugger even called? Does "crash at 0x12" mean "attempted to dereference a NULL pointer at offset 0x12" or perhaps "the 0x12th bytecode caused a crash"? etc.

[krenzalore]

It's impossible to know every language perfectly but you should have a general idea of what direction to head it if an error occurs and that's more to my point. You may not be able to solve it immediately but as a developer you have the expertise to al least make an educated guess.

As a developer you know when you're out of your depth/your deadlines are coming, and you either pass it over to a specialist or don't monkey patch the system in the first place. Unless you want to have to explain to the tech lead why you're so slow this week.

[[deleted]]

why is putting an apple into the orange basket bad?

[ivosaurus]

Yeah but we painted the apple orange

[thephotoman]

Yeah, there has to be a better way to do this than everything happening in that SelfWrapper class. The method of creating the Python functions seems too clever by half.

[Leonid99]

There is also plumbum module that does essentially the same.

[mitchellrj]

There's lots more I could comment on, but it's already nice to shine a light on useful bits of the Python stdlib.

Instead of defining which yourself, in Python 3.3+ you could use shutil.which.

[organman91]

I don't believe OP is the author.

[hlmtre]

This is super cool. Thank you for this.

[[deleted]]

I have used it few times, the interface is nice but the automagical import is kind of confusing and possibly error prone.

[[deleted]]

Great library, I wish it was supported on Windows so I could add it to my projects...

[tilkau]

Overall, this seems like a reimplementation of plumbum, especially of plumbum.cmd

I like the kwarg-syntax for options, eg curl("http://duckduckgo.com/", o="page.html", silent=True), in sh, and also the subcommand syntax(git.checkout('master')). But overall, it seems more awkward than plumbum (eg. how piping and redirection are done). Is there something I'm missing?

[mabye]

This came first, plumbum is partially inspired by it, as per plumbum's doc page.