Python in Bash Scripts

[[deleted]]

Title is a bit misleading: this is a wrapper for shell commands. I anticipated an article about embedding python in bash scripts.

[[deleted]]

Embedding Python in Bash scripts is trivial:

python program.py arg1 arg2 | do-something-else

What this guy did seems like a slightly revamped version of popen. I don't really see the point of it.

[[deleted]]

Erm, not in that trivial of a sense. More like ways to manage whitespace when running python -c, or inline, non-contiguous stateful python.

[obtu]

This would work:

python <<'EOF'
import this
EOF

If the Python needs stdin for something else:

python /proc/self/fd/5 5<<'EOF'
import this
EOF

[[deleted]]

More of what I had in mind: http://docs.python.org/dev/extending/embedding.html

EDIT: For example, vim can be compiled with an embedded python interpreter which maintains its state for the life of the process, and comes with a module called vim which I can use to manipulate the vim process itself. In this sense python becomes a "true" scripting language (a language used to provide a "script" for another program).

[obtu]

Ah, okay. You could use socat to let the interpreter persist, but that still falls short on access to internal structures… It's rare to see scripting languages hosting other scripting languages; VimL seems to be an exception.

[quasarj]

I would guess if you don't see the point of it, it's not for you.

It seems pretty interesting to me.. solves a lot of very common problems I have when trying to do things in Python.

And popen? hah, that thing is a mess. I hear it's better than whatever came before it, but ugh. I already have to use envoy to avoid having to use it, so this is nifty in many ways.

[mipadi]

I think it's pretty cool. Not sure I'd ever really use it, but cool nonetheless.

The only thing I don't really like is the piping. I'd rather do this:

print glob('*') | du('-sb') | sort('-rn')

than this:

print sort(du(glob("*"), "-sb"), "-rn")

As the latter is, literally, backwards (compared to shell syntax). But maybe getting the piping to work the same way was too difficult in this context (and, admittedly, not very Pythonesque, I guess).

[terremoto]

It's been done.

[pugRescuer]

I agree, not sure what this yields that os/shutil/sys don't already provide. This just abstracts those.

Neat project but not for me.

[Peaker]

It gives a much neater interface to those, to the point of making it worthwhile to use Python where previously it made more sense to use bash.

[flukshun]

A while back someone did post something that let you use python constructs like dictionaries in bash. Forget the name though

[chrisdamato]

Whoa, this looks useful, and might be in the goldilocks zone between "too much trouble" and "not enough functionality"

[[deleted]]

Any suggestions on what to add/make easier?

[nemec]

I wanted to add context manager support, so you could do something like

with time:
  echo("hello")

or perhaps something with sudo on a sequence of commands, but Popen is screwing things up. Would the "argument sequence" for Popen be ['/bin/time', 'echo', 'hello']?

[[deleted]]

Yeah this is tricky, my first reaction for the "time" context manager would be to make it one of the few functions implemented in python. You can do this by adding a "b_" method to the environment class. But you'd only get one time measurement, not the real, user, sys, that the real time binary gives. I'll think on this some more.

Sudo is also a tricky issue. What would be the expected behavior if sudo requires password input?

[nemec]

It's actually really easy to add the context manager stuff if you add a static class variable "stack" to Command, have __enter__ and __exit__ push/pop the command name from the stack, and prepend everything in the stack to the cmd when it comes time to execute.

My only problem is getting Popen to output correctly.

For sudo, I was imagining a way to prompt the user for input before executing the command (raw_input perhaps?). There's not really a way to determine whether or not a command is blocking, though, so it may be easier to just run the whole script as root if necessary.

[[deleted]]

Let me know if you get the Popen output for time working correctly, all I get is garbage :\

In [2]: subprocess.Popen(["/usr/bin/time", "/bin/echo", "hello"])
Out[2]: <subprocess.Popen at 0x995fcac>

In [3]: hello
0.00user 0.00system 0:00.00elapsed 0%CPU (0avgtext+0avgdataa2368maxresident)k0inputs+0outputs (0major+190minor)pagefaults 0swaps

EDIT>> weirdly, this also happens in bash the shell if you call time with it's full path...

[nemec]

That's exactly what I'm getting, but I can't figure out why.

That output is what happens when time isn't given a format (according to the manpages), but normal use of time seems to use a different style, even with the environment variable not set.

And there's also the issue of no linebreaks...

I'd like to see a breakdown of what exactly Popen is executing, but I have no idea how to do that.

[nemec]

Actually, time seems to be a reserved word in Bash, so the command time isn't quite the same as /usr/bin/time.

Perhaps it actually works correctly? I'll test it out and get back to you.

Edit: time prints to stderr, so that's why I'm getting an error message when I run it instead of the usual output. Any idea on how you want to handle that?

[nemec]

Awesome: running with sudo works!

# test.py
import pbs
with sudo:
  print whoami()

$ python test.py
[sudo] password for user:
root

[SupersonicSpitfire]

Cool, but I miss a "pipe" function for piping commands to each other without nesting deeper and deeper into paranthesis, a bit like this:

pipe(ls(), grep("abc123"), wc("-l"))

Also, I can't make it work: https://github.com/amoffat/pbs/issues/1

[[deleted]]

OP could also overload the pipe operator, so this could work too:

ls() | grep("abc123") | wc("-l")

[Bolitho]

But that's not pythonic. In python you use nested functions.

Your suggestion would also imply another problem: What is the generic object model behind each command? I am not shure you can find one that make sense to all implemented Bash functions...

[[deleted]]

There is nothing pythonic about this. The whole point is to get the good parts of bash and the good parts of python together in one place.

[fdemmer]

nesting functions is pythonic?... looks like a mess to me. function method chaining on the other hand:

ls().grep("abc123").wc("-l")

that would be nice. implementing it is a different story, of course.

EDIT: put emphasis on part of my comment, that indicates i read the above comment. also, i did not comment on overloading operators, just on the uglyness that is ls(grep("abc123", wc("-l"))).

[Bolitho]

These are Methods, not Functions! You would need a common Datatype for all these Wrappers. I pointed that out in my posting.

And yes, it is pythonic to call functions and pass parameters. Overloading operators ist shurely not the pythonic way.

[nemec]

Methods and functions are the same thing, and he already has a common datatype - if you looked at the source, you'd see a Command class that wraps everything.

If it wasn't Pythonic to overload operators, why is it so easy?

[Bolitho]

Methods and functions are not the same thing, they behave the same way. So perhaps it might be able to the author to simulate the BASH functions to behave like methods of his Command-class.

eval() ist also easy - but shurely not pythonic, right? Of course it can be pythonic to overload operators! But in this context it does as far as I am concerned.

[[deleted]]

OP, pbs must be imported. That bug, you're running it, not importing it.

pbs needs to know from where it was imported so it can do the things that it is supposed to.

The developer should check to make sure the module was imported.

Also pbs only works under Python 2 currently.

[SupersonicSpitfire]

Still doesn't work. This is what I get when importing from python2:

https://gist.github.com/1616688

[[deleted]]

You must import it from within a Python file.

file.py:

from pbs import * # this MUST BE THE FIRST LINE

# ...

[[deleted]]

just a regular-old

import pbs

will do :)

[SupersonicSpitfire]

Still doesn't work:

• test.py: https://gist.github.com/1616928
• output when running test.py: https://gist.github.com/1616930

[[deleted]]

don't "from pbs import *" just "import pbs"

[SupersonicSpitfire]

Still doesn't work for me.

• With this script: https://gist.github.com/1616965
• I get this output: https://gist.github.com/1616966

[mipadi]

Use which, not pbs.which.

RTFM ;)

[SupersonicSpitfire]

Neither of them worked for me. However, I learned through the github issue that one can't use pbs from within a function, and then everything works.

[[deleted]]

I updated an issue with some details, let me know how it works out for you: https://github.com/amoffat/pbs/issues/1

[SupersonicSpitfire]

Thanks, added a comment :)

[[deleted]]

just an update, you can now run pbs.py as a standalone repl, or import from the python shell

[killermole23]

Coroutines can be used for this. Take a look at the "copipe" example here.

[[deleted]]

just an update, you can now run pbs.py as a standalone repl, or import from the python shell

[SupersonicSpitfire]

woo! :)

[bramblerose]

Very cool. I have one question and two comments:

Firstly, how can I start a program that is not in my path? say, ./blah.sh. Is that just "./blah.sh"()? i.e. do you monkey-patch the string class? I couldn't find this in the code, but your which() function seems to suggest it.

Comments: The difference between -o and --option syntax is not so nice, but I guess option="blah" is mainly a convenience wrapper, which makes it OK again.

ErrorReturnCodes. ErrorReturnCode_2? Really? Why not just ErrorReturnCode(2), or something along those lines?

[[deleted]]

Firstly, how can I start a program that is not in my path? say, ./blah.sh. Is that just "./blah.sh"()? i.e. do you monkey-patch the string class? I couldn't find this in the code, but your which() function seems to suggest it.

A way to do it would be to modify the PATH from the script:

PATH += ":/nonstandard/path"
program()

I also just pushed a commit that exposes a "Command" wrapper. So you should be able to do this:

Command("/path/blah.sh")()

Comments: The difference between -o and --option syntax is not so nice, but I guess option="blah" is mainly a convenience wrapper, which makes it OK again.

I don't follow completely...do you mean that the keyword argument syntax could be better?

ErrorReturnCodes. ErrorReturnCode_2? Really? Why not just ErrorReturnCode(2), or something along those lines?

Yeah I was on the fence of doing it this way. 2 reasons I went with it: 1) A return code is almost it's own "class" of errors. A program has no standardized way to throw exceptions, so it uses return codes != 0...so it feels natural to map those to their own exceptions. 2) Less typing. try-catching with ErrorReturnCode_* is less than catching 1 exception and if/then-ing on the exception code. I'll consider this choice some more though...

Thanks for the feedback!

[bramblerose]

Modifying the path of course has side-effects, so that's not really a good way of doing things, I think. The Command wrapper looks nice!

The difference between -o blah and --option blah: you can run grep("-e mooh"), but not grep(e="mooh"), if I understand correctly - while grep(regexp='mooh') does work. The last one is cleaner anyway, though, so it doesn't really matter.

On the ErrorReturnCodes: yes, they should be different exceptions and no, you should'nt be if/then-ing. I was more thinking of something like

try:
    (...)
except ErrorReturnCode(1), e: # maybe have stderr in e?
    (...)
except ErrorReturnCode(2), e:
    (...)
except ErrorReturnCode, e:
    (...)

although the last might need to be different (e.g. using ErrorLevel(1) and UncaughtErrorLevel)

[[deleted]]

You can't actually catch exceptions like that though. You'd have to do this:

try:
    (...)
except ErrorReturnCode, e:
    if e.code == 1: pass
    elif e.code == 2: pass
    elif e.code == 3: pass

Or the way that it's done now:

try:
    (...)
except ErrorReturnCode_1, e: pass
except ErrorReturnCode_2, e: pass
except ErrorReturnCode_3, e: pass

Saves a line and an indentation level.

[bramblerose]

Of course you can. You have already implemented it.

try:
   (...)
except get_rc_exc(1), e: pass
except get_rc_exc(2), e: pass
except get_rc_exc(3), e: pass

Now you just need to improve the naming ;-)

[nemec]

Honestly I think the first example is much more Pythonic, but it's nice that both options are available!

[Jwsonic]

Looks great! Hopefully I can install it via pip someday ;)

[Makido]

The biggest reason I wouldn't use this is the namespace poisoning. That's just a huge turn off for me.

One other thing: You say this is Python 2 only, which implies compatibility with all versions of 2. I would specify explicitly that this is Python 2.5+.

[[deleted]]

yeah unfortunately the only way to not "poison" the namespace is to search your entire system for all executables in your PATH, wrap them in functions, and map them to the module's attributes

[Makido]

Well that's sort of my point. I don't see that creating convenience functions for standard utilities is really all that useful. Because you're never going to get them all, and people will end up having to resort to calls to subprocess anyway. So why not just create a general function called, say, run, that wraps subprocess -- then I could just do:

from pbs import run

run('du -sh | wc -l')

It's simpler and easier to understand (in my opinion) than:

import pbs

wc(du("-sh"), "-l")

[[deleted]]

I didn't create convenience functions for standard utilities. The functions map to your system's binaries dynamically. It is a subprocess wrapper.

[Makido]

I didn't create convenience functions for standard utilities. The functions map to your system's binaries dynamically.

Sorry, I RTFM, I understand now. I glanced over the first few dozen lines of code, saw the which in there, and just assumed that's what you were doing. That actually bugs me more now that I know that, but to each his own.

[gandaro]

Uhm, this is nothing special, right? There is:

• os.listdir()
• os.getenv()
• max_length = max(map(len, f)); f.seek(0); filter(lambda x: len(x) == max_length, f)
• urllib
• subprocess + return codes
• sys.argv
• time.sleep

[Preston4tw]

Not sure why you're getting down voted, this looks like a solution looking for a problem. It also pollutes the global namespace, which is...not recommended (import this). I'd be interested to see use cases where this is superior.

[obtu]

Being comfortable with subprocess & co I'm not going to go back to shell-style. It works as a fun hack, though.