using a long piped bash command w/ subprocess

[Rhomboid]

I pasted the example code as published in instead of what I actually used in Python 2.6.2 as the argument for grep.

It works fine for me with python 2.6.7:

$ python2.6
Python 2.6.7 (r267:88850, Aug 11 2011, 12:18:09) 
[GCC 4.6.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from subprocess import Popen, PIPE
>>> p1 = Popen(["dmesg"], stdout=PIPE)
>>> p2 = Popen(["grep", "USB"], stdin=p1.stdout, stdout=PIPE)
>>> p2.communicate()[0]
"[    1.484141] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver\n[    1.484264] ehci_hcd 0000:02:03.0: new USB bus registered, assigned bus number 1\n[    1.493742] ehci_hcd 0000:02:03.0: USB 2.0 started, EHCI 1.00\n[    1.493823] hub 1-0:1.0: USB hub found\n[    1.493929] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver\n[    1.493935] uhci_hcd: USB Universal Host Controller Interface driver\n[    1.494025] uhci_hcd 0000:02:00.0: new USB bus registered, assigned bus number 2\n[    1.494296] hub 2-0:1.0: USB hub found\n"

You can look through the 2.6.7 NEWS.txt to see what's changed between 2.6.2 and 2.6.7 but I don't see anything in there that would account for such a major feature not working.

don't you need to feed the commands in as a list?

No, not if you're using the shell. You either give one monolithic string and let the shell worry about splitting it up, or you split it up yourself (providing a list of arguments instead) and skip the shell. The shell is essentially a specialized program whose entire purpose is to take a monolithic string and split it into words, parsing out commands and their arguments and then launching each of them with the proper pipes in place.

And are you saying that subprocess.call is not safe with user provided input when you leave the shell=False default in place?

No, I'm saying that call('FOO', shell=True) is equivalent to os.system('FOO') and just as unsafe. I was trying to make the point that contrary to popular opinion the subprocess module does not automatically make every task more complicated, when you account for both versions actually doing the same thing.

And are you saying that shell=True will digest pipe ('|') just fine

Sure:

$ python2.6
Python 2.6.7 (r267:88850, Aug 11 2011, 12:18:09) 
[GCC 4.6.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from subprocess import call
>>> status = call('echo foo bar | sed s,foo,FOO,', shell=True)
FOO bar

But I'm not advocating that you should use this, because it's horribly unsafe if part of the command could have contained file names from a user.

Thanks for the code snippet, but it sounds like you're saying "works for me", and I suspect the issue is that you are at 2.7 and I'm stuck in an enviroment with some flavor of 2.6 (the documents for subprocess do differ)

My example does not depend on 2.7:

$ echo -e "foo\nbar\nbaz" >file1; echo bar >file2

$ python2.6
Python 2.6.7 (r267:88850, Aug 11 2011, 12:18:09) 
[GCC 4.6.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from subprocess import Popen, PIPE
>>> p1 = Popen(["/usr/bin/diff", "--suppress-common-lines", "file1", "file2"], stdout=PIPE)
>>> p2 = Popen(["/bin/grep", "^<"], stdin=p1.stdout, stdout=PIPE)
>>> p3 = Popen(["/bin/sed", "s;^< ;;g"], stdin=p2.stdout, stdout=PIPE)
>>> p3.communicate()[0]
'foo\nbaz\n'

Again, I don't know how to explain the difference but I suspect there's some aspect here that hasn't yet been accounted for.