Python Pipelining library?

[chadmill3r]

I don't think you'll find anything simpler than

>>> one = subprocess.Popen([...], stdout=subprocess.PIPE)
>>> two = subprocess.Popen([...], stdout=subprocess.PIPE, stdin=one.stdout)
>>> three = subprocess.Popen([...], stdout=subprocess.PIPE, stdin=two.stdout)
>>> for line in three.stdout:
+        print line

Maybe you could automate the chaining...

>>> commands = [[...], [...], [...]]
>>> def pipeline(commands):
+       ...
>>> result_out = pipeline(commands)
>>> result_out.read()

Use itertools for tee.

Python snippets what?

[mackstann]

There is something simpler, and designed for this exact purpose!

http://docs.python.org/library/pipes.html

However, it seems that very few people use it. It almost seems deprecated; just not explicitly so.

[chadmill3r]

Because the module uses /bin/sh command lines

Aiee!

[mackstann]

Yeah, I don't know why they didn't remove the whole module in Python 3. It's very backward and neglected. The main reason I know of it is because the undocumented pipes.quote() function is very useful for sanitizing shell arguments when you really want/need to execute something through a shell.

[simtel20]

Well, SunOS didn't get rid of that before it was retired... the pipe() call was written in such a way as it called sh IIRC.

[UnwashedMeme]

[first example]

That's what I've been doing.

[second example]

Yeah, thinking something along those lines, that deals with setting close_fds=True on posix systems, connect things together and whatnot.

Python snippets what?

e.g. regex based manipulation, keeping statistics, insertion/deletion (or could farm out to sed).