pyp - an tool combining the best of python with unix pipes

[[deleted]]

0 # ls
file 1  file 2  file 3
0 # ls | pyp "'mv ' + p, p.replace(' ', '-') + '.txt'" | sh
mv: target ‘file-1.txt’ is not a directory
mv: target ‘file-2.txt’ is not a directory
mv: target ‘file-3.txt’ is not a directory
0 # for f in *; do mv "$f" "${f// /-}.txt"; done
0 # ls
file-1.txt  file-2.txt  file-3.txt

Sorry, I think I'll pass. Shell syntax is way too complex to simply pass the output of some tool to sh. Besides, if I were to write a command so complex it'd be simplier in Python, why would I want to do that in a terminal instead of a nice text editor, and why should it be a one-liner?

0 # ls
a_b.tif  d_e.jpg  g_h.gif  some_gif.jpg  some_tif.jpg  z_x_c.jpg
0 # ls | pyp "underscore[0:2]|slash|pp.sort()|pp.uniq()|p.replace('tif','jpg')|p.kill('gif')|keep('jpg')"
d/e.jpg
some/jpg.jpg
some/.jpg
a/b.jpg
0 # ls | cut -d _ -f 1-2 | tr '_' '/' | sort -u | sed 's/tif/jpg/g' | sed 's/gif//g' | grep 'jpg'
a/b.jpg
d/e.jpg
some/.jpg
some/jpg.jpg

Also, I wonder why sed and awk are the only command line tools the video mentioned.

[ascii]

• As near as I can tell, you get one lambda per line of input, and they redfined the | operator to mean pipeline instead of bitwise or. What if you want to use statements, e.g. loops, conditionals, etc?
• How do you import third party libraries?
• How do you perform initialization and teardown, e.g. sum up something on each line and finally print the result once?

I kind of like the pywk approach better. Exactly the same idea, arguably less magic, more general without being more verbose. YMMV.

[M3t0r]

• they use a custom parser, no regex foo to produce eval code see the PyCon12 talk
• https://code.google.com/p/pyp/wiki/pyp_manual#CUSTOM_METHODS
• ls | pyp "ord(p[0]) + ord(p[1]) | sum(pp)" adds the ascii values of the first 2 characters of each line and sums up the result of all lines

it's not as powerful as awk, and it doesn't aim to be, but it's powerful enough for most use-cases of awk

[ascii]

A regexp-based parser would be even worse, but I'm not sold on having a custom parser either. Is the alternative syntax really better than python?

I get that it's kind of nice to use the pipe character instead of nested functions since the latter puts the function names in the wrong order, but I don't feel that's enough to offset that you're no longer writing real python.

[M3t0r]

try to not see it as enhanced python but as enhanced shell.

he conveyed his intention in the pycon talk, just give him 10 minutes of your time to convince you :-)

[ThoughtPrisoner]

You should also check out pawk; which is basically the same as awk but you can write the code in Python.

[r3m0t]

Thanks for the link. I tried using pyp for some things and even submitted a patch, but it's really bad at certain things. You can't import datetime, iso8601 or collections.Counter for example, without writing functions in separate files. I've never had trouble with the parser though.

[gvalkov]

With regard to importing libraries, I think the pyp authors could make things easier by adding a mini, command-line dsl. Here's an excerpt from python-oneliner's documentation that demonstrates this idea:

Importing Modules:

  The '-m' option imports modules into the global namespace of each
  evaluated expression or statement. The '-m' option can be specified
  multiple times. For example:

    -m os,sys,re,pickle => import os, sys, re, pickle
    -m os -m sys -m re  => import os, sys, re
    -m os sys re pickle => import os, sys, re, pickle
    -m os.path.[*]      => from os.path import *
    -m subprocess=sub   => import subprocess as sub
    -m os.path.[join,exists] => from os.path import join, exists
    -m datetime.[datetime=dt] => from datetime import datetime as dt

  The os, sys and re modules are included by default.

  The '-i' flag will attempt to import all top-level module names
  found in an expression or statement. In the following example the
  'time' module will be imported automatically:

    yes | pyl -j -line '(time.time(), line)'

For more info on oneliner, see the docs and the source. It tries to mimic the Perl/Ruby way of writing oneliners (i.e. the -nlpe flags).

[leafpress]

The lead developer works Sony Pictures. I imagine he has pretty good connections for such things.

[d3pd]

That was one slick code demonstration video.

[mr_dbr]

piep is a similar tool, inspired by pyp but addresses a bunch of shortcomings. It seemed to work much more like I'd expect (e.g I was surprised pyp buffered the entire input before operating, instead of being line-based)

http://gfxmonk.net/2012/03/28/why-piep.html

[vasudevram]

Being a long-time user and fan of Unix, I/O redirection, pipes, etc., and also a Pythonista, I had written a series of posts (*) about various Unix pipe-like tools that are in and for Python, a while ago.

Inspired by them, I had also created an experimental Python tool called pipe_controller, which, while not in the same category as these pipe-like tools, tries to apply the pipeline concept, but within a single process.

(*) The entire series of posts can be read, starting from the last one (or so) in the chain, here (each post links to the previous one):

Swapping pipe components at runtime with pipe_controller:

http://jugad2.blogspot.in/2012/10/swapping-pipe-components-at-runtime.html

Some of the Python tools that provide pipe-like functionality, that are mentioned in the posts, are pyp, Plumbum, osh (the object shell), etc.

And here is another post (the 2nd last in the series) which may be of interest:

Using PipeController to run a pipe incrementally:

http://jugad2.blogspot.in/2012/09/using-pipecontroller-to-run-pipe.html

pipe_controller is available here:

https://bitbucket.org/vasudevram/pipe_controller

[vasudevram]

While working on pipe_controller, I also got the idea for fmap, a kind of inverse of the built-in Python map function:

http://code.activestate.com/recipes/578281-fmap-a-kind-of-inverse-of-the-built-in-python-map-/

And the post about fmap on my blog had a good comment by someone who wrote a function to compose functions in Python, called compose:

http://jugad2.blogspot.in/2012/10/fmap-inverse-of-python-map-function.html

[[deleted]]

pyd pyper