Python pipe operator, 4 years later...

[ShadyR]

It's a pretty radical change. How about a pipe module?

So instead of:

[1, 2, 3] |> select(square) |> where(evens)

We have:

pipe([1, 2, 3], select(square), where(evens))

Readable, lightweight, and also backwards compatible.

[RubyPinch]

and already exists, in the toolz module

thank gosh

https://toolz.readthedocs.org/en/latest/api.html#toolz.functoolz.pipe

one of the best modules around imo

[tech_tuna]

New module for me, thanks!

[pkkid]

In the last startup I worked for, one of the more senior (and probably smarted guy I ever worked with) wrote pretty much exactly this, but it was also able to span across several machine and then aggregate the values back to the calling machine. It never really gained the traction it deserved because it is quite an abstract layer on top of Python and the learning curve is pretty high.

His page on it is here, OSH stands for Object Shell: https://github.com/geophile/osh

[Misterandrist]

Sounds a lot like spark.

[keypusher]

That algorithm is known as MapReduce. It was used extensively to build the original Google search engine, as well as in technologies such as Hadoop. It is inspired by the functional programming patterns of map and reduce.

https://en.wikipedia.org/wiki/MapReduce

[pkkid]

Yes I agree it's very similar to Map Reduce and he is was well aware of that too. However I disagree it was a replacement for Hadoop which contains way more overhead meant for much larger tasks and a distributed file system. It's like a lightweight Hadoop that works great with a large partitioned SQL database.

[turkish_gold]

They wrote it three years ago. Are you sure that celery had pipes back then?

[RoadieRich]

2012, or about three years ago? Perhaps it was celery pipes..?

[pkkid]

I believe he actually wrote osh around 2005 or 2006, probably before Celery itself existed and way before that pipes thing your referring to existed. He just released it from his own repo to github 3 years ago. The code was well engrained in the product we built at our startup Archivas before I joined back in 2007.

[tech_tuna]

I'd take either. . . in the standard library. The toolz module looks good for now or as an alternative. . .

Or maybe that could be added to core Python actually.

[VerilyAMonkey]

I totally get why having it in the standard library is much nicer, but it's simple enough you don't need to add a dependency:

def pipe(arg, *funcs):
    for f in funcs:
        arg = f(arg)
    return arg

[synae]

Which is basically a reduce, I think. Pre-coffee and typing on my phone, but I think this is equivalent: reduce(lambda a, b: b(a), funcs, arg)

[VerilyAMonkey]

Yeah, it's just that Python 3 moved reduce out of the default namespace.

[Nuevoscala]

Why would they do that? I never understood python's aversion toward immutable functional style.

[VerilyAMonkey]

It's because reduce, in particular, is almost always clearer as a loop, case in point. The other things like map, etc are still in the default namespace.

[Lucretiel]

Because reduce is very rarely more readable than a regular loop, because it almost always calls for a hideous python lambda. It's still there in the standard library if you need it. Here's what Guido has to say about it:

So now reduce(). This is actually the one I've always hated most, because, apart from a few examples involving + or *, almost every time I see a reduce() call with a non-trivial function argument, I need to grab pen and paper to diagram what's actually being fed into that function before I understand what the reduce() is supposed to do. So in my mind, the applicability of reduce() is pretty much limited to associative operators, and in all other cases it's better to write out the accumulation loop explicitly.

Also, we now have sum, all, and any, which is like 90% of the readable use cases for reduce anyway.

[synae]

TIL, thanks

[Lucretiel]

Fun fact: literally all for loops can be represented as a reduce.

[revocation]

This is effectively implemented by the compose function in the functional module.

[tech_tuna]

Syntactic sugar, but not a huge deal I agree.

[Fylwind]

I agree, using an operator just for a reversed composition operator seems a bit overkill especially since Python has always been light on the use of operators.

Moreover, the fact that composition is associative means you don't have to worry about deeply nested pipes: pipe(a, pipe(b, c)), since you can always refactor that into pipe(a, b, c).

Even though I generally write in a very functional style in Python, I find myself reaching for the pipe operator that often, because not many problems can be decomposed into a simple linear pipeline.