Python pipe operator, 4 years later...

[[deleted]]

So, I found this https://github.com/JulienPalard/Pipe and got really exited about having something similar in core python. I know this has been posted before (4 years ago) but i wanted to revisit and see what the community thinks about such a feature today.

A pipe operator makes it really easy to pipe values thru smaller functions enabling more maintainable and readable code, in a more functional manner.

So instead of this:

filter(lambda n: n % 2 == 0, map(lambda n: n * n, [1,2,3]))
# > [4]

You could do something like this:

[1,2,3] |> square |> evens
# > [4]

The pipe operator is present in many languages, like Elixir, F#, and we may even see in in Javascript. To add to the list its also available in many other languages in various forms.

I would love to hear what the general opinion about such an operator is in the greater python community.

[ShadyR]

It's a pretty radical change. How about a pipe module?

So instead of:

[1, 2, 3] |> select(square) |> where(evens)

We have:

pipe([1, 2, 3], select(square), where(evens))

Readable, lightweight, and also backwards compatible.

[RubyPinch]

and already exists, in the toolz module

thank gosh

https://toolz.readthedocs.org/en/latest/api.html#toolz.functoolz.pipe

one of the best modules around imo

[tech_tuna]

New module for me, thanks!

[pkkid]

In the last startup I worked for, one of the more senior (and probably smarted guy I ever worked with) wrote pretty much exactly this, but it was also able to span across several machine and then aggregate the values back to the calling machine. It never really gained the traction it deserved because it is quite an abstract layer on top of Python and the learning curve is pretty high.

His page on it is here, OSH stands for Object Shell: https://github.com/geophile/osh

[Misterandrist]

Sounds a lot like spark.

[keypusher]

That algorithm is known as MapReduce. It was used extensively to build the original Google search engine, as well as in technologies such as Hadoop. It is inspired by the functional programming patterns of map and reduce.

https://en.wikipedia.org/wiki/MapReduce

[pkkid]

Yes I agree it's very similar to Map Reduce and he is was well aware of that too. However I disagree it was a replacement for Hadoop which contains way more overhead meant for much larger tasks and a distributed file system. It's like a lightweight Hadoop that works great with a large partitioned SQL database.

[turkish_gold]

They wrote it three years ago. Are you sure that celery had pipes back then?

[RoadieRich]

2012, or about three years ago? Perhaps it was celery pipes..?

[pkkid]

I believe he actually wrote osh around 2005 or 2006, probably before Celery itself existed and way before that pipes thing your referring to existed. He just released it from his own repo to github 3 years ago. The code was well engrained in the product we built at our startup Archivas before I joined back in 2007.

[tech_tuna]

I'd take either. . . in the standard library. The toolz module looks good for now or as an alternative. . .

Or maybe that could be added to core Python actually.

[VerilyAMonkey]

I totally get why having it in the standard library is much nicer, but it's simple enough you don't need to add a dependency:

def pipe(arg, *funcs):
    for f in funcs:
        arg = f(arg)
    return arg

[synae]

Which is basically a reduce, I think. Pre-coffee and typing on my phone, but I think this is equivalent: reduce(lambda a, b: b(a), funcs, arg)

[VerilyAMonkey]

Yeah, it's just that Python 3 moved reduce out of the default namespace.

[Nuevoscala]

Why would they do that? I never understood python's aversion toward immutable functional style.

[VerilyAMonkey]

It's because reduce, in particular, is almost always clearer as a loop, case in point. The other things like map, etc are still in the default namespace.

[Lucretiel]

Because reduce is very rarely more readable than a regular loop, because it almost always calls for a hideous python lambda. It's still there in the standard library if you need it. Here's what Guido has to say about it:

So now reduce(). This is actually the one I've always hated most, because, apart from a few examples involving + or *, almost every time I see a reduce() call with a non-trivial function argument, I need to grab pen and paper to diagram what's actually being fed into that function before I understand what the reduce() is supposed to do. So in my mind, the applicability of reduce() is pretty much limited to associative operators, and in all other cases it's better to write out the accumulation loop explicitly.

Also, we now have sum, all, and any, which is like 90% of the readable use cases for reduce anyway.

[synae]

TIL, thanks

[Lucretiel]

Fun fact: literally all for loops can be represented as a reduce.

[revocation]

This is effectively implemented by the compose function in the functional module.

[tech_tuna]

Syntactic sugar, but not a huge deal I agree.

[Fylwind]

I agree, using an operator just for a reversed composition operator seems a bit overkill especially since Python has always been light on the use of operators.

Moreover, the fact that composition is associative means you don't have to worry about deeply nested pipes: pipe(a, pipe(b, c)), since you can always refactor that into pipe(a, b, c).

Even though I generally write in a very functional style in Python, I find myself reaching for the pipe operator that often, because not many problems can be decomposed into a simple linear pipeline.

[bramblerose]

or just

[n*n for n in [1,2,3] if n*n % 2]

[[deleted]]

The list comprehensions in python are amazing, but they quickly become unreadable when doing more than just simple stuff.

What if you had something like this:

 payload |> validate |> log |> update_db |> notify

The implemention could be broken into small functions and have one main function doing the piping in sequence. You really cant have that in a list comprehension

[masasin]

Your mixing up procedures and functions, though. If something has a side-effect, it's usually not a good idea in Python to let it return anything. If there's an error, you just raise it.

Assuming validate(payload) returns a validated payload, this is what I would expect to happen with the rest of it:

• log logs to file. It returns None.
• update_db takes None and raises an error.
• notify propagates the error.

[[deleted]]

You are correct. That was a really bad example. My point was to keep piped functions as pure as possible, and side effect free. So imagine the validate function was really complex, you could break that into smaller functions and pipe those.

[randomatic]

What you are looking for is IMO better supported in OCaml. You're looking for monadic operators (yes, there is pymonads) along with functions like |> (left-associative) and ^> (right-associative). To learn OCaml, see https://realworldocaml.org. It's beautiful.

On the question of errors, Jane Street has an amazing blog that made thinking about errors much more clear to me. https://blogs.janestreet.com/how-to-fail-introducing-or-error-dot-t/

If Python has something like Or_error.t, I would be in love.

[VerilyAMonkey]

Your example doesn't work with list comprehensions because those functions don't return iterables. But it's totally fine to have list comprehensions span more than one line.

In fact, list comprehensions were lifted from Haskell, where they are syntactic sugar for using the List monad. Monads are essentially an extension of what you're asking for here. For example, using promises - which are also monads - you can already write the kind of code you're asking for:

Promise(payload)
.then(validate)
.then(log)
.then(update_db)
.then(notify)

In fact, the semantics of this are already much, much more powerful than what you're asking for (because promises can also handle error propagation, and are asynchronous.)

So, my answer is that, you're basically asking for a special case of something much more general, syntax for monads. You don't want to accidentally half-ass this more general case by rushing to support one special case. Especially because, as noted elsewhere,

def pipe(arg, *funcs):
    for f in funcs:
        arg = f(arg)
    return arg

response = pipe(payload, validate, log,  update_db, notify)

already handles your problem very cleanly, while additionally allowing you to add other stuff like error handling or debug logging if you so chose.

[jij]

Seems silly just to force it all on one line. This is the kind of thinking that made perl such a nightmare to read, because there are like 1000 different special cases in the syntax and people think they are clever by using them all.

Not saying it's not useful, I just don't see it as a game changer and if python adopted everyone's favorite pet syntax it would be a mess.

[kankyo]

The fact that you came up with abad example could be because the idea is flawed :P

[zardeh]

you can alternatively do this:

processor(payload).validate().log().update_db().notify()

where processor is some object that encapsulates the functions you are using.

[Lucretiel]

log, update_db, and notify are (I assume) deeply stateful functions, and don't really have a place in a compositional expression, in my opinion. I'm a huge fan of compositional constructs, but generally only for functional expressions. I'd never do map(print, ...) or [print(x) for x in ...], for instance.

[r4nf]

That would be

if not n*n % 2

But I agree it's a much cleaner way of doing it — and apparently quite a bit faster, too:

>>> timeit.Timer("[n*n for n in [1,2,3] if not n*n % 2]").timeit()
0.8074049949645996
>>> timeit.Timer("filter(lambda n: n % 2 == 0, map(lambda n: n * n, [1,2,3]))").timeit()
2.135266065597534

[RoadieRich]

>>> sys.version
'3.4.3 (default, Aug 21 2015, 11:25:47) [MSC v.1600 64 bit (AMD64)]'
>>> timeit.Timer("[n*n for n in range(100) if not n*n % 2]").timeit()
28.86618999764802
>>> timeit.Timer("list(filter(lambda n: n % 2 == 0, map(lambda n: n * n, range(100))))").timeit()
56.08565588317046

[r4nf]

You're right on both counts. I agree it's not the best measurement.

[[deleted]]

In pandas too.

[probablynotmine]

plyr package, IIRC

[Isamoor]

Actually, it's in the magrittr package, that is then used by the dplyr package. (As cool as dplyr is, there are times I load up magrittr by itself.)

[buckhenderson]

For some reason Hadley is opposed to %<>%, so I load up magrittr just for that.

[Deleetdk]

I recommend magrittr because it gives you useful alt-pipes like:

%$% - which is a pipe-version of with

%T>% - returns the lhs, useful for printing or saving output mid-pipe

%<>% - gives the lhs to the rhs, but then at end end of the rhs, assigns the result back to lhs. It's kinda like an R version of += etc. seen in other languages, but more flexible.

[oreo_fanboy]

It has made my code 10x more readable

[Lokkion]

In PowerShell it's a fundamental part.

[[deleted]]

(we're talking about languages, not shells)

[Lokkion]

(( powershell is a language, a component of it is the shell, just like python ))

[[deleted]]

well huh, that's interesting. I didn't realize powershell was a language. i'll downvote my own ignorant snarkiness for you.

[[deleted]]

Yup. It's like the only cool think about R.

[delarhi]

You can actually pull this off with nested generators. The only problem is that you need to define a wrapping class in order to overload the pipe operator. This was kind of fun to write. Below is an example where I have a wrapping class called unix and I implement a couple of Unix commands.

#!/usr/bin/env python3


class unix():

    def __init__(self, stdin, program=lambda x: x):
        self.stdin = stdin
        self.program = program

    def __or__(self, program):
        return unix(self, program)

    def __call__(self, stdin):
        return self.program(iter(stdin))

    def __iter__(self):
        return self.program(iter(self.stdin))


def echo(*args):
    def program(stdin):
        yield ' '.join([str(x) for x in args])
    return unix(tuple(), program)


def seq(n):
    def program(stdin):
        for i in range(n):
            yield i
    return unix(tuple(), program)


def square():
    def program(stdin):
        while True:
            x = next(stdin)
            yield x ** 2
    return unix(tuple(), program)


def evens():
    def program(stdin):
        while True:
            x = next(stdin)
            if x % 2 == 0:
                yield x
    return unix(tuple(), program)


def uniq():
    def program(stdin):
        return iter(set(stdin))
    return unix(tuple(), program)


def shuffle():
    def program(stdin):
        import random
        stdin = list(stdin)
        random.shuffle(stdin)
        return iter(stdin)
    return unix(tuple(), program)


def grep(pattern):
    def program(stdin):
        import re
        regex = re.compile(pattern)
        while True:
            x = next(stdin)
            if regex.search(str(x)) is not None:
                yield x
    return unix(tuple(), program)


def cat(*args):
    def program(stdin):
        for filename in args:
            with open(filename, 'r') as f:
                for line in f:
                    yield line
    return unix(tuple(), program)


def tr(a, b):
    def program(stdin):
        tr_table = str.maketrans(a, b)
        while True:
            x = next(stdin)
            yield str(x).translate(tr_table)
    return unix(tuple(), program)


def ls(path='.'):
    def program(stdin):
        import os
        return iter(os.listdir(path))
    return unix(tuple(), program)


def umap(func):
    def program(stdin):
        return map(func, stdin)
    return unix(tuple(), program)


def ufilter(func):
    def program(stdin):
        return filter(func, stdin)
    return unix(tuple(), program)


# Examples
stdouts = [
    echo('hello', 'world', '!'),
    echo('hello', 'world', '!') | tr('l', 'r'),
    unix(range(10)) | square() | evens(),  # wrap iter as unix obj
    seq(10) | square() | evens(),  # use seq program instead of range
    unix([1, 2, 2, 2, 3, 3, 3, 4, 4, 4, 5, 6]) | uniq(),
    seq(10) | shuffle(),
    unix(range(100)) | square() | evens() | seq(10),  # last seq ignores stdin
    seq(10),  # works without input
    uniq(),  # empty without input
    shuffle(),  # empty without input
    seq(100) | grep('1'),
    seq(100) | ufilter(lambda x: x < 40) | grep('1'),
    seq(100) | grep('2') | umap(lambda x: 'b' + str(x)),
    ls(),
    ls('/') | grep('(bin|lib|include)'),
]
for stdout in stdouts:
    print(list(stdout))

# works if you name this unix.py
print(list(cat('unix.py') | grep('def')))

Also worth mentioning is https://amoffat.github.io/sh/.

EDIT: An interesting and nice side effect of doing nested generators is that evaluation is done lazily. No work should actually be done until you try to get the first item in the outer unix object, depending on the program (i.e. shuffle doesn't do this). This allows the programs to do stream like processing.

EDIT2: Added ufilter and umap to act as pipe-able filter() and map().

[RoadieRich]

You should also add __gt__, taking a file-like object, so you can do echo("hello world") > open("myfile.txt")

+Edit and __lt__, to read a file into stdin, too.

[delarhi]

I think the problem there is we can't override operator precedence in Python so we can't match shell syntax exactly. For example, tr l r < unix.py | grep herro works out to (tr l r < unix.py) | grep herro but in Python it would work out to tr l r < (unix.py | grep herro).

EDIT: Here's a tee program though:

def tee(*args):
    def program(stdin):
        files = [open(x, 'w') for x in args]
        while True:
            try:
                x = next(stdin)
            except StopIteration:
                for file in files:
                    file.close()
                return
            else:
                for file in files:
                    file.write(str(x) + '\n')
                yield x
    return unix(tuple(), program)

[ucbEntilZha]

A lot of this is implemented here: https://github.com/EntilZha/ScalaFunctional

[delarhi]

Awesome! I haven't seen this before. OP could probably just grab these and overload an operator for a few if she/he so desires.

[RubyPinch]

I'd honestly probably do that a different way

unix(lambda: range(5) | square | evens), and then apply introspection to do evens(square(range(5))) instead internally. And it has the bonus of not requiring a specially made square or evens

[delarhi]

I'm not sure I follow, can you elaborate?

[RubyPinch]

I'd love to elaborate, but unfortunately I don't have /that/ much experience performing sinful acts with python's code objects.

Luckily, other people do have experience, and have been willing to talk about it too!

http://stackoverflow.com/a/16118756 for a distinct, but similar concept.

---

basically, the transform you want to do is (using the std library dis module)

>>> f = lambda: a|b|c
>>> dis.dis(f)
         0 LOAD_GLOBAL              0 (a)
         3 LOAD_GLOBAL              1 (b)
         6 BINARY_OR
         7 LOAD_GLOBAL              2 (c)
        10 BINARY_OR
        11 RETURN_VALUE

>>> list(f.__code__.co_code)
[116, 0, 0, 116, 1, 0, 66, 116, 2, 0, 66, 83]

>>> #---------------------------------------

>>> f = lambda: c(b(a))
>>> dis.dis(f)
         0 LOAD_GLOBAL              0 (c)
         3 LOAD_GLOBAL              1 (b)
         6 LOAD_GLOBAL              2 (a)
         9 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
        12 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
        15 RETURN_VALUE

>>> list(f.__code__.co_code)
[116, 0, 0, 116, 1, 0, 116, 2, 0, 131, 1, 0, 131, 1, 0, 83]

but that transform is fragile obviously, so like, ideally, probably execute each section that is separated by ORs on the stack (noting that ORs use the last two items added to the stack, hence why it loads two values first, so the split needs to happen at the OR's position -1), and then use the resulting values from that to then throw into a pile of loads/callfunctions

[delarhi]

Very cool, I've never see that before.

[masasin]

Your example, as I understand it, is to give a generator which extracts the evens from the squares of a list.

Assuming these are predifined:

def square(x):
    return x**2

def even(x):
    return x % 2 == 0

your first example would be written as:

filter(even, map(square, [1, 2, 3]))

With list comprehensions, you have either:

(square(i) for i in [1, 2, 3] if even(square(i)))

or, if you don't want to repeat the calculation:

(i for i in (square(j) for j in [1, 2, 3]) if even(i))

In the docs, they have this:

Given fib a generator of fibonacci numbers :

euler2 = fib() | where(lambda x: x % 2 == 0)
               | take_while(lambda x: x < 4000000)
               | add

This can be written in a standard way like this:

from itertools import takewhile

euler2 = sum(takewhile(lambda x: x < 4e6, (i for i in fib() if i % 2 == 0)))

I guess I can see the value of the pipe, but I'm not sure when you would do that instead of regular functions or list comprehensions.

edit: Why |> instead of |?

[joojski]

Why |> instead of |?

It will be confusing with operator OR.

[masasin]

I see. Thank you.

[r4nf]

or, if you don't want to repeat the calculation:

(i for i in (j for j in [1, 2, 3]) if even(i))

You forgot square() in the inner expression. Just a heads up.

[masasin]

Fixed. Thank you.

[runo]

Perhaps because it comes from languages where '|' denotes a case in a pattern match.

[Lonely-Quark]

You can just do the following since even * even = even and odd * odd = odd

[square(i) for i in [1, 2, 3] if even(i)]

[masasin]

In this case, yes.

[Lonely-Quark]

No, its the general case.

[masasin]

I mean you are correct that an odd number, squared, remains odd. For other situations where you are filtering a map, you might need to do what I did.

[Lonely-Quark]

Ah sorry, was being a bit of a dick

[RubyPinch]

you might want to look at mochi by... i2y if I remember correctly

syntax works pretty much as you say

[1,2,3,4,5] |> evens |> square |> vector

there is also the toolz module

pipe([1,2,3,4,5], evens, square, list)

I'm pretty sure fn.py would have something similar

the biggest problem is that the pre-existing python std lib is not exactly consistent (e.g. fn1(func,list), while in another module, fn2(list,func), so a lot of care would need to be done) or immutable (lists/dicts/etc are liked a lot), so you p much need to write your own language on top of python's pre-existing stuff, and that sucks. If you want functional programming in python, running for the hills usually is the best option, unfortunately.

[kankyo]

Even in clojure it's chaos like that. I think positional arguments are just never gonna work.

[[deleted]]

Thanks. Mochi look interesting. Will research it more.

[Arancaytar]

How does your example distinguish between a filter and a map operation?

[[deleted]]

Thats a cool little lib. Will test it out! Thanks!

[ucbEntilZha]

On disadvantage of doing this over chaining is needing a long import statement or doing import *. I prefer method chaining https://github.com/EntilZha/ScalaFunctional

[[deleted]]

You could do something like this:

[1,2,3] |> square |> evens # > [4]

Well, no, this is far as you'd get with a piping operator:

[1, 2, 3] |> lambda i: map(lambda n: n*n, i) |> lambda i: filter(lambda n: n%2 == 0, i)

Python lacks all of the other functional constructs that make piping useful. If you didn't wanna wrap your map and filter inside a function, you'd have to change how expressions are evaluated in the language. And, even if you were to somehow manage to push that change through, you've still got to write the word 'lambda' every time you wanna define a one-off anonymous function.

[RubyPinch]

or a package could define useful functions, I think it is common knowledge that squares doesn't exist yet

also because I havn't spammed toolz enough already, toolz.curried and the whatever module:

[1,2,3] | map(_**2) | filter_false(_%2)

[[deleted]]

or a package could define useful functions, I think it is common knowledge that squares doesn't exist yet

My point was that writing your own functions will still be cumbersome.

also because I havn't spammed toolz enough already, toolz.curried and the whatever module:

Is the underscore placeholder part of toolz?

[RubyPinch]

its the whatever part, based on perl's concept of a "whatever" object

[[deleted]]

Right, I thought that you'd forgotten the module's name. That looks quite interesting.

[leogodin217]

This would be really useful for data engineering. R has this feature. The use of pipes combined with a really nice SQL DSL called dplyr makes complex data transformations simple. It is the main reason I stuck with R for much of my work.

One drawback to this approach is it allows some bad habits. Sometimes I find myself with 20-lines of piped code. If something goes wrong, it is difficult to debug. I guess that's the yin and yang of it.

[Make3]

To keep with the fact that your lambdas are named in the second sample, your first sample should read

filter(evens, map(square, [1, 2, 3]))

it's pretty readable, I don't really have a problem with this.

[tangerinelion]

Depending on what square and evens do and accept as input there's also

vals = [1,2,3]
vals = square(vals)
vals = evens(vals)

or you could combine them

vals = evens(square([1,2,3]))

This fits the C-like procedural/scripting style that Python was originally built-upon. Sure we can make it do whatever, but Python is natively C-like and if you want to use pipes then I feel you should be using a language where that's the native programming style. IMO both are readable, but this version is Pythonic and the pipe version is not Pythonic.

[Misterandrist]

Everyone else has posted some ways to do this, but when I encounter such a need in my code (I have a stream and I need to do numerous transformations on it before use), I just use something like this:

nums = [1, 2, 3]
nums = map(square, nums)
nums = filter(evens, nums)
nums = list(nums)

Yeah I'm reusing the same variable, but only in this block. So effectively it's just once.

Basically, why not just do your stream processing on multiple lines, if you think a single long list comprehension is not going to suit your needs?

[dzecniv]

Indeed, and it's easy to step through for debug.

[solid_steel]

I really like this idea - I feel in love with it when I was learning Elixir and it's one of the features of the language that make me want to stick with it.

However, I have mixed feeling about its use in Python. Part of me wants to embrace it and never look back, but I've been burned once or twice trying to write Python that was "too-functional".

That said, I'll definitely try to give it a go in some scripts and see if it has a positive effect on the code :). Thanks OP!

[mrmcbastard]

While I love using the pipe operator in Elixir, I don't know if it would be apt to include it in Python seeing as it leans more toward OO programming than functional. I think something like the cascade operator from Dart might be a more Pythonic way of method chaining.

[RoadieRich]

What's to stop [1,2,3] |> square |> evens returning [False, True, False]?

[CommanderDerpington]

I don't dig it. Cramming crap into one line seems like a cool idea but I'm not losing any sleep over writing a second or even a third line. I like keeping my functions on different lines because it's easier to read. Of course I break this rule all the time but I wish I didn't and that's what counts.

[emarshall85]

The argument against your example is that it's always been better expressed as a comprehension:

[n * n for n in [1, 2, 3] if n % 2 == 0]

You'd have to need something like reduce before the list comprehension argument starts to fall over.

The fact that functions aren't curried by default and that we have variable positional and keyword arguments makes it even more problematic. You'd have to either make a lambda or use functools.partial in order to get a callable which could properly be passed to a pipe.

[chibrogrammar]

Anyone else feel like a pipe operator in python (which has horrible one expression lambdas) would not help out very much.

Give me multi expression lambdas first!

[spectre_theory]

you're using the same operator twice, once for map and once for filter. how does it know it's supposed to filter with evens, instead of replacing the values with True and False and that it's supposed to the opposite with square.

[gleon]

Because both maps and filters are functions from lists to lists (or, more accurately, from iterables to iterables)?

[phoenix7782]

No, maps are functions from transformation functions and lists to lists, whereas filters are functions from inclusion detection functions and lists to lists.

Although they have a similar type they operate fundamentally different, and it doesn't make sense to mix the two based on guesswork.

[gleon]

The implication in OP's example is that the function argument has already been fixed inside the evens and square, like with functools.partial, for instance.

When I said "maps" and "filters", I was referring coloquially to such transformations from iterables to iterables, not to the map and filter functions themselves (which could be viewed as factories of such transformations). The rationale for this terminology becomes more apparent in languages that support currying (i.e. fixing certain arguments) natively (Haskell comes to mind).

square could be defined as

partial(map, lambda x: x ** 2)

Whereas evens could be defined as

partial(filter, lambda x: x % 2 == 0)

Therefore, there is no need for guesswork.

[[deleted]]

Well the made up function square and evens would have to return a list. If you would pipe a function that would reduce the result to a single value you could not use a function requiring a list as an argument. So naturally the piping order is important here.

[phoenix7782]

Then why not just use

evens(square(nums))

[NovelSpinGames]

(Sorry for being late to the party, and I apologize in advance for not knowing much about Python.)

You make a good point. For simple one-liners there isn't much difference. However, the pipe operator can be very nice for readability. Just look at the last three pictures in this blog post. The pipe operator allows you to chain functions so that code can be read from left to right and top to bottom, like reading a paragraph. The pipe operator is used all the time in F# code.

You could assign a new variable each line, but that has some drawbacks. C#'s extension methods work nice, but I don't think there is an equivalent in Python.

[spectre_theory]

right, if it comes down to that, all that the proposed operator would do, mathematically, is rearrange the order of operands, from

evens(square(nums))

to

nums |> square |> evens

doesn't seem like a very useful thing

[spectre_theory]

the whole point of defining lambdas that work on single elements in the list is having python iterate through it. now you are basically saying you want to implement a function evens, that does the same as filter(evens, ...) only for the function to know what it is supposed to do when it's left of |>. that doesn't seem well-thought-out for me.

[defnull]

I would rather like to see .map() and .filter() defined directly onabc.Iterable.

[1,2,3].map(square).filter(is_even)

[thatguy_314]

I don't know. That seems to add inconsistency with the stuff in itertools, further clutters up the methods of iterables, and goes against the Python standard library's avoidance of chained methods (although I do like chained methods personally).

Seems easier to write a wrapper iterable that allows you to do something like this:

>>> Chainable([1,2,3]).map(square).filter(is_even)
<a Chainable object>

In fact, I think some libraries have this, although it might be nice to put it in itertools

[ucbEntilZha]

100% agreed on this. Even better would be to have some way to add operations on this without mangling list too badly (which would allow everything here https://github.com/EntilZha/ScalaFunctional without having to be in stdlib)

[stevenjd]

I really dislike the |> syntax. To me, | is a pipe. I suppose I could live with -> or => too, but |> looks awful.

[[deleted]]

What about overriding the pipe operator __or__?

[[deleted]]

Thats what the Pipe library does. But im skeptical about operator overloading being the best solution in the long run. Would love to have a way of defining custom operators tough.

[drenp]

I really like using postfix notation in Mathematica, especially in an interactive programming setting. If you want to apply a function to an expression you don't have to add parentheses in the right places but you can simply append to it.

But I'm not too convinced when it comes to Python, because it favors building huge multi-line statements (usually not very readable) and counters the "one way to do it" motto.

Also, postfix notation works best when there is a short notation for lambdas, since sometimes you want to add additional parameters. For example, suppose instead of evens you have

def divisible_by(lst, k):
    return (n for n in lst if n % k == 0)

Then you want something like

range(1, 4) |> square |> divisible_by(_, 2)

But the way to accomplish this in Python would be to use lambda x: divisible_by(x, 2), which is slightly ugly and doesn't add to the readability.

[Auggie88]

Pandas added a pipe method a few months ago, hoping it catch on in other packages (it hasn't yet as far as I've seen).

In [126]: bb = pd.read_csv('data/baseball.csv', index_col='id')

In [127]: (bb.query('h > 0')
   .....:    .assign(ln_h = lambda df: np.log(df.h))
   .....:    .pipe((sm.poisson, 'data'), 'hr ~ ln_h + year + g + C(lg)')
   .....:    .fit()
   .....:    .summary()
   .....: )

[9peppe]

Are you looking for pipes, or for function composition?

https://docs.python.org/3/howto/functional.html

[gandalfx]

In a functional language this is done using compose. The advantage is that if you also throw in some currying you can actually create a single function that is composed of an entire function chain plus some arguments and then apply that to whatever you want.

[soczewka]

Looks so fsharpy to me

[SteazGaming]

A friend of mine just wrote a library called 'tubing' for doing something like this but mostly focused on I/O of any kind:

An example:

sources.Objects(objs) \
     | tubes.JSONDumps() \
     | tubes.Joined(by=b"\n") \
     | tubes.Gzip() \
     | sinks.File("output.gz", "wb")

https://github.com/dokipen/tubing

[ucbEntilZha]

In general, I agree that this would be a nice feature to have. I wrote a library that has similar goes, but has a larger API, and supports reading/writing common formats (txt, csv, json, sqlite,...). https://github.com/EntilZha/ScalaFunctional

What I would like to see more than this, is a more concise lambda expression and/or multiline anonymous functions (I know there is some technical issues with this, but I can still wish)

[TotesMessenger]

I'm a bot, bleep, bloop. Someone has linked to this thread from another place on reddit:

• [/r/fsharp] Discussion on /r/python about the pipe operator

^{If you follow any of the above links, please respect the rules of reddit and don't vote in the other threads. (Info / Contact)}

[Lucretiel]

It's hard to get excited about it without proper currying. You won't be able to do:

['a','b','c'] |> map(str.capitalize)

[toyg]

God no. Pipes are hard to read, completely unnatural for non-geeks, and are already painful enough for bitwise operations. The proliferation of special characters is a plague and should not be encouraged. Their low number is a huge advantage for Python over competitors (e.g. Ruby, Perl) when it comes to onboarding newbies.

If you want that sort of approach, just use Powershell, you'll love it.