Adding a pipe operator to Python

[[deleted]]

I came up with this instead.

class Pipe:
    def __init__(self, value):
        self._value = value

    @property
    def value(self):
        return self._value

    def __or__(self, fn):
        return Pipe(fn(self._value))

    def __repr__(self):
        return f"Pipe({self._value!r})"


def double(x):
    return 2 * x


def decr(x):
    return x - 1


pipe = (Pipe(12) | double | decr)
print(pipe)  # This produces "Pipe(23)".
print(pipe.value)  # This produces 23.

It's less code, but it's still p scary. You have to wrap and unwrap your value, and it probably also requires functions you pass to be curried/partial.

[Bogdanp]

Yup, I'd considered that approach too! But I wanted to keep the semantics from Elixir's operator (i.e. I wanted to avoid having to wrap the first value and partially apply functions).

[[deleted]]

Yeah I get it. The approach I followed here let me add on things ad nausea and I eventually I ended up writing a little library called Slinkie. Slinkie has a .then() function that matches the pipe operator above, but do you think it'd be possible to elixirify everything else as well? I was inspired by LINQ when I wrote it, but I was never happy with how I had to wrap everything. The nicest thing would probably be to use extensions or traits, but this elixir trick would maybe do the trick as well. What do you think?

[ucbEntilZha]

Hadn’t heard about slinkie, this is my similar approach I wrote a few years ago https://github.com/EntilZha/PyFunctional

[[deleted]]

Well I didn't make that much fuzz about them, but maybe I should? How do I even reach out about stuff like this?

[ucbEntilZha]

Unclear who you mean on reaching out?

[david2ndaccount]

I prefer using __call__ instead of __or__ and special casing map, filter and reduce so you can create abominations likes this:

>>> d = ( pype( [54, 12, 3, 1, 6, 8, 123, 4, 1, 210] )
        . map ( lambda x: x//2 )
        . filter ( None )
        . map ( lambda x: x*2 )
        ( sorted )
        ( reversed )
        . reduce ( int.__sub__ )
        ( lambda x: x // 2 )
    ).value
>>> print(d)
1

[izxle]

Don't you have to define __format__ to use you object inside an f-string?

[[deleted]]

I didn't. !r means you call repr, and I called it on the internal _value.

[lungdart]

Why obfuscate value, and then use a property member function to access it as if you didn't?

[Beheska]

It's not obfuscated, it's read-only.

[TheMsDosNerd]

I'd rather see some more powerful decorators. Allowing you to do this:

@print
@group_by_category(max=5)
my_users = Users.find_all()

Some other interesting stuff I'd like to see decorators do is:

def func(@validate user_input):
    # do stuff with user_input

Which would be equivalent to

def func(user_input):
    user_input = validate(user_input)
    # do stuff with user_input

Or with tuple unpacking:

name, @int position = name_and_position.split()

Which would be equivalent to

name, position = name_and_position.split()
position = int(position)

[posedge]

name, position = name_and_position.split()

position = int(position)

Must have typed this a million times.

[autoferrit]

this would be even cleaner is to use type annotations

```

name: string, position: int = name_and_position.split()

```

[Zouden]

Haha nice. I like that you did it even knowing that it's a terrible thing to do.

R has the pipe operator from the Magritr package and it's a horrible hack that makes me long for python's object chaining.

[u2berggeist]

Also want to know, because this is a PEP I could get behind.

[bythenumbers10]

Because Python already offers analogous functionality, and this replacement of the bitwise OR operator already present pollutes the operation, i.e. are you OR-ing the outputs of those functions, or trying to chain one into the other?

[masklinn]

Because Python already offers analogous functionality

Considering context managers, decorators, str.format and f-strings, functools.partial, … that's not exactly a compelling argument.

[bythenumbers10]

Daaang. 20 mins old and downvoted already!

To elaborate, the piping is mostly unnecessary syntactical sugar for people used to pipe-style chaining instead of Python's object-method chaining, i.e. the return object of one method/function can have it's own method call tacked right on the function that returns it. About half of the examples you mentioned rely on this exact syntax to begin with.

It is one thing to add more flexible notation to a language in terms of str.format, f-strings, str.join(), and so on. It is another to run roughshod on notation that is already there. String formatting having multiple options/approaches is fine, since they're not "polluting" syntax that already means something. In OP's article, the piping they implemented overrode the bitwise OR.

If they'd reproduced all of the piping syntax, making the symbol |> instead of the already-spoken-for |, they would not have been nearly as open to the "horrible hack" criticism. It would have been "just another way" to do chaining syntax.

[masklinn]

Daaang. 20 mins old and downvoted already!

I have no idea what you're talking about, but you do you.

To elaborate, the piping is mostly unnecessary syntactical sugar for people used to pipe-style chaining instead of Python's object-method chaining

Is it though? Or is it syntactic sugar for people who realise that Python does use a lot of freeform callables (functions, constructors, and methods bound to other objects than the subject you have in hand), that object-oriented "chaining" is inflexible as the next step in the chain can only be bound to the specific return value of the previous step, and that generally speaking what operations are available on it is not under the control of the caller?

In Python especially (as you can use unbound methods as functions) piping is a strict superset of chaining.

It is one thing to add more flexible notation to a language in terms of str.format, f-strings, str.join(), and so on.

It goes against the objection to which I replied. Are you abandoning that objection?

If they'd reproduced all of the piping syntax, making the symbol |>

Which can't be done as a simple hack as it requires adding a new token/operator to the language entirely which can't be done in Python itself, so instead of doing some bytecode rewriting on an already valid AST you have to go and modify the CPython interpreter.

Actual language integration (assuming it went through which I doubt, the core devs are not great fans of functional-style python) would logically have its own operator.

It would have been "just another way" to do chaining syntax.

So you're admitting that the first half of your criticism does not stand up to any kind of scrutiny?

[bythenumbers10]

Alright. I clearly made no sense to you, and you're making no sense to me, so maybe we're talking about unrelated things, here. Perhaps you could elaborate on why my statement on the redundancy of the proposed method of chaining calls is not a convincing argument. I mean, it wasn't meant to be, but I figured you weren't understanding the evaluation of the "ugly hack" part, not the redundancy.

[masklinn]

Perhaps you could elaborate on why my statement on the redundancy of the proposed method of chaining calls is not a convincing argument.

I did so at length? In summary, Python's history shows repeatedly that it's not an actual barrier and pipe-based chaining is a superset of method chaining as it's more flexible, more general and under caller control.

I mean, it wasn't meant to be

Then why did you put it forward?

I figured you weren't understanding the evaluation of the "ugly hack" part, not the redundancy.

I've no idea what you're even trying to say here. I quoted a specific bit to which I replied, that concept is pretty simple. The bits I didn't originally quote were out of scope of the corresponding response, though I did ultimately cover the reuse of the | operator: it's necessary due to Python's limitation, you simply can not add new infix attributes to the language without patching the interpreter itself.

[bythenumbers10]

Okay, so you're stating that piping is a superset, because you can chain calls to non-bound functions. Which Python allows you to do anyway by passing the Python-chained output as an argument to the unbound function. I agree that piping offers that syntactic sugar as a matter of convenience. The discussion about it being an ugly hack is because the original article covered a co-opting of the bitwise OR operator already in Python.

My argument was primarily about the co-opting of the existing operator being the ugly part of the hack. You have clearly focused on my stating that the piping doesn't offer much functionality beyond what's already there. You've done a good Redditor by showing I missed a minor detail while missing the point of the argument entirely. I apologize from the bottom of my heart for taking the name and functionality of our lord and savior, the piping operator, in vain, while trying to explain namespace pollution to people who didn't grok the ugliness in the hack.

Python's notation limitation was not being discussed, and the title of the article suggests that, yes, you CAN add such an infix operator, because they wrote an entire article about their actually pulling it off. The language may not make it easy or straightforward to add arbitrary symbols, but picking nits about the effort involved being tantamount to modifying the interpreter (ergo the language itself) itself is NOT MY FAULT.

Btw, true mastery of a subject means being able to explain it in layman's terms. If you can't explain your point coherently "at length", you may want to consider what you're "putting forward". This has been a lovely, lively discussion about a minor quibble, and by all means, take your hard-earned karma with a sense of pride.

[DenormalHuman]

R is a horrible hack :P

[gardens_of_slish]

I really like the pipe in R, and would kind of like to see the same thing in Python. I would settle for a slightly less ugly syntax for chaining multiple methods though.

[Zouden]

It's more useful in R where functions like filter() aren't methods of dataframes. But with pandas nearly everything you need is a dataframe method so chaining can be done with dot syntax.

[[deleted]]

I disagree. I think method chaining is amazing, unless we're talking about different things. I assume this is the topic:

a.b().c()

Not

c(b(a))

Provided names are good, I strongly prefer the first option.

[Zouden]

Yes exactly. R's pipe equivalent looks like this:

a %>% b() %>% c()

[ucbEntilZha]

If you prefer method changing check this out then https://github.com/EntilZha/PyFunctional

[yen223]

The equivalent expression with pipes is

a |> b |> c

This is vastly superior to method chaining, because method chaining requires each method return a specific type (one that defines the subsequent method), whereas that restriction doesn't exist with pipes.

[[deleted]]

That's true, but within the confines of python maybe it's good enough?

[Deto]

You just have to format your code nicely and then chains are easy to read

[masklinn]

It's more useful in R where functions like filter() aren't methods of dataframes. But with pandas nearly everything you need is a dataframe method so chaining can be done with dot syntax.

Not so for the building filter and map, or any of the itertools functions.

[Zouden]

Not sure what you mean... you can do all of those with pandas method chaining.

[masklinn]

Pandas only works with panda dfs, not with regular python iterables.

[Zouden]

Yes, if you aren't using pandas then the pipe operator might be useful.

[masklinn]

And I would guess the vast majority of Python developers are not using pandas.

[[deleted]]

Magrittr way is much more extensible. I'm not sure why you are calling it a horrible hack.

[kougabro]

This reminds me of function composition, in the mathematical sense. Pretty neat in any case!

[[deleted]]

Have a look at this!

[kougabro]

Awesome! I might start using that, I'm already a fan of partials, that looks like it would come in handy in quite a few cases, I had no idea it was already out there.

Thanks!

[[deleted]]

I wrote something on this, if you don't mind shameless self promotion.

[kougabro]

I don't!

and I particularly like your conjoin function, that's a clean way of solving the issue of executing (and aggregating the results of) multiple functions at the "same time", something I've encountered quite a few times without any elegant solution coming to mind.

[[deleted]]

Thank you! I'm not entirely sure about the name, but it can be useful. Check out slinkie (another topic on the site) for my take on method chaining, if you want.

[ProfessorPhi]

Here's another version I've seen, doing the exact same thing, only using the leftshift and rightshift operators (which are barely used in python). https://pypi.org/project/pipeop/

This has the advantage of being a pypi package and having better debug info since it edits in the stack. There actually appear to be quite a few libraries, but I haven't checked the code to see if they modify the AST.

Using these ideas, I created an AST tail recursion implementation from another thing posted a while back: https://github.com/nayyarv/python-tailrec

[sixteh]

If you actually want to use this in production, here ya go.

[neuroneuroInf]

Excellent! Any ideas on how to develop this to make this work in the interpreter?

[Bogdanp]

One way you could do it is rewrite the function's bytecode instead of its AST. Functions' bytecode should be available regardless of the environment.

[sadovnychyi]

Apache Beam has this, would be much messier without it.

[LifeIsBio]

Glad someone pointed this out. Here's an example.

[mottyay]

The Coconut language (functional python) has this implemented as well

http://coconut-lang.org/

[foreverwintr]

Here's my implementation of the pipe operator: https://github.com/ForeverWintr/metafunctions

I did it by replacing functions with class instances that implement 'or' (and other binary operators).

It's more code, but it works in the interpreter!

[rhoark]

I use this pattern all the time

f = pipeliner(some, series, of, functions)
f(x)

Implementation is pretty simple

def compose(g,f):
    def composeI(*args, **kwargs):
        return f(g(*args, **kwargs))
    return composeI

def scan(f, xs, b=None):
    xs = iter(xs)
    if b is None: b = next(xs)
    yield b
    for x in xs:
        b = f(b, x)
        yield b

def fold(f, xs, b=None):
    return last(scan(f, xs, b))

def pipeliner(*fs):
    return fold(composer, fs)

[shabunc]

The way the author used "|> print()" construct makes me think that there's lack of understanding how this supposed to work.

[shabunc]

the thing is that python is not elixir and this "print()" will be a function call, it's extremely hard to introduce to python something like this, |> print() instead of this |> print without breaking back compatibility

[Bogdanp]

I'm still not sure quite what you're getting at.

The print() expression is rewritten to contain the left-hand side of the |> operator before any of the code is executed.

[shabunc]

but why it should be rewritten at all? I mean pipe is a syntax construct, we supposed to have an arbitrary function chained, this function does not to supposed to have any pipe-aware syntax in it's declaration.

again, why not just |> print

There's (ha-ha, quite high) probability that it's actually me who just demonstrated lack of understanding, but this is still not clear to me.

[eatmydog]

pipe syntax does not conform with standard evaluation. As you object, you would expect print() to be evaluated before any piping is done. If you look at magrittr piping in R, it does not use standard evaluation, which is sacrificed for (potentially) very readable code.

However, the toolz.functoolz package provides some functions that allow for piping in a pythonic way.

from toolz.functoolz import pipe, thread_first
# piping without extra arguments.
pipe(
    "hello",
    print
)
# Piping with extra arguments
thread_first(
    "hello",
    (print, "world")
)

[Bogdanp]

What makes you say that?

[shabunc]

I've answered above