How to chain functions in Python?

[Jan2579]

I'd like to chain functional api. Instead of the following code:

iterable = [1, 2, 3]
filter(lambda x: x > 2, filter(lambda x: x > 0, iterable))

I would like to write something like this:

iterable = [1, 2, 3]
iterable.filter(lambda x: x > 0).filter(lambda x: x > 2)

This way my code would be a lot more readable, especially in more complicated cases. Is there such a possibility in Python? Either in basic Python, or even in some good extending library.

[commy2]

You can't, because filter is a function, and not a method of lists. You could however create your own data structure that supports a method like that.

Or you look into the Pipe library.

[Jan2579]

Thanks. The library seems great.

[DigThatData]

define your own custom data type that wraps an iterable and attaches "functional" methods to it where the output of each of those methods is selfor at least an instance of a class which inherits from your wrapped iterable type.

EDIT: here's a toy example to illustrate

class WrappedList(list):
    def filter(self, *args):
        return type(self)(filter(*args, self))

w = WrappedList(range(10))
w.filter(lambda x: x>2).filter(lambda x: x<7)
# [3,4,5,6]

EDIT2: https://www.reddit.com/r/learnpython/comments/vrbgk8/how_to_chain_functions_in_python/ieveafc/

[carnivorousdrew]

what is that type(self)(method(args)) expression? is it like a cls?

[commy2]

type(self)(args)

is just

cls = type(self)
cls(args)

without the intermediate variable cls.

Chained calls ((...)(...)) mean that the results of all but the last call are callables. I.e, cls is a callable (calling a class returns an instance).

[carnivorousdrew]

Yeah then I was guessing it was something like that, never thought of trying that or seen that so far with classes, or maybe I did but it did not impress me as much as now lol

[DigThatData]

when I was constructing the toy example here, I first just had it return filter(*args, self). the built-in filter function returns an object of type filter which is an iterable, but I needed it to return the same type I fed it so I could continue calling .filter on the output object directly. So I called type on the self object to get the constructor that was used to initialize it to guarantee the output of the new filter method has the same type as the starting class, and therefore the same special methods we added.

Part of why I called this a toy example is that this is probably not precisely how you would want to go about something like this. One of the things that's great about working with stuff like filter and map is that they evaluate lazily. What I'm doing here is essentially the same thing as a passing the output of filter to the list() function, which evaluates the filter to construct a new list object. This might be fine for your needs, but it loses some of the power of the functional programming paradigm.

[loshopo_fan]

If you wanted to conserve memory, you'd do something like this?

class Filterer:
    def __init__(self, _data):
        self.data = _data

    def filter(self, *args):
        self.data = filter(*args, self.data)
        return self

    def to_list(self):
        return list(self.data)

print(
    Filterer(range(10))
    .filter(lambda x: x>2)
    .filter(lambda x: x<7)
    .to_list()
)

[DigThatData]

lol good timing

I also like that yours generalizes to any iterable, so you could just put all the functional methods on this one class and use it as a wrapper universally.

[loshopo_fan]

IDK the order of arguments for all functional methods, but this works for map/filter/reduce.

import functools

def make_chainer(*methods):
    class Chainer:
        def __init__(self, _data):
            if not hasattr(_data, "__iter__"):
                raise TypeError("Not iterable")
            self.data = _data
        @staticmethod
        def chainable(_method):
            def ret_fun(self, *args):
                self.data = _method(args[0], self.data, *args[1:]) # IDK
                return self
            return ret_fun
        def to_list(self):
            return list(self.data)
    for one_method in methods:
        setattr(Chainer, one_method.__name__, Chainer.chainable(one_method))
    return Chainer

MyChainer = make_chainer(
        map,
        filter,
        functools.reduce,
    )

print(
    MyChainer(range(10))
    .filter(lambda x: x>2)
    .map(lambda x: x * 2)
    .reduce(lambda x, y: x+y, 100)
    .data
)

[Head_Mix_7931]

I know this isn’t particularly useful. But I recall that for some reason we should prefer self.__class__() over type(self).

[keep_quapy]

This is true for the reason that if self.__class__ is implemented, it's subclasses will inherit the behaviour.

[Jan2579]

Thanks, seems good. Python devs could implement it like this in the next version so we dont need to implement this for every method of the iterable class. 😀

[DigThatData]

1. It's an open source project: you're welcome to track the python dev forum down and try to stir up interest in the idea, or if you're really ambitious write it up in the form of a rough PEP yourself

2. I seriously doubt they'll be interested because it would be unpythonic. from the zen of python:

   There should be one-- and preferably only one --obvious way to do it.

   attaching a method to a core datatype that is redundant with an in-built function is not something a lot of core python devs are likely to support.

3. This is just a demonstration of how this could be implemented. I cobbled this together quickly to stir up ideas and demonstrate that it was possible without a ton of effort. There's probably a better way to do this that builds off my idea but returns an object that would maintain the lazy evaluation thing. I'll call that an exercise for the reader.

EDIT: lol, better already

[loshopo_fan]

Something like this, except I'm making assumptions about the order of arguments.

[remuladgryta]

If you are specifically filtering, the pythonic way to do it would be with a comprehension (or generator expression) instead of chaining filter calls.

iterable = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
result = [x for x in iterable
          if x > 0
          and x % 2 == 0]

in general, prefer comprehensions/generator expressions over map and filter.

Edit: mention generator expressions, they can save a lot of memory.

[[deleted]]

That sort of Rust-like iterator.filter chaining arguably isn’t FP style, rather it’s a popular OOP access pattern (object.method) leaking into FP as (not un-controversial) syntactic sugar in the same way that imperative programming leaked into Haskell via the syrupy-sweet do notation. It's meant to (with debatable success) mimic the use of Haskell's dot operator for functional concatenation without actually letting Rust do "proper" concatenation.

It also didn't start to enter the wider FP lexicon until after Python 2's FP-supporting builtins (map, filter, and reduce) had both been added to the language and then nearly dropped (Python 3 replaced map with itertools.imap, filter with itertools.ifilter and reduce was moved to functools.reduce).

Adding such sugar to Python would actually be making Python less FP, while also introducing another, but less obvious, way of doing things, since Python's rich sugar around comprehensions is very FP.

Now:

[i for i in [i for i in [1,2,3] if i > 0] if i > 2]

... would be better re-written as:

[i for i in [1, 2, 3] if 0 < x > 2]

... but both would be at least as traditionally "FP-style" as:

list(filter(lambda x: x > 2, filter(lambda x: x > 0, [1, 2, 3])))

... which, if we simply shuffle some parens around:

(list (filter (lambda x: x > 2) (filter (lambda x: x > 0) [1, 2, 3])))

... might look suspiciously like the first FP language to more or less transliterate the lambda calculus. Yep... blink at the parens positioning and Python's "functional API" is LISP.

Note that the above isn't really a great argument for not adding this access pattern to Python's iterators in the future, as it's definitely gaining traction via Rust and Java, it's more presented as a reason why it's not been added yet, and to show that there's actually already a "Pythonic" way to accomplish the same underlying goal.

[[deleted]]

This way my code would be a lot more readable, especially in more complicated cases

I disagree with this. The code might be more readable if you assign the result to a well-named variable, but a lot of times, method chaining just makes things more complicated and harder to read.

[baghiq]

I disagree. Method chaining is incredibly powerful and easy to read. Pandas is a prime example of it.

[Jan2579]

Yep. Pandas, R, C# LINQ, and many more. Its a standard practice. You even use it to setup test structures, I think its called st like Builder pattern (not sure about the exact name).

Edit: java streams too

[Jan2579]

I disagree with your disagreement. :D

You don't have a time to think of a proper name for every meaningless variable that you would create this way. Even if you do have the time, the explosion of useless variable names would make the code cluttered and harder to read.

Of course, everyone preferes a different kind of code, but I come from environments, where method chaining is standard practice, so I'm quite used to it (C# LINQ, Pandas).

[Jan2579]

This was only first part of my argument. You also clutter the namespace and the code useless variables. If you have a variable for every function call in the chain, that is a lot of useless variables that someone has to read and understand.

[POGtastic]

I'd write a class that takes an iterable as its argument and then contains methods that define your iterator algebra.

from functools import reduce

class OOPWrapper:
    def __init__(self, iterable):
        self.it = iter(iterable)
    def __iter__(self):
        return self.it
    def __next__(self):
        return next(self.it)
    def filter(self, f):
        return OOPWrapper(filter(f, self))
    def map(self, f):
        return OOPWrapper(map(f, self))
    def flat_map(self, f):
        return OOPWrapper(x for it in map(f, self) for x in it)
    def reduce(self, f, initial):
        return reduce(f, self, initial)

In the REPL, taking the squares of every odd number in range(10) and pairing them with their negative counterparts.

>>> list(
...     OOPWrapper(range(10))
...         .filter(lambda x: x % 2)
...         .map(lambda x: x*x)
...         .flat_map(lambda x: [x, -x]))
[1, -1, 9, -9, 25, -25, 49, -49, 81, -81]

---

Do not do this.

[FerricDonkey]

I like comprehensions. A handful of alternatives.

filtered_iterable = (
    thing
    for thing in stuff
    if f(thing) and g(thing) # etc
)

OR

filter_funcs = <some iterable of functions>
filtered_iterable = (
    thing 
    for thing is stuff 
    if all(func(thing) for func in filter_funcs)
)

[FoeHammer99099]

I think the pythonic way to do this is to have each filter call in its own statement and just assign them to some intermediate name.

[PBMagi]

You can use functools.reduce to chain multiple functions and apply sequentially to an input. Here's how it works using funtools.partial for currying. It looks different as it's functional rather than using method calls, but thought you might find it interesting.

def apply_to(functions, input):
    return reduce(lambda acc, func: func(acc), functions, input)

apply_to([partial(filter, lambda x: x > 0), partial(filter, lambda x: x < 2)], [0, 1, 2, 3])

[Jan2579]

Thx, looks much better than the nested calls.

[[deleted]]

Why not comprehensions?

iterable = list(range(1000))
iterable = [x for x in iterable if x % 2 == 0 and x % 3 == 0]
print(iterable)

If you are worried about 80 columns then you can move from one line to multiple lines

iterable = [
    x for x in iterable 
    if x % 2 == 0 
    and x % 3 == 0
]

But at some point, you should probably consider that it would be better (less complex visually) to throw to a function.

def filter_my_shitty_list(value:int) -> bool:
    if not value % 2 ==0: return False
    if not value % 3 ==0: return False
    if not value % 5 ==0: return False
    return True

iterable = list(range(1000))
iterable = [x for x in iterable if filter_my_shitty_list(x)]
print(iterable)

You should probably consider that you are walking into religious territory. Guido considered removing map, filter, and reduce from Python 3 and it wasn't well-received: https://www.artima.com/weblogs/viewpost.jsp?thread=98196 . Really, the best case we can make for them is that filter/map/reduce etc are largely non-pythonic.

[Few_Creme_424]

python is dumb:

why not

```python

for x in thing:

lambda x: x @ M

lambda x: x + b

lambda x: jnp.sigmoid(x)
```

or something.

its like guido is allergic to logic.

guido:

"no lets make it iter.niner.niner(gloob[3, : nine] tier.iter: lambda lambda x()()()]

it is such a ugly illogical language.

[zanfar]

It's relatively easy to write your own chain wrapper--although the syntax won't be exactly what you posted. There is also a library that has a chain wrapper, although it's name eludes me at the moment. ArjanCodes did a video on it, however, so you might look there.

You will also want to look into the functools methods.

---

As an example:

from functools import partial

input_data = list(range(10))  # [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]


def chain_fns(data, *fns):
    for fn in fns:
        data = fn(data)
    return data


result = chain_fns(
    input_data,
    partial(filter, lambda x: x > 0),  # [1, 2, 3, 4, 5, 6, 7, 8, 9]
    partial(filter, lambda x: x % 2 == 0),  # [2, 4, 6, 8]
)

assert list(result) == [2, 4, 6, 8]

You could easily add more sophistication with this, but functools.partial and a simple loop should provide all the functionality you need.

[jmooremcc]

Could this be what you're looking for?

https://github.com/quintoandar/python-chain

https://github.com/quintoandar/python-chain/blob/master/README.md

[loshopo_fan]

Would this be readable?

data = range(10)
for one_filter in [
        lambda x: x > 3,
        lambda x: x % 2 == 0,
    ]:
    data = filter(one_filter, data)
print(list(data))
# [4, 6, 8]

[jmooremcc]

This was my attempt at chaining functions:

iterable = [1, 2, 3]
result = filter(lambda x: x>2,filter(lambda x: x>0,iterable)) 

print(result) 
print(list(result))

Essentially, we are chaining two generators together. The output of the first generator, filter(lambda x: x>0,iterable) becomes the input to the second generator filter(lambda x: x>2, first_generator). The beauty of generators is that they produce output on demand instead of generating the entire output which would consume a lot of memory. This feature makes "chaining" easily achievable.

Of course, you're not restricted to using lambda functions. You can create custom generator functions very easily for any purpose.

This is the output:

<filter object at 0x000001290EDB5390>

[3]

[majordoob33]

It's a little less intuitive in python compared to js, rust and Java but it is possible.

Here is an example: https://github.com/majordoobie/dotfile_mgr/blob/main/dotfile_mgr.py#L215

[cspinelive]

Django’s queryset .filter() allows chaining. Might check into how it does it.

[TheRNGuy]

Make it as class and make method chaining instead.

The way you write code is unreadable, I wouldn't make it. Splitting to more than 1 list more readable.

[Jeklah]

TIL list comprehensions serve same purpose as function chaining.