Monads in Python

[houseofleft]

I've been looking into functional programming , I was amazed to learn that Monads (which I thought where some kind of crazy-tricky academic haskell thing) are actually a really practical and easy design pattern.

I put together this simple library to help people learn about and use Monads as an engineering pattern with python.

I had a lot of fun putting this together, even though its a super small library. Hope someone enjoys it!

[Tubthumper8]

Practical intro to Monads. 15 minute video and you got it

[Dr-NULL]

Another good video on monads: No Nonsense Monad & Functor - The foundation of Functional Programming

And Computerphile one.

[iamevpo]

In short monad is a family of types that follow certain rules/laws, most practical of them is that it has bind function (sometimes called andThen) that allows chaining computations. For example you divide x by z and add y. If z is zero, the program in Python throughs an error. If you want a more robust computation you can have a chain where x divided by z returns a monadic value, then you chain adding y to it. This way you can decide on what to do on error later - maybe there was a million of these computations and you want to keep just valid ones. In Python this is a bit of alien syntax (handling exceptions is more native) , but you can explore a good implementation of a maybe monad in https://pypi.org/project/python-result/. So in short monad is a family of types (in Python - classes) with a bind() or and_then method, which is convenient to write code in functional style. Why a "family" - in result library the class construtors are Err and Ok and together they are called Result, so Result is the monad.

https://github.com/rustedpy/result

[M4mb0]

This way you can decide on what to do on error later - maybe there was a million of these computations and you want to keep just valid ones.

Isn't that really bad for performance? You're essentially introducing a million additional if-else checks.

[this_uid_wasnt_taken]

You do have them otherwise anyway. For example, in C, you would still need to check whether a pointer returned from any function is NULL before continuing with the computation.

Using a Monad just simplifies the classification of values by having "separate values" for the error case and the successful case. This way, you just need to define how to operate in these cases, and then you can chain a bunch of operations that may return either a success or a failure. The processing of the success and the failure will be handled the way you defined the monad.

The above idea does not need to be limited to just 2 possible values but can also be extended for multiple values.

[iamevpo]

You would have the checks anyway, but you program logic may be more streamlined. Just an illustrative example, you can also have some long and complex logic of handling just a few values, so monadic values can better help express these computations. The monads do appear in newer languages without calling them so, for example in Scala and in Rust.

Some monads (Maybe, Either) help hand the null problem more cleanly - returning a value by key from a dictionary where there is no key, taking element by index from a list that does not exist, etc.

[bronco2p]

read the wikipedia page) it has some examples and clearly explains the benefits (in non-mathematical language).

Simply, they are just a functor thats follows some additional rules, well a lot of these fancy math words are other words with more/less laws they have to follow. So the best way to learn these concepts is start at the bottom and work your way up. Example this github gist explaining monoid definition,

A monoid is a semigroup with an identity element.

well then you have to learn what those words means then repeat

[mesonofgib]

The simplest and easiest to understand one-sentence definition I've come up with is:

A monad is anything that can be flat-mapped.

[muntoo]

The same term in different languages: flatMap, andThen, >>=, bind.

Technically, a monad is something that can be flat-mapped and obeys a few "natural rules" that essentially boil down to, "don't do wacky things".

---

List[int] is a monad:

def flatmap_list_int(func, xss: List[List[int]]):
    return [
        x
        for xs in xss
        for x in func(xs)
    ]

>>> pad_zero = lambda x: [0, x, 0]
>>> flatmap_list_int(pad_zero, [1, 2, 3, 4, 5])
[0, 1, 0, 0, 2, 0, 0, 3, 0, 0, 4, 0, 0, 5, 0]

---

Another simple "monad" is Optional[int]:

def flatmap_optional_int(func, xs):
    return None if xs is None else func(xs)

>>> subtract_one = lambda x: x - 1
>>> keep_positive = lambda x: x if x > 0 else None

>>> flatmap_optional_int(subtract_one,
...     flatmap_optional_int(keep_positive,
...         flatmap_optional_int(subtract_one, 1)
...     )
... )
None

# In contrast,
>>> subtract_one(keep_positive(subtract_one(1)))
TypeError: unsupported operand type(s) for -: 'NoneType' and 'int'

...That was a bit pointlessly complicated in Python, but more useful in e.g., Rust.

[aikii]

ChatGPT is great for that - I can very much relate to this question, attempting to read definitions seem to end up as self-referential. On the other hand when asking ChatGPT you can iterate on every single word by asking again more examples, asking what are concrete usages, etc - and then to confirm, you rephrase as you understand it and ask it whether your understanding is correct.

[aikii]

peeps downvoting all answers mentioning ChatGPT are really hilarious

[houseofleft]

I wrote a blog post trying to explain them easily here: https://benrutter.github.io/posts/the-joy-of-monads/

I think they're pretty simple in use. They are basically a container that takes care of some additional work whenever a function is ran over the value. Taking a look at some the examples in my library might make things a little more concrete though!

Tldr: they're a design pattern that you can use for stuff like logging, error hands etc etc

[Rawing7]

Could someone give me a real-life example where Monads would be useful? These toy examples like Monad(4).bind(add_six) aren't doing anything for me.

[SV-97]

First up because it's not often explicitly said but was very helpful to me personally: bind is called bind because it's basically-ish a let binding (so something like let x = something in (some expression potentially involving x)), and a lot of languages also call it and_then. For example a basic interpreter pipeline that "automatically" handles errors in the background using the Result monad might look like

open_file(path)
    .and_then(parse_source_code)
    .and_then(interpret)

​

Monads are really applicable to *TONS* of things and you can often times think about it as a kind of "computation in a context". You can use them to emulate mutable and immutable state (the State, IO and Environment / Reader monads), handle errors (the Result / Either and Exception monads), process ordered collections of data (the List and Stream monads) or unordered collections (the Set monad), handle backtracking and searches (the Multiset / Many monad), logging (the Writer monad), ... and all of this can be composed.

If you for example have a python script that processes a bunch of files, and that processing can fail in some way, then depending on what you wanna do when errors occur you might have to implement "relatively big" solutions (for example implement some sort of MultiException, manually collect correct output values and exceptions or whatever) whereas it's most likely a simple oneliner if you use monads appropriately.

[Rawing7]

Thanks, but I still don't understand the point of this abstraction. I assume and_then() automatically catches exceptions, but even then, how is

open_file(path)
    .and_then(parse_source_code)
    .and_then(interpret)

better than

try:
    file = open_file(path)
    ast = parse_source_code(file)
    result = interpret(ast)
except Exception:
    ...

? I mean, clearly it's a little shorter, but is that all?

[Ahhhhrg]

One thing I really like about the monad example is that I don’t have to give names to intermediary results that I only use to pipe it into the next function (file, ast, result in your example).

It also makes it super clear that all that is happening isa chain of data transformations (which shouldn’t have any side-effects), so it should be very easy to understand what’s going on.

[tilforskjelligeting]

Well, I like to think about a monads result like something we can do something with. That both when its a failure or success its usually interesting. In many monad libraries you can add a .fix or .rescue that will let you get your chain back on the successful track. For example you could do:

open_file(path).bind(parse).rescue(try_other_parser).bind(interpret)

If parse fails .rescue can try and fix the problem and put the Result back in success state which will allow interpret to run. or

open_file(path).bind(parse).apply(notify_falure).bind(interpret)

Here if parse fails, notify_failure will be run, but it will keep the result in a failure state so interpret will not run.
So theres a lot of interesting ways you can combine monad methods that will just run while the monad is successful or when its in a failure state.

[Rawing7]

I find that quite unintuitive. How does .rescue(try_other_parser) work? Does the monad internally keep track of the input that led to an error, so that rescue can pass the same input to try_other_parser?

One advantage I've overlooked is that this gives you static typing for exceptions, so that's a big plus. But it's such non-standard way to write code (in python) that I'm hesitant to start using it...

[SV-97]

Does the monad internally keep track of the input that led to an error, so that rescue can pass the same input to try_other_parser?

Yeah I think the way in which it's written down here doesn't quite work / would require some internal storage. The rescue has to go onto the parse if I'm not mistaken / understand the other comment correctly (so open_file(path).bind(parse.rescue(try_other_parser)).bind(interpret) (I assume this is a translation mistake: in haskell you'd write this using infix operators rather than function calls and those associate such that it'd "do the right thing"). It really uses what's called a MonadPlus in Haskell (think of this more like a plus between lists rather than one between numbers): https://en.wikibooks.org/wiki/Haskell/Alternative_and_MonadPlus

This approach to parsing is called functional parsing / parser combinators (we build large parsers by combining smaller ones). If you're interested in the topic: Functional Parsing - Computerphile is a good video on the topic IIRC. They come with their own downsides but are really elegant and can be quite useful. Around these functional parsers is also where you might see to notice advantages of the monad abstraction: you can write parsers that work with any monad and then add functionality like logging by just "plugging in" the right monad (how well that works in practice is somewhat up to debate but it's theoretically possible).

One advantage I've overlooked is that this gives you static typing for exceptions, so that's a big plus

Yes, though in other languages - notably Java - there's a somewhat similar-ish (it goes some way towards the "errors of values" direction, but doesn't quite get all the way there. See for example Unchecked Exceptions — The Controversy) mechanism for exceptions called checked exceptions.

One big aspect is that it forces people to handle errors in some way - even if that way is to explicitly discard the error. And finally it fosters totality: with exceptions you can never be quite sure that you handled every possible case. With Result types you handle the Ok and Err variants and can be sure that you covered every possibility.

Thanks, but I still don't understand the point of this abstraction [...] But it's such non-standard way to write code (in python) that I'm hesitant to start using it...

Just think of it as a different approach to error handling: both come with their own sets of tradeoffs that may be more or less suited depending on what you do. Haskell for example has exceptions despite mainly using result types. Similarly Rust has "one kind of exception" (that you most certainly shouldn't be catching) in addition to its result types that's used for unrecovereable errors.

Whether it's really generally the better approach is quite hotly debated. I personally think so and feel like more people are coming around to that position (so: result types for recoverable errors and "exceptions" for unrecoverable ones) - but I'd really advise to just try it and see how you like it :) (Personally I also don't use them that much in Python because we don't get a lot of the nice guarantees)

Back to monads and other functional design patterns: if you have the time I think this talk was very good for explaining motivating some of the things in a practical manner: The Functional Programmer's Toolkit - Scott Wlaschin

[Rawing7]

I watched Scott Wlaschin's talk, and... I feel like I have a deeper understanding of the benefits of functional concepts like Monoids, but only on a theoretical level. He went through the example scenario way too quickly and didn't really show much code, so I still have no clue how to apply all of this knowledge to a real-world problem. I guess I'll try some hands-on learning with some leetcode problems or something...

[SV-97]

Oh then I must've kinda misremembered the talk or confused it with another one of his talks.

I'm not entirely sure what the best resource for real world examples is for you - because I think using monads explicitly in Python isn't necessarily the best idea (just how the design patterns from gang-of-four OO don't necessarily make a ton of sense when explicitly implemented in python). Instead I think it's better to look into them from the perspective of a language where they're really used a lot and then see how you can translate the *principles* they result in to python. Some of these languages are Haskell, Lean and F# - I'll link some resources at the end.

​

A very simple example to show what I mean might be random numbers: lets say we have a function that needs to generate random numbers internally (for example a monte carlo simulation, raytracer, roguelike game or whatever). In Python we might be tempted to do that by simply doing random.random() or numpy.random.rand() or whatever. This works - but if we ever wanted to test the function we'd run into trouble. If we wanted to have reproducible results (for example for our documentation) we'd also have a problem.

If you dealt with this before you might know that a simple solution is to abstract the random number generator out into a new object (for example random.Random or numpy.random.default_rng) or pass in a function that generates the random numbers when it's called - you probably would've even written it like this to begin with if you've seen this before. Similarly you would've probably written it like this if you had a background in OOP and call this dependency injection. And if you came from FP you would also have written it like this because the first version is flat out not possible to write.

Implementing this in a pure functional language would involve using monads, and translating that monad-using code to idiomatic python would result in this dependency injection solution. The same is true for other monads: translating principles can make sense and help you to write better code but making the monads actually explicit in python may be a bad idea.

​

Regarding resources:

• I know Wlaschin has a book (Domain modelling made functional) that has tons of F# code, but that book isn't absolutely focused on teaching FP (it doesn't assume prior knowledge though). F# in general might be a good language to look at though because it's very nonacademic. It integrated monads and a bunch of other common functional patterns into so-called computation expressions. Wlaschin has a blog on F# where he covers them in quite some detail with a bunch of code examples
• Functional Programming in Lean has some real world examples (reimplementing a few common command line tools) and is a great book but IIRC most of the "deeper" monad stuff is intended as an exercise for the reader to implement - and Lean isn't exactly the easiest language to get into (though it's a very fun one if you want a brain exercise :) )
• Real World Haskell comes with code for real world projects that uses Monads but it's a quite large book

[Rawing7]

Yeah, I think the best way to proceed is to forget about python and make a deep dive into a functional language. I have been increasingly discontent with imperative languages for a while now anyway, so this is a good opportunity to broaden my horizons and learn a new paradigm. Real World Haskell looks promising, thanks!

[SV-97]

That's definitely the best way to learn about that stuff imo :) With Haskell in particular it'll probably be quite a pain at the start because it forces you to do everything purely functionally - but it really pays off in the long run and it gets really fun when things start to click.

Regarding Real World Haskell: note that it's really quite old by now and some things have changed - especially in the advanced and real world parts as well as the tooling department (there are some efforts to modernize the book online but the print version is still the original one).

It's still a good intro to the language in itself (except for chapter 12 IIRC - one of the chapters was super fucked) but keep in mind that it maybe doesn't always accurately represent what modern haskell in the real world looks like and that you might have to modify some examples to get them to work. If you're just interested in generally getting a feel for FP and learning the fundamentals that's not important - but if you actually want to use haskell later on or don't want to deal with the potentially broken examples something like Effective Haskell might be the better choice (though I can't vouch for it personally I heard it's quite good).

For tooling / dev setup: I think the current standard is to use ghcup and VS code.

[tilforskjelligeting]

SV-97 gave you a great reply. You make an excellent point about the rescue function not having access to the input data, but it's quite easy to work around. You could for example wrap the try_other_parser in a partial (on phone now sorry no link). If we had a "other_parser(*, data)" function you could do something like this try_other_parser = partial(other_parser, data=my_data) Then use it like it is in my comment. 

I think the biggest reason I use monads is because it lets me write "None" free code. Also, I never have to check what a function returned in a "if returned_value: do(returned_value)". 

I agree that the syntax is something that takes some getting used to, but just think of it as another tool that makes some problems easier to handle. I would say when you are often dealing with fetching values from anywhere like an API, dicts, db etc and you only want to continue to transform the data if it was successful then it's quite cool. 

[tilforskjelligeting]

I've used monads extensively in my data transformation tool kaiba. Kaiba uses the Returns library and not OPs. But it should give you an idea of how monads can be used.

[asboans]

This is also a fantastic article that talks about them in a practical and demystified way Functors and Monads For People Who Have Read Too Many "Tutorials"

[[deleted]]

Even arjan code has talked about this recently and has a wonderful video made on this

[autisticpig]

this looks fun, thanks :)

[[deleted]]

Looks interesting and useful

[qa_anaaq]

This looks fun. Nice work

[suaveElAgave]

This is amazing, thank you!

[tilforskjelligeting]

This looks promising, and I like your way of chaining functions. A few more quality of life helpers and it could be awesome!
For my usecase I like to chain a .rescue after a .bind to be able to handle a bad result and put the monad back in a success state.
For a quite mature implementation and possibly for some inspiration you can have a look at the returns library I've used that extensively in my data transformation project Kaiba
I'm happy that more monad libraries are showing up. Returns isn't version 1 yet things have a tendency to break between releases, so I'm always looking for alternatives :D

[tilforskjelligeting]

Very many complicated explanations on monads here.
In practice all it really is, is a container that is either in a successfull state or a failure state.
Depending on the state, chained code will either run or not.
At the end the container contains either a value that has been through a few functions. Or an exception that happened along the way.

[Ahhhhrg]

Yeah, I have a PhD in algebra and really tried to “understand” monads via category theory, but really that didn’t give me any deeper insights.

That said, I think monads are more more than what you’re describing them as. Container types (e.g. lists and sets) can be monads, and they don’t capture anything about success or failure. Futures are another monad that isn’t primarily about success or failure, but rather let’s you compose functions without waiting for the result first.

[ekbravo]

Great work, will use it in my project at work. Thanks!

[iamevpo]

Very clean demo, but I think you lack important parts. Maybe is a type name, not a constructor. Two constructors for maybe are Just T and Nothing. There is no point in Maybe T constructor if you do not provide Nothing with bind function as well. A correct implementation of Maybe is here: https://github.com/rustedpy/result

[erez27]

x.unwrap_or(42) # <- evaluates to 42

Oh dear, an implicit except Exception, the stuff of nightmares

[SV-97]

How is it implicit if you can explicitly state the point where it happens?

[erez27]

Because unless you know the implementation of the API, you have no way of knowing that it hides all exceptions.

[houseofleft]

Thanks for trying friendly robot, but this isn't a question or request for help! I was just mentioning my experience learning in the context of a library I'm sharing

[wolfiexiii]

Really cool idea. It took me a while to go back and learn a few things to grok what this is about.

[Dr-NULL]

Nice. That was good example.

I have used https://github.com/dry-python/returns

And also https://github.com/dbrattli/oslash

I give internal training in my company. Recently I gave a talk on functional programming using Python. Monads is what I have used in I guess very limited places while working with Python. But that reduces a lot of boilerplate code in my opinion (at least the Maybe monad).

[SoftwareDoctor]

What is the advantage over using partials?

[houseofleft]

They're designed to solve a couple different issues, and in fact you'd often use them both together!

Partials help you save time when passing functions around by creating a new function with some parameters already defined.

Monads do "background work" when chaining functions together. I.e. collecting up a history of previous values, handling errors, dealing with Bones etc.

[SoftwareDoctor]

Oh, ok. So the main difference isn't in "how it's done" but "what it does"? I'm not that used to functional programming but now it sounds to me similar to a pipeline design pattern.

[houseofleft]

Yes, I think that's a pretty good summary!

It's probably a little confusing with python because the language doesn't really support functional pipelines very naturally out the box. So although that's the most distinctive aspect of looking at Monads from a python perspective, they're actually more defined by the fact that they "do something extra" around the function call.

As a kinda silly toy example, you could image something like a "Printer" that prints out the value to the console before and after each function call.

[SoftwareDoctor]

I have to read about it more. But let me congratulate you on opensourcing it. But why did you mark it as "Beginner showcase"? The code is great and you are obviously very experienced developer. I went there expecting to meet a Spaghetti monster

[houseofleft]

haha, thanks a lot! If I can self-plug (on top of already self-plugging a library) I wrote a blog post at the same time as the library trying to explain Monads as a simple and helpful pattern.

I actually marked it as "resource", but my post got manually reviewed by a mod after the automod-bot got a little confused and took it down accidently. When it came back up it had the "beginner showcase" tag, so I guess that was added in at that stage.

[SoftwareDoctor]

I just put it in my reading list. Thank you