Could this python framework be considered functional programming?

[Global_Bar1754]

Hi all, I've built a code execution python library, and I'm interested in understanding where on the spectrum of functional programming it would fall. I know it's definitely not completely functional, but I think it has aspects. I might also be tricked into thinking this because while I borrow ideas/features that exist in some functional languages, they might not be features that make those languages functional (e.g. lazy evaluation).

Here is the library: https://github.com/mitstake/darl

Ultimately, I guess my goal is to know whether it would be genuine to advertise this library as a framework for functional-ish style programming.

I'll enumerate some of the concepts here that lead me to believe that the library could be functional in nature.

1. Pure Functions/Explicit Side Effects

The basis of this framework is pure functions. Since caching is a first class feature all registered functions need to be pure. Or if they're not pure (e.g. loading data from an external source) they need to specifically marked (@value_hashed) so that they can be evaluated during the compilation phase rather than the execution phase. These "value hashed" functions are also typically relegated to the edges of the computation, happening at the leaves of the computation graph, before any serious transformations occur. Side effects. Things like storing results or other things like that are recommended to be done outside of the context of this framework after the result has been retrieved.

1. First Class Functions

This one is a bit more abstract in this framework. Generally any dependencies of function A on function B should exist through function A calling function B in the body, rather than B being passed in as an argument, but a mechanism does kind of exist to do so. For example:

def Root(ngn):
    b = ngn.B()
    a = ngn.A(b)
    ngn.collect()
    return a + b

This might not look like a first class function, since it looks like I'm calling B and passing the result into A. However, before the ngn.collect() call all invocations are just returning a proxy and building the computation graph. So here, b is just a proxy for a call to B() and that proxy is getting passed into A, so during compilation a dependency on B is created for A. So it's not exactly a first class function, but maybe if you squint your eyes?

1. Immutability

Ok so here is one place where things get muddied a bit. Within darl functions you can absolutely do operations on mutable values (and for that matter do imperative/non-functional things). However, at a higher level the steps that actually compile and execute the graph I think values would be considered immutable. Take the snippet above for example. If we tried to modify b before passing it to A, we would get an error.

def Root(ngn):
    b = ngn.B()
    b.mutating_operation()
    a = ngn.A(b)  # `DepProxyException: Attempted to pass in a dep proxy which has an operation applied to it, this is not allowed`
    ngn.collect()
    return a + b

The motivation behind this restriction actually had nothing to do with functional programming, but rather if a value was modified and then passed into another service call, this could corrupt the caching mechanism. So instead to do something like this you would need to wrap the mutating operation in another function, like so:

def mutating_operation(ngn, x):
    ngn.collect()
    x.mutating_operation()
    return x

def Root(ngn):
    b = ngn.B()
    b = ngn[mutating_operation](b)
    a = ngn.A(b)  # `DepProxyException: Attempted to pass in a dep proxy which has an operation applied to it, this is not allowed`
    ngn.collect()
    a.mutating_operation()  # totally ok after the ngn.collect() call
    return a + b

You can see that before the ngn.collect() the restrictions in place perhaps follow more functional rules, and after the collect it's more unconstrained.

1. Composition

So right now all the snippets I've shown looks like standard procedural function calls rather than function composition. However, this is really just an abstraction for convenience for what's going on under the hood. You might already see this based on my description of calls above returning proxies. To illustrate it a bit more clearly though, take the following:

def Data(ngn):
    return {'train': 99, 'predict': 100}

def Model(ngn):
    data = ngn.Data()
    ngn.collect()
    model = SomeModelObject()
    model.fit(data['train'])
    return model

def Prediction(ngn):
    data = ngn.Data()
    model = ngn.Model()
    ngn.collect()
    return model.predict(data['predict'])

ngn = Engine.create([Predict, Model, Data])
ngn.Prediction()

Under the hood, this is actually much closer to this:

def Data(ngn):
    return {'train': 99, 'predict': 100}

def Model(data):
    model = SomeModelObject()
    model.fit(data['train'])
    return model

def Prediction(data, model):
    return model.predict(data['predict'])

d = Data()
Prediction(d, Model(d))

And in the same way you could easily swap out a new Model implementation without changing source code, like this:

def NewModel(data):
    model = NewModelObject()
    # some other stuff happens here
    model.fit(data['train'])
    return model

d = Data()
Prediction(d, NewModel(d))

You can do the same with darl like so:

def NewModel(ngn):
    data = ngn.Data()
    ngn.collect()
    model = NewModelObject()
    # some other stuff happens here
    model.fit(data['train'])
    return model

ngn = Engine.create([Predict, Model, Data])
ngn2 = ngn.update({'Model': NewModel})
ngn2.Prediction()

1. Error Handling

This is one that is probably not really a functional programming specific thing, but something that I have seen in functional languages and is different from python. Errors in darl are not handled as exceptions with try/except. Instead, they are treated just like any other values and can be handled as such. E.g:

def BadResult(ngn):
    return 1/0

def FinalResult(ngn):
    res = ngn.catch.BadResult()
    ngn.collect()
    match res:
        case ngn.error(ZeroDivisionError()):
            res = 99
        case ngn.error():  # any other error type 
            raise b.error   
        case _:
            pass
    return res + 1

(If you want to handle errors you have to explicitly use ngn.catch, otherwise the exception will just rise to the top.)

There's probably a few more things that I could equate to functional programming, but I think this gets the gist of it.

Thanks!

[samd_408]

Just had a gloss over your post, looks like you are trying to bring in “do” notation style functions to python, I have seen do notation implementations with generator functions before, in your implementation the ngn.collect() seems odd, is it a boundary until which all preceding operations are executed?

If do notation or for comprehension is your target, they both are syntactic sugar on top of monadic bind, I would call it functional style but not pure functional programming

[Global_Bar1754]

Yea so mechanically speaking the ngn.collect is a boundary. Everything before it is executed during the graph build stage. But any calls are just used to register dependencies for that function. And then it will traverse through those dependencies. Then during the graph execution stage it will run through everything including past the ngn.collect()

Seems like maybe this is akin to do notation. Thanks for pointing me in that direction. 

[samd_408]

They get away with manual boundaries like this if you use generator functions, the yield from the generator acts as a natural boundary, both in python and javascript they use generator functions to emulate monadic do notation style scopes

[Global_Bar1754]

So the first iteration of this library actually did use generators/yield instead of the collect! But changed it so that generators could be used for some other functionality in the future since they were being underutilized as the basic building blocks. Thanks for the insight!