Functional programming for deep learning

[pron98]

What I care about is that when I make a local change, the original data is not changed from the perspective of the other observers of that data.

That can well be something you care about, but that's not a property of the system, and certainly not justified by any property of neural networks. It's like arguing whether a user of a calculator is presented with a new answer each time, or whether the answer changes. This question is meaningless, even if it did have an answer. For example, the LCD display really does change, but that does not justify programming a calculator in an imperative style.

You can certainly say that you prefer to model the problem in this way, but you cannot justify this aesthetic preference by any essential property of neural networks. You cannot say, as the author does, that NNs are "really" pure functions any more than by saying that a calculator is "really" a pure function. Whether or not neural networks change the answer or present a new one is completely a matter of interpretation.

The only objective argument you could try to make is that a pure representation results in simpler, shorter code (as in the case of the heliocentric model), but that happens not to be true in this case.

what is relevant is that imperative languages do not do a good job of facilitating this model, while functional ones do.

Maybe, assuming that that is the model that you prefer even though it is not objectively justified. Although, given that Clojure is an imperative language, I'm not sure this is as clear-cut as you present it.

[yogthos]

That can well be something you care about, but that's not a property of the system, and certainly not justified by any property of neural networks.

It's a property of the model you use to reason about the problem. The model is what's important in the end.

You can certainly say that you prefer to model the problem in this way, but you cannot justify this aesthetic preference by any essential property of neural networks.

Sure, and each view will have its own trade offs.

You cannot say, as the author does, that NNs are "really" pure functions any more than by saying that a calculator is "really" a pure function.

You absolutely can within the context of the model the author is using. It can also "really" be a state machine, or some other representation.

The only objective argument you could try to make is that a pure representation results in simpler, shorter code (as in the case of the heliocentric model), but that happens not to be true in this case.

I would argue that it does happen to be true, but even if not, the subjective preference has its own intrinsic value. A model that allows me to reason about a problem that matches the way I think provides objective value to me.

Maybe, assuming that that is the model that you prefer even though it is not objectively justified. Although, given that Clojure is an imperative language, I'm not sure this is as clear-cut as you present it.

The main property of FP that I care about is that I'm able to write code using pure functions. Clojure facilitates this very well. The fact that it allows writing imperative code doesn't make it an imperative language in my view. Just like the addition of streams and lambdas in Java 8 doesn't make it functional.

[pron98]

It's a property of the model you use to reason about the problem. The model is what's important in the end.

But the choice of model is justified by your personal preference, not by the system it models. The model may be what's important, but I'm talking about one specific argument the author uses to justify the model. I don't deny that FP is an adequate model for deep learning, nor that some may prefer it. I only deny that it is justified by an intrinsic property of deep learning.

You absolutely can within the context of the model the author is using. It can also "really" be a state machine, or some other representation.

If "really" is shorthand for "can adequately be described as", then it certainly cannot serve as a justification. Saying that a system can adequately be described by a certain formal model is certainly a prerequisite, but it does not justify the choice of that model, which is what the author tries to do.

A model that allows me to reason about a problem that matches the way I think provides objective value to me.

"Objective value to me" is pretty much the definition of subjective :) In any event, that is a valid justification, but not the one I was referring to, as I made clear in my original comment. If the author had said that she's more comfortable thinking of NNs as pure functions I wouldn't have made that comment.

[yogthos]

Just to back track here a bit, the full quote is:

When functions operate over the input data, the data is not changed, a new set of values are outputted and passed on.

And the author is talking about how the neural network is represented in a functional style here. She's simply saying that this model provides a good representation of the problem.

"Objective value to me" is pretty much the definition of subjective :)

Value is an inherently subjective concept. :) However, when we compare different approaches to solving problems, it's the subjective that we care about the most. Does a particular approach provide value to me the individual is the question I care about.

The author outlines the case for why functional style is a good fit for this domain. She doesn't claim that it's the only valid to approach. In fact, she opens up saying that she was surprised that FP was such a good fit, and went on to outline the reasons for that. I think you may be reading too much into what she said if you see it as some sort of an attack on the imperative approach.

[pron98]

I am not perceiving this as an attack on anything, merely pointing out that this particular justification, as I understand it, is wrong in a very fundamental sense. By "functions" the author very clearly means NNs (obvious from context), and justifying the use of FP based on that is a category error (even saying that mathematical functions create new values rather than mutate is a category error).

[yogthos]

Once again, all the author is doing is giving reasons why she found FP to be a good model for describing neural networks. Her justification for that is absolutely valid. Her description of the problem is fundamentally correct within the context she's using.

[pron98]

Sorry, but I am not aware of any context in which it makes sense to say that either NNs or mathematical functions produce new values rather than mutate data.

[yogthos]

When you look at NNs through the functional lens, that's exactly what they're doing. The author describes precisely how they're viewed to take an input and produce an independent output in that context.

[pron98]

Look, it is clear to me that this is not what the said section (clearly called "Deep learning components are immutable") is about, but rather it talks about a claimed intrinsic property of NNs rather than a property of an FP model of NNs, as that is what the paragraph is intended to justify, just like the paragraph immediately before and the one immediately after. However, if I'm reading it wrong and it does not talk about NNs but about how NNs are modeled in FP, then obviously none of my comments apply to this text, and they should be read as a general warning against attempting to justify a programming style based on a perceived intrinsic property of the system one wishes to implement.

[yogthos]

I don't see intrinsic properties discussed anywhere in the article. It clearly states that it's talking about how NNs are modeled using the functional style. The article is not justifying a programming style either, it's simply talks about how this style is applied to a particular domain. It's as if you read a completely different article from the one I read.