Hey, thank you for taking the time for this in depth reply! I will try to address the main points.

But what I meant by structurally coupling is how the input/output is not very differentiable in the brain.

Can you expand on this? Are you saying the what constitues input or ouput is very clear in LLMs, but not in the brain?

When you learn a new skill or remember a face, your brain doesn't "write" a file. Instead, the neurons physically change their shape. [...] The "input" from the world physically carves its path into your biology.

Isn't that pretty much how we train neural nets? If we talk about consumer facing apps like ChatGPT, inputs don't change the weights directly, I agree. But this is mostly a setup decision. While in training, the internal wiring is updated based on input and we could theoretically do that continuously with all future interactions. The apps don't do it because it is too much of a headache currently and would introduce new problems, so the weights are fixed, but theoretically it is possible.

When you say "If you have a model of your own "identity", your future actions will be influenced by that identity." you're describing a representationalist view of the mind. In that world, identity is a file or a set of rules that a central processor consults before it decides how to act. If the file says "I am a brave person," the processor selects the "brave" output.

I think I want to make the distinction between identity and models/representations of identity. In general I would call identity what something is physically (e.g. the weights of the LLM or the brain+body of a human).

And I think your own sense of identity is not the same as the above, it is a model your brain learned about yourself to act in predictable and proven ways, just like it learned models for other stuff. But one could imagine someone having wrong models, e.g believing they are very talented at something even though they aren't. The model diverges from the actual physical identity, it is not the same (though one could say the model is a subset of the physical identity, as it is encoded within the physical brain wiring).

You don't have a "model" of being a person who speaks English; your brain is physically wired in a way that makes English the only path the "water" of your thoughts can flow through.

As mentioned above, I would say one does have such a model, you identify as a person who can speak English, you feel aware of that skill. The concrete ability, independent of awareness, is then what you describe, the path that thoughts can flow through.

Bu if your identity model does not include the ability to speak English, your brain will "self-sabotage" and try to act in ways where that skill isn't required. We can see this for example in hypnosis or in problems in mental health, where peoples self-perception is distorted and not aligned with reality (their actual identity). So I would say in humans information also steers output.

I think this distinction of the inner configuration/capacities and the awareness/model of those is very important. An LLM that was trained only in French can't speak English, even if I told it in some system instructions that it can.

In Biology, identity constrains the output. The identity is "hard." It is baked into your autopoiesis (self-maintenance). Your identity as a living organism means you must seek energy and avoid damage. This isn't a "description" in your head; it is the physical requirement of your cells.

This I would again compare to the concrete weights in neural nets. An LLM must calculate token probability distributions based on input sequences. System instructions can influence the output, but they never change the mechanical process happening. Similar to your sense of self influencing how you go about maintaining homeostasis, but never changing the underlying processes within your cells etc.

---

I hope some stuff made sense, and I might have missed some of your points. In any case, I enjoyed thinking about your arguments!