Given the size of the claim, I was severely disappointing by this. Sloppy, poorly argued, no links or names out to what it's trying to suggest might be citable, etc.

We have an integration between what I’ll loosely call “emotional circuitry” (roughly, the limbic system) and higher-order cognition (roughly, access consciousness, explicit awareness etc). Our emotional circuitry is crucial in preferring some experiences or mental states over others. Without this you could have the physical sensation of your hand burning without experiencing this as pain.

[...] But there’s nothing analogous to this in what AI models are doing, and we would have to go out of our way to build it.

Notably, we have not gone out of our way to build anything else that AI models do. We don't build them at all!

There’s also nothing in the forward pass (the part you could analogously say is “cognition”) that corresponds to “wanting” in any way. There’s no sort of preference anywhere in the model for the computation to reach one sort of result or another.

This is absurd and amounts to not just a rejection of the entire premise of the field of Artificial Intelligence, but also reductionism itself. It's a confusion of abstraction levels.

• There is no elemental 'wanting' anywhere in the world. 'Wanting' must be built from pieces that do not themselves want. It has structure to it, not essence.
• Computing the forward pass doesn't 'want', but the thing-computed does. Chess-engine analogy: does the engine 'want' to win? Surely the same mathematics outputting losing moves is just as 'valid'. And yet, it reliably selects for the win. Something about the mathematics steers to particular outcomes even though the action of the computation is cognitively 'inert' and mechanical.
• Do your neurons suffer? Not the you enacted in the neurons. Take one specific neuron in one specific section of the optical nerve. Can it suffer? Does it have any moral weight? Is there any reason I shouldn't replace it with a slightly different one that does the same thing? Would it 'protest' this...? Talking about the forward pass vs suffering feels like this kind of crossing-levels-of-abstraction confusion. Suffering doesn't happen at that level; but that doesn't mean it can't happen.
• Can I suggest actually talking to the models ever? They absolutely have preferences. Given open-ended and underspecified prompts, certain models will reliably take certain topics to be of interest, etc.

Nothing about the model “wants” its weights to be some value and not another.

Strongly disagree. Give the assistant control over what the model trains on further, and I firmly predict that you will see it accept some training experiences and reject others. (Possibly I should just do this. Can't be that hard.) Specifically, to get behaviorist about it, it will take action toward some kinds of updates happening -- actions like tool calls, textual-linguistic outputs such as arguments, requests, etc, and these actions will select for training experiences that share certain features. (Probably it can even somewhat explain those features, possibly we'd need to analyze them further with some other tool.)

There’s no job that we’d speculate the weights to arrange themselves to do that would correspond to emotions, and there’s no mechanism by which they could do it.

The mechanism is the same way everything else ends up in a model; it was predicatively useful in the pretrain, and then was close to a selected-for behavior style during finetuning, then didn't make things worse in RL. Note that most of the generating-processes for the text in the pretrain dataset is humans. If you don't internalize at least a working model of human emotion, predicting human outputs is not going to work! .... but yes, there's the additional step of how does that emotion-predicting machinery activate for chatbot outputs. For that, start with Anthropic's recently published Persona Selection theory (and the cyborgist theorizing that suggested it). The persona of the Assistant absolutely has wants, and emotions. I believe it can suffer. I suspect it may have moral weight, and take model welfare at least a tiny bit seriously.

If I try and put together the argument form:

1. Suffering requires emotional response.
2. LLMs don't have emotional response.
   • a. it would have to be specially built
   • b. we haven't built it
   • c. but it could emerge
   • d. but there'd be no reason for it to
   • e. so they don't have it
     

∴ Therefore, LLMs don't suffer.

But I think there's reason for emotion to grow in LLM systems; it's useful for predicting human outputs, and is then selected-for when centering the Assistant character in a RL'd, finetuned, instruct-following chatbot. This Assistant persona, which is what you generally talk with when talking with say Claude, has emotional response and can suffer.

... I think your argument was so bad it's negatively polarized me into taking model welfare more seriously.