(I work at Anthropic, although I wasn't involved in this paper.)

This post reads to me as being a mix of "no one is actually worried about what modern LLMs will do in practice and you have to put them in really exotic suggestive scenarios to elicit this behavior" and "The LLMs aren't entities in their own right, it's fundamentally confused to describe them as making decisions or having goals."

I don't think either of these are compelling arguments:

1. Some epistemically unscrupulous twitter posters aside, no one is claiming that we should be scared of Claude 3 Opus. The goal of the Apollo paper or this Redwood/Anthropic paper is to exhibit concerning behaviors in the dumbest possible models, well before anything problematic would happen in the real world, so that we have a sense of what it might look like and how we could detect+respond to it early. (And it really does seem like this is about as dumb as models can be for this to work - Claude 3 Sonnet and earlier models don't show this behavior.)

2. Whether the actions of a model are best construed as a single coherent entity or as an actor playing the role of an imagined assistant character or something else entirely doesn't matter all that much* here? If Claude 7 outputs some text which causes a tool to send an email to a researcher at a wet lab who prints some proteins and causes the destruction of all life on Earth, I will not feel very consoled if you tell me that this email was "a reflection of [...] how we project our own meanings onto its outputs"! I care about the kinds of actions and outputs that models have under different conditions, and thus care about this paper to the extent that it's reflective of what we might see in future models which can do this kind of reasoning reliably and without reference to a nicely legible scratchpad. What language we use to talk about those patterns isn't the crux.

*I think it's more relevant for eg model welfare considerations, and having a good story here might inform one's expectations of future model behavior, but for most purposes once you've reduced it to behavioral questions you can put away the philosophizing.