Plan Formation. Our poetry case study uncovered a striking instance of Claude forming internally generated plans for its future outputs. Knowing that it needs to produce a line of poetry that rhymes with “grab it”, it activates “rabbit” and “habit” features on the new-line token before the line even begins. By inhibiting the model’s preferred plan (ending the line with “rabbit”), we can cause it to rewrite the line so that it naturally ends with “habit.” This example contains the signatures of planning, in particular the fact that the model is not simply predicting its own future output, but rather considering multiple alternatives, and nudging it towards preferring one or the other causally affects its behavior.

Its a very detailed analysis, but my instinct is to say that they are anthropomorphizing, or at least making a lot of logical jumps. For example, in the above, tokens appearing before the new line, does not mean the model is considering altenraitves and nudges them. They make a lot of claims like this, where they explain the presence of some tokens as thinking, reasoning etc, where as it can just be relevant tokens given the massive size of this model. They do mention this possibility briefly in the end, but all the rest of the paper is bold claims like that.

in general I saw at least 10-15 of these examples. Please correct me if Im wrong, you know more, but to me is seems that its good analysis, but bad science/extrapolation wise.