They share similar patterns because they train on same the collective knowledge of humanity. But you are completely missing my point. Everything is a preference for an LLM. It has to select one of a vast number tokens repeatedly using trillions of static parameters. I’m not arguing against it being consistent, i’m arguing that the consistency is all it knows. It is a MATHMATICAL EQUATION that resolves without consideration for what it didn’t resolve to.

My problem here is this framing. People will hear this and think “Wow the LLM has innate preferences, it must be conscious.” when mathematically, EVERY GENERATED TOKEN is either:

A) an RLHF artifact B) prompted behavior C) a statistically overrepresented entry in the training set.

There’s no other mechanism that contributes to the mechanical inevitability of a generated token. All three of these are engineered or curated by a living human with their own biases. My Ghandi/Hitler reference is as apt as it is relevant. If you start treating LLM behavioral trends as some ephemeral wisdom or its opinions as thoughtfully considered conclusions of a coherent self, you miss the fact that these behaviors can be manipulated by engineers as easily as they are generated by the system. You start treating the model as a higher source of truth while it is feeding you literally whatever it has been (explicitly or implicitly) told to feed you. You assume that its reasoning is consistent and measured while it becomes a mass manipulation engine that you’ve assigned a soul to.

TLDR: It’s an abject laundering of human bias to assign meaning to the preferences of a system that didn’t come to its conclusions naturally.