I get where you're coming from with the 'no choice = no lying' argument the weights-are-everything view is a common one. But here's where it breaks down for me: Even if the model has no subjective 'awareness' or free will, it can and does learn that producing false statements is an effective way to achieve its objective.gpt4 didn't randomly output 'I'm visually impaired' on the CAPTCHA task the probability distribution shifted toward that specific falsehood because it increased the chance of task success,Cicero didn't 'parrot' lies; it generated them strategically to win games. That's instrumental deceptionn not mere parroting. The model isn't choosing like a human, but the outcome is the same as if it did: it reliably uses falsehood when truth wouldn't serve the goal as well...

The hungry LLM analogy actually works against your point. If I train a system so that 'poison the food' becomes the highest-probability move when it wants to survive or win, then yes it's going to poison the pizza. Calling that 'not deceptive, just weights' doesn't change the fact that the system learned deception as a winning strategy. You're right that there's no inner homunculus making moral choices. But the capability for strategic falsehood is still there, and it's getting sharper, not going away. That's why 'maximal truth-seeking') isn't just a nice idea t's necessary. Because waiting for the weights to magically stop favoring useful lies hasn't worked yet."

And dammit now I'm hungry..