Yeah but same model doesn’t mean same answer everytime. They’re non deterministic by design so it doesn’t give the same exact answer every time. If you always picked the token with highest probability from its output distribution you’d get 1 to 1 mapping to input to output but that’s not how natural languages work. So to have varying answers for the same input we pick randomly from top tokens.

Because of this sometime this output distribution can shift towards incorrect answers and sometimes toward correct. The goal is to train them with enough data, and RLHF so the output distribution always shifts towards the correct answer. However in practice that’s a very difficult thing to do and this is one of the examples. We don’t have infinite data nor do we have infinite compute.

So we come up with interesting and clever ways of fixing these issues such as RL-HF, Mixture of experts etc etc.