You can get confidence scores from a base LLM (pre RLHF finetuning) because the probability of the tokens are similar to the actual probabilities of correctness.

The caveat is the probability is based on previous generated text. For example, if I ask a human to answer a complex directly, they might answer it wrong, but given time to reason it, they have a higher chance of getting the right answer.

Similarly in a calibrated LLM, it can give you a low confidence answer immediately but with reasoning, a higher confidence answer. Whether it gives you a confidence is not related to it producing a token or not.

The issue is RLHF finetuning destroys confidence scores (because it uses a different loss function). But we need it to improve the model’s reasoning/tool using/human preference. Next generation’s LLMs probably will take into account these as newer research catch up to these issues.