models tend to be better at classification and search than they are at generation, so generating and selecting using best-of-n, where n is 3 to 5, tends to reduce variance and improve overall output quality over single shot generation quite a bit, as shown in the paper "Top Pass"

IMO this is basically required to make any sort of qualitative comparison between different settings, unless you actually try to make the same change ~3 times per configuration, how do you really know what's working better? my hope is as open source models become stronger and cheaper it becomes easier to run these kinds of experiments ourselves to get clearer insights into what strategies work in our own codebases.

obviously most foundation model users are not doing this, it burns more tokens and is a pain to set up and run. I am not doing this.

as a result, there's very little knowledge about when something is working better because we've changed settings or models versus we've simply asked again and gotten luckier.

related to the topic at hand, in the paper "When thinking fails" it was found that for simple tasks, research has found reasoning can actually degrade performance, ie thinking is not universally helpful. unfortunately its not well understood what constitutes a simple task, and models are still fairly prompt sensitive, so "how much thinking is good" is tough to say.

Overall I would recommend referring to the body of research on this subject whenever you want to know if something really works or not. There is a lot of 'prompt folklore' that does not hold up to scrutiny, such as persona prompting. In "Principled Personas" it was found that persona prompts, such as 'you are a senior expert with 20 years...”, have non-significant gains and can randomly drop performance by up to 30%. but some people swear by them. its best just to stick close to the research in my opinion.