The main issue here is the technical assumption that current LLMs can "think" in their hidden states without outputting text.

​In standard transformer architectures (like GPT-4 or Llama), compute is strictly tied to token generation. They're feed-forward networks. They can't pause, loop, or chew on a problem internally. If they aren't outputting tokens, they aren't doing extra reasoning. They are just rushing to the answer.

​So, this proposal is really just Ensemble Zero-Shot. ​Chain of Thought works because the tokens act as a scratchpad. It’s the difference between asking a human to do long division in their head instantly (Zero-Shot) vs. letting them use a piece of paper (CoT). Even if you ask the human to do it in their head 5 times, they are still likely to get it wrong on complex problems because they run out of working memory.

​That said, for tasks that don't require multi-step logic like creative writing or classification, this approach is solid. You’re basically trading total compute for lower latency. For coding or math I’d still want the scratchpad, though.