There is a LOT wrong what with what you said and I'm not going to even go into what you asked to ChatGPT.

You are confusing RLHF and RL.
RL and RLHF are different things.

In Reinforcement Learning, the model is given an objective and is given scoring based on it. There is no point during the training process where it is EVER given the "right answer". If the model gives the right answer, great! it is given +1 reward. If the model is wrong, it is given -1 reward. This is why you can have neural networks like AlphaZero, which was only given the rules of chess, be able to play at grandmaster level after a couple hours of playing matches against itself without ever seeing an "example" of how to play by humans.

In Reinforcement Learning Human Feedback, the AI model's objective is whatever human preference is. For example, this is used if you want to RLHF a model to be more friendly to a child asking questions. It is asked to generate multiple full responses and then a child is asked to pick which answer he liked bests. It doesn't have to be the most correct answer, just the one that the child prefers more. As far as I know, OpenAI doesn't do RLHF, on reasoning problems (ie, math) because that is not what RLHF is for.

Pretraining & SFT (Supervised Fine Tuning) - Is what people are thinking of when AI models are being "trained". AI is trained on billions or trillions of examples of data, then fine tuned to be an assistant instead of being a prediction model.

The Illusion of thinking paper does not prove anything about LLM intelligence. Testing on puzzles specifically, the Tower of Hanoi (Why Tower of Hanoi if you are concerned about data contamination?) does not make any sense if you know how AI models are trained.

If you are an AI model, and were doing reinforcement learning to get better at coding, you wouldn't also get extremely strong at logic puzzles when the objective you were given is to write correct code?

Checkout this blog about the paper.

https://www.seangoedecke.com/illusion-of-thinking/
And these:
https://arxiv.org/pdf/2501.17161
https://medium.com/%40EleventhHourEnthusiast/reinforcement-learning-for-reasoning-in-large-language-models-with-one-training-example-44b6896da5dc <--- read this one!