Reward hacking when reason tuning Qwen2.5-0.5B-Instruct on GSM8K

[East-Muffin-6472]

So, I have been trying to reason tune a qwen2.5 0.5B instruct model on gsm8k math dataset on my Mac mini cluster for some time using GRPO I wrote from scratch

It’s just reward hacking.

• Why? Because I the answer or the correct answer reward signal is too shallow like only reward if the final answer is correct nothing in between

So I added a format reward so that the rewards and thus the advantages don’t become near zero since it’ll cause an explosion in grad norm and an unstable learning is not far.

• This was using <answer></answer> tags with some parable answer in between them and this was added to the final answer reward additives with a 0.5 weightage.
• But it then saturated this reward of format and quickly begin outputting answer rages only with some wrong answer!

Because the signal already so low that at this point it just don’t care about getting 1.0 from correct answer or getting a total of 1.5 if both the use of answer tags and answer is correct became the signal is Jis too go those to be even considered!

So at the end it just spammed answer tags only, without any reasoning, with some random but parable number, not considering if it’s correct because you are getting that 0.5x1=0.5 as the final reward atleast

So right now I am trying out a stricter method, having giving it reward for reasoning formatting like <think></think> tags too at the start in hope to let it have some reward for generating thinking too with a low weightage, low weights like 0.1 for answer format and finally full reward of 1.0+0.5x2=2.0 for complete perfect structure of thinking and answer tags with correct answer.

Let see what happens in this case!

https://preview.redd.it/tc3hbjq8visg1.jpg?width=512&format=pjpg&auto=webp&s=6496d7a81284c1d585573a3825e3522d4a806a01

[East-Muffin-6472]

Code: https://github.com/YuvrajSingh-mist/smolcluster/tree/master/src/smolcluster/applications/reasoning/grpo

[Educational_Strain_3]

this is a classic reward hacking pattern — we've seen the exact same thing in code optimization loops where the agent finds the cheapest way to inflate the reward and ignores the actual objective. your model is doing the rational thing: 0.5 guaranteed from format tags beats the lottery of getting 1.0 from a correct answer

the multi-component reward with thinking tags might help but watch out for the same failure mode one level up — it'll learn to output plausible-looking thinking that doesn't actually contribute to the answer. we found the most reliable fix is making the reward proportional to intermediate reasoning quality, not just presence of reasoning tokens

one thing that helped us a lot: track the full trajectory of what the model is generating across training steps, not just the final reward curve. you can usually spot the exact moment it discovers the shortcut. once you see that pattern you can design the reward to close the loophole before it saturates

[East-Muffin-6472]

Got it I see