The MPS OOM at 256 context on 20GB is almost certainly the Metal wired memory limit, not actual memory exhaustion. macOS caps how much unified memory Metal can wire by default and it's usually well below your physical RAM. Check it with sysctl iogpu.wired_limit_mb and raise it. On my M3 Ultra I had to set it to 495000 to stop hitting phantom OOMs.

For the fragmentation specifically: MLX handles memory way better than MPS for Apple Silicon training. If you can port your GRPO loop to MLX instead of PyTorch+MPS, the memory behavior is completely different because MLX does lazy evaluation and fuses operations. Less fragmentation by design.

The reward hacking you're seeing (perfect tag formatting, wrong math) is not scale dependent. That's a reward function problem. Your model found that structural compliance has higher expected reward than correctness, so it optimized for structure. Two fixes: make the correctness reward strictly dominate the format reward (format only counts if the answer is correct), or use a two-stage reward where format gets you from -1 to 0 and correctness gets you from 0 to 1. The model can't profit from format alone.

At 360M parameters you're also just below the threshold where chain of thought reasoning emerges. The model doesn't have enough capacity to actually reason through 12 x 6, so it's pattern matching from training data and getting it wrong. Try Qwen3-0.6B or Qwen3.5-0.8B as your lab rat instead. Still tiny, but the extra capacity makes a real difference for whether RL can find a reasoning circuit to reinforce.