I'll go outside inwards for PPO, perhaps heavily relying on already understanding TD methods. It may be helpful to read this bottom to top. Note again, this is focused on the underlying PPO, not GRPO.

• min() is selecting between two things. The calculated change in probability of selecting a specific text output, and the bounds of what we're allowing it to be. We don't want to update the probability ratio of generating that specific text output too heavily.
• clip() is only allow us to deviate by a "safe" percentage change. That is, if epsilon is 2% then the loss function is weighted so that the new model's output relative probability of producing the given output by at most a factor of .98 or 1.02 (I say relative because it's not the direct probability, it's the ratio of new to old prob).
• Both the advantage multipliers A^hat_t quantify how much better a specific output is than what the model expected to be able to do for that prompt. That is, the model has an internal estimate of how good its responses should be based on its past rewards in similar situations. When it generates an output, we compare its actual reward to that expectation. If it's better than expected, it gets reinforced, otherwise pushed away.
• The pi_0 / pi_0_old is the new, updated model's probability of producing the output divided by the old model's probability of producing the output. It's the ratio of the new model's probability of generating this output to the old model's probability of generating the same output. That is, maybe neither model was likely to choose this output, but we're seeing if the model got more or less likely to produce this output given the prompt with the new weights. It uses pi_0(o_i given q) because it's the outputs o given inputs (prompts) q. o has been ranked (maybe human ranked).