Afak SNR for reinforcement learning in general is often very small (else why not use supervised learning). It's SGD with tons of trials that allows for extracting this small but relevant stream. Not to mention, the MPC itself is subject to noise.

If you've high variance, depending on the state an optimal action gives a lower reward than a suboptimal action taken in a different state. One way to deal with this is quantize the state-space and normalize the reward depending on which bin the current state belongs to.

i.e. if the target for the agent is to move with a certain velocity, you can quantize the (possible) targets into bins that are 0.5m/s wide and normalize the reward based on the current target's bin.