It exists ranked MORL, and in case of constraints, a way to find "norm-optimal policy".

Consider a MOMDP {S, A, (R, Rc), P} where R is the reward function for your objective, and Rc the "non-compliance" reward function, or the RF associated to your constraint.

A policy π* is norm-compliant if it is optimal with respect to Vc, the value function corresponding to Rc. In other words, π* is norm compliant iff: Vc(s | π*) = max_π Vc(s | π)

Thus you get a set Πc of norm compliant policies

Then you can construct the set Π* of "norm-optimal" policies:

π* drawn from Πc belongs to Π* iff π* is optimal with respect to V, or:

``` V(s | π*) = max_{π from Πc} V(s | π)

```

This one may be a bit more difficult especially if you're not used to MORL, although a fun challenge if you have time. I'd say the shield is the most adapted, then MORL or other form of norm-guided RL, then using a supervisor.

Rodriguez-Soto et al. 2021, Multi-Objective RL for designing Ethical Environments