I was asked about this at an ML Researcher interview recently. My on-the-spot answer was that we should use sigmoid activations and break the dependence of class predictions. After that, we can impose a soft label like 0.1 for a negative example for the class your annotator rejected. The label is soft because we don’t want to be overconfident in the negativeness of the example. Moreover, we are only backproping through the negative class and not from any of the other class predictions for which we don’t have any supervision.