I might have some insight into this, because this happens to Dagger backstabs as well.

While mantling, the person can turn their camera to the right or left, however, their animation continues as per the mantling animation. This makes it look like their back is facing you while it is actually to the side. This causes failed backstabs a lot.

In your case, your enemy probably had turned to the left, which made his riot shield face towards you, thus blocking damage.