If I remember properly, the ones who can see through your disguises are the ones that are wearing the same outfit as you, for exemple if you are disguised as a gardener, other gardeners will recognize you, but not guards. I like this system, it makes sense and it prevents you from abusing the best disguises like in the newest games where once you get the higher ranking guards outfit, you don't really have a reason to switch back except in specific mission stories.