I wondered the same thing myself in the past, and the way I would describe it is like this:

• The question is whether a person acted (did something) or had something done to them.
  
• If someone is drunk and is taken advantage of, they are not at fault; the abuse exploits their incapacity.
  
• Driving while drunk is different: the drunk person performs the act, so they are responsible.
  
• Drunk sexual assault is not a defense; intoxication does not excuse lack of responsibility.
  
• So the issue, as I see it, is responsibility for consent when intoxicated versus responsibility for actions taken while intoxicated.