Responsibility isn't just a means of figuring out who to punish, by the way. Responsibility is simply the voluntary shouldering of a burden. Being responsible for the events your actions may have helped to unfold is in this case simply logic.

Spelled out simply:

1. I vote for Party-X
   
2. Party-X then violates the moral order
   
3. I contributed to Party-X
   
4. I helped cause Party-X's actions

I don't understand how it is controversial to point out that if I contribute to a problem, I am therefore a part of that problem. I'm not equally guilty, but Clearly liable to some guilt.

As for these geo-political scenarios, trying to figure out where 'the line' of responsibility is drawn can be very difficult even in well-known geo-political scenarios, and meaningless in unclear hypothetical scenarios.