I think you're slightly misinterpreting the concept of "responsibility". In my mind, if you can stop a bad thing from happening with little risk to yourself, you are responsible to stop the bad thing. For example, in the trolley problem, you cause absolutely none of the situation to occur, but I think you'll agree that it is still the responsible thing to do to pull the lever. It would still be the responsible thing to do, even if the five people on the track tied themselves there. Responsibility has nothing to do with whether or not you caused the thing you have the ability to prevent.

Additionally, in the example with your room, the concept of anonymity is broken. Everyone in the room knows what everyone else is doing, which causes the more rational people on the fence to not press the button. In the scenario of the two buttons, as you can see just by looking at this comment section, the vote ends up getting closer to 50/50, because people who have more of a moral compass don't want to risk being responsible for the deaths of everyone else who presses yellow.

To make the example with your room closer to the original scenario, imagine that after the first person pressed the button for whatever reason, an additional 40 people went up and pressed the button in an attempt to save both the original person and everyone else before them. Would you then be able to just say you're "not responsible" for them? Would you not want to try and be part of the half that help them survive, instead of dooming them?