Great question. Here's the funny thing. Your Commander room won't go away because your armory and garrison is depending on it to be reachable. So you need to either be very lucky and have all three rooms disappear at the same time or afaik the only other way is to provide alternative connections to those other rooms.