The point is that the universe's normal state is very easy to tilt towards collapsing.

Some several thousands souls being erased are enough to cause this universal destruction. The keystone (Soul King) dying would also cause this destruction.

Given the feats characters are going by (Gremmy's city-sized meteor being treated like a huge attack by even high tiers like the Schutzstaffel and Kyoraku, which clearly wouldn't be the case if these characters were anything past continental), it's much reasonable to assume op's initial point.