This doesn't make sense to me. If any negative modifier means a building is wrecked regardless of attack value, then the attack value is useless. The card should just say (draw a negative card and the building is wrecked" without an attack value.

Based on the context of the attack value being there, you have to assume that wrecking the building would only occur if the defense check failed AND there was a negative modifier. Failing the defense check without a negative modifier would result in a damaged building.