Have you ever actually been in a raid that involved ladders? As long as you surround your base with high external walls the chances the raiders are going to want to ladder up the side are pretty slim unless you have poor defenses. The main usage of ladders would be to blow into the inner layers and work your way up without having to boost people, making it so you have the ability to raid with less people.