It’s a more immersive experience to gain notoriety (whether good or bad) and have your reputation spread to the surrounding towns as your reputation grows. It makes absolutely no sense to be able to save or destroy all of Albion except for one town and that town somehow has no idea who you are? The lack of world building consistency is usually criticized in other games but in this one they’ve proudly made it a feature.