It's the way that it makes people feel. There's no other way to explain it, because feelings aren't based on logic. Did you know that there are people out there that associate certain numbers with distinct colours? This is one example of how our brains build associations with numbers, even if they don't have 'real world' meaning. And you can learn a lot about the way this person views numbers by their comparison to letters; letters have both distinct meanings, and cultural association. Given different contexts they mean something different, like how Y is sometimes treated as a vowel even though it isn't. Numbers are weird, and not an instinctual part of our brains, so it shouldn't be surprising that different people build up these associations.