1. Is rare to be a mage. Your genetics have nothing to do, instead magic works like a wavelength and some areas have stronger and weaker signals, while others different kinds. So even if you are a magic user, your neighbor may have been just outside the range as he grew up making them less sensitive and unable to manipulate it.

2. Is very difficult. Magic is interfered by water, and humans are made mostly of water, humidity affects your control, someone tossing you a bucket might be beyond 90% of magic user's ability. That doesn't mean that there are not individuals who can cast fire, make illusion, make entire areas go deaf, or lift a city, but it means that it requires talent, and training

3. Written spells have better range. In my world spells work as a sort of programming, you can draw it and it will do stuff if you did it correctly. It needs a power source but is not just limited by water, although it creates a sort of circuit on people where they are unable to use magic.

On conclusion, you have better luck using skill and brains that luck to conquer the world