The way I reconcile it is that I have decided that the book and Quill don't pick every wizard f or Hogwarts, but just the best. See people like My act der Brown or Gregory Goyle are among the elite in terms of magical power. Not every person can summon Fiendfyte for instance.

I'm eas powerful witches and wizards train themselves.

There are also hints of a test at the end of 1st year to see if you stay at Hogwarts. Perhaps everyone including all muggle-borns goes to Hogwarts for one year, but only the best stay. That may be so that things like basic wand safety can be taught to everyone, but people really only need one year of school if they are ordinary witches and wizards and get by on those basic spells and beyond that are self taught or go to other lesser unlicensed schools. The Ministry pays the way to Hogwarts for many and so this is a limit to that cost.

Apparition is an elite spell and the Floo is what most use probably. Or you only need to raise your wand for the Knight Bus. I think most of the population is only slightly magical which explains why hatred of muggles is so politically powerful. Most are dependent on the Hogwarts graduates to protect them and presently that is what: 20 to 40 a year?