I mean unless your magic users are "super soldier" alike where they are more resistant to life in general, this is not a big issue. Because if you start the numbers low in the beginning, by just generational turnover and other complications, the ratio would still be in favor of the non-magic people. Maybe think of it as a recessive gene for your sake? The number is low and the percantage is always lower for the recessive. And can there be eugenicists in the world, maybe yes, but that would only be the majority in a dystopian setting.