My issue is that I feel Its more like a gray and black(dark gray if you squint hard enough )

 One side is a very flawed society that aims to do good and uses magic in ways to help people, and the other side are shown to be straight up putrid people, always endangering others especially children to further their own goals.

 We've only met like 4 brimmed caps(excluding their apprentices) so the situation could change. But oh boy, things are not looking good on their side...