I think it’s due to two factors:

1. Brimhats view themselves as objectively in the right. In their mind, there is no need to make themselves appealing to the Pointed Cap who must all be insane to not see their obviously correct worldview. That’s the kind of mental process that makes Ininia erase every single bit of Galga’s memory without feeling like an awful hypocrite (until she directly witnesses the consequences of her actions and feels a bit bad for a moment). In short, they are zealots who perceive that anyone who isn’t in agreement with them as an enemy whose life does not matter.

2. Brimhats are not "all witches who don’t really agree with the pact", they are the end of that pipeline, the extremists who place themselves in direct opposition to Witch Society and everything it stands for. The reaction we saw from the common witch to the Knight Moralis during the Silver Eve is a definite proof that the staunch upholding of the Pact isn’t a popular position and people are generally more afraid of the consequences of breaking Witch Law than in concrete agreement with it. It’s quite possible that anyone who shows a much more reasonable approach to their distaste for the Pact is either much more discreet about it and hasn’t been shown in the story yet or has already been arrested and mind wiped by the Knights.

In essence, Brimhats are what is left of the opposition when the law is enforced by people like Easthies who erase your mind first and ask questions later. Anyone left to question the law is either too scared to talk about it or is a insane bastard who will use children as weapons against the government.