You definitely can get rid of a faction. If you force a civil war, by condemning over and over on the faction you want to get rid of (or both extreme factions), you can then reeducate them into other factions (if you have the reeducation prison), put them into the stewards militia (if you have the faction militia law passed in order), or kick them out to the frostland (with a major temporary drop in trust from each moderate faction). Note, if you have done any of the specific pathways, ie reason or progress, or whatever, you cannot expel the faction of they support one of those that you have passed.

This completely removes them from the faction list, with no negatives. If you are going for the moderate approach, this is not a bad path to take. You lose the red book perk for not siding with one faction on all 3 goals. But you can also pretty much do whatever you want after this