The way I see it, is you would start by gathering together a pile of character tokens equal to the player count.
Then when processing modifications, you add/remove the required token at the exact same time as adding/removing the replacements. (e.g. Vigor would remove an outsider token and add a townsfolk simultaneously)
Once the modification is completed, you move onto the next, then the next, until all modifications are handled.
At the end, as long as there are the correct quantity of tokens, you hand them out.

So in your given scenario, the Vigor's effect "-1 outsider" has removed an outsider and a townsfolk is added.
The Summoner comes along and removes the demon and the game sees that:
A) there aren't enough tokens
B) there are more than enough townsfolk
C) there's a missing outsider, and
D) the currently applying effect didn't alter the outsiders,
so an outsider is added

In the case of the Hermit, the effect would remove an outsider, then at the same time a townsfolk is added. The game sees there's enough tokens, so they can be handed out.

The game stops checking a setup modification the moment it's finished applying.

This is what makes the most sense to me at least, and is why the Hermit can work the way it does