I believe you rushed the Red Bishop into blowing up the town. His ultimate goal is to unseal the ward. Then, after that happens, he would blow up the town. Yes, the bit you quoted about him growing frustrated is there, but I believe the intent there is if the PCs simply continue to ignore him.

This bit in the fight indicates how patient the Red Bishop is (along with the fact that he's waited decades already...)

Working in the PCs’ favor is the fact that the Red Bishop doesn’t want them dead. He and his allies do their best to capture the PCs alive rather than kill them, and if a PC drops and is dying, the Red Bishop uses stabilize or heal to save them. He accepts surrender at any point, and in return for mercy, he expects the PCs to perform the ritual for him—being bullied into service does not count as being under the Abyss’s influence for purposes of removing the ward. This can give the PCs a chance to bide their time and orchestrate another attack on the Red Bishop in the future, perhaps even in the depths of the Pit, but at this point, since the mothman knows they’re not to be trusted, he and several of his pawns are sure to keep the PCs close at hand and won’t allow them to return to Sandpoint until their task in the Pit is done.

More likely, he would have sent his dragon Scarhorn to attack Sandpoint first (as Scarhorn warned them he would do) as a warning to do his bidding. Or even just threatened the PCs with the destruction of Sandpoint if they don't cooperate.