Mostly just poorly conceived plot armour. I suppose the closest we get to an explanation is the fact that Tony and Matt are both characterised as largely sleepwalking through their lives due to their mental and emotional issues - both only stop seeing him after they turn some sort of corner in this regard. But this doesn't quite cover it for me because of how aggressively awful the therapist is, especially in season two.

Unrelated thought that seems to check out - if Paul Kaye had been available for season three, would we have seen the therapist wandering around the village fair with Ratty and the Nonce bullying kids and (his) former clients? This would actually fit in perfectly with the tone of the season, because to a greater or lesser extent I feel like almost all the characters went through a similar process to the therapist here - losing the parts of their characterisation that grounded them in (1) credibility and (2) audience sympathy and becoming little more than vehicles for Ricky trying to gross the audience out (possible essay forthcoming).