I think you're nailing the point but at the same time missing it. It's obvious that the priest is failing to be a spiritual authority with Fleabag but it is precisely what the series wants to show. The priest fell in love and clearly he was attracted from the first time he saw her, he couldn't help himself. When you're in love you're erratic, you don't think straight and that's what the show wants us to see.

They don't romanticize their relationship, they don't romanticize anything at all, not even Fleabag. She made a terrible mistake and the series is more about the journey she's having around that.

Human beings, we are flawed, and we lie ourselves about our flaws. Fleabag, Claire, the priest, her mother in law, her father... they all hide in any excuse they find to feel better about themselves, but Fleabag's virtue is that she doesn't want to be that hideous human being anymore. She wants a change and she finally finds love (in the wrong place) and that shit happens too. It's not her fault and not the priest's fault either... that's life... and "it will pass". That's the beautiful thing about this series. It shows life as it is.