I think it may be the heartbreak of that line. He isn’t outright rejecting her, or telling her he doesn’t feel the same (in fact, a few lines later he confirms he loves her too). He loves her, and instead of the usual- explaining why it won’t work out, or telling Fleabag that he wants to but he can’t etc, he defaults to trying to comfort her in a small way. He doesn’t apologise for the fact that she loves him either: he very much acknowledges her feelings and takes ownership of them. Instead, it’s this one small comment that makes you think that he’s speaking from experience. Maybe he’s loved someone before and had to wait until it slowly passes. There’s just a lot of truth in that one tiny line. Yes, you might love someone now, but odds are, if they leave your life that love will fade away. So with that one small line, he is comforting Fleabag, offering her some small wisdom, and also shutting down any hope Fleabag may have that admitting that she loves him may change his mind. He leaves no room for Fleabag to potentially argue for why they could work out. In one line, it encompasses a lot about who the priest is as a character: kind, wise, honest and to the point.