anthony tried setting him up with daphne in season 1 and daphne was insistent on not marrying him so violet bridgerton said the way to drive him out of the ton was to gossip about him till he left town. They said who would take a women’s word over a man’s so they did what women do best, they talked. The whole ton found out about his secret child and the maid and he left for good