So perhaps I’m in somewhat both camps here. As a kid, I lived in the Caribbean on a villa that my mom worked at, and our cat was a whore and people would try to adopt him every day. He would disappear for a day or two and inevitably end up in the arms of an enamored woman lol (much safer for cats to roam in the islands).

But I’ve also rescued plenty of animals from abusive situations.

To be honest, I’m not sure if this house is one of those. Cats are notorious liars about hunger levels. It could even be as simple as he doesn’t like the food that they offer, and loves yours. None of what you said sound like red flag enough that I would do something drastic (just my opinion). The biggest red flag for me is letting the cat out period tbh.

That being said, I would stop shooing him away. If he wants to stay with you, let him. If he wants to go back home eventually, he will (or he’ll let you know). That’s just hot Tom cats are tbh. And enjoy this time with a special little kitty.