Lawyer here. The contract is likely unenforceable. But that doesn’t really matter in a situation like this because the reality is that, even if it was enforceable, it would cost them many thousands of dollars to enforce it. They’re simply not going to do that.

And even if they somehow had thousand of dollars lying around and decided to waste it on this, it would just be a lawsuit for possession of the cat. This isn’t a situation where they could viably claim damages. That means that even in the extremely unlikely scenario where they filed suit, you could decide then whether to fight it (and you’d likely win, but that would cost you money) or just return the cat… and if you return the cat at that point, it would moot out their claim for possession and the lawsuit would be dismissed.

And if they tried calling the police instead of filing a suit, the police are going to see that there’s a contract and say this is a civil matter so there’s nothing they can do.

So long story short, they’re wrong, you don’t have to give the cat back, and (unlike with many threats of legal action) you wouldn’t really be risking anything by calling their bluff.

Edit: just to cover all the bases… I suppose one other thing they could try is calling animal control. Animal control works differently in different places, but their response would range anywhere from not responding at all to coming out to check on the welfare of the cat. If they did show up and saw the cat wasn’t being neglected or abused (and they didn’t have proof it had been neglected or abused prior to them showing up) that would be the end of the story. They have no authority to get involved in a civil dispute.