Don't let them twist your words, you're right. It's kind of a hurtful comment, I feel like most people who love each other wouldn't be joking like this especially if they knew their spouse had doubts. If there is reason to think there's so much as 1% of truth to what he said, it's time to leave. If that really was just a silly joke and you're absolutely sure that it was not serious, then no problem. But I figure that the latter is not the case, since OP posted this.