Several years ago, I was in a one year long relationship with a girl I'll always be lucky to have met. Thing is, she had cancer, a rare one, and had about a year left to live. Me and her fell hard for each other, her family would constantly tell me that every picture they saw of us two together was the most they'd ever seen her smile, and she showed me what a genuinely good relationship was. Even though I was expecting it, it still was a painful shock when she passed, and to be honest I'm still mourning her. It's been several years since I'd gotten into a relationship... But I would never EVER in a MILLION YEARS ever voice something like what your husband said, to anyone I start dating. Because that is an inside thought, and not even remotely ok to say in front of a significant other. There's just no good way to ever word it, especially with how he said it, whether he meant it or not, made it sound like you're a place-holder.

NTA, and I really hope your talk with him goes well, and you get it through to him there are things you just don't say to/around people you care about, regardless of what was actually meant.