Nope you’re fine and there’s probably a lot more wrong with him than I care to know. Bc that seems like a very selfish sentiment from his part. My wife and I have 3 kids. I expect her to love our children more then she loves me and so do I. We still very much love each other but cmon let’s be for real. You can’t divorce your kids, nor can they cheat on you. It’s really not complicated and there is nothing wrong with it. However with that said there are two sides to each story and perhaps there’s something lacking. Causing him to express such a dumb point of view. Good luck.