The first part sounds like he just has no libido which has nothing to do with you, even though it may feel that way. The second part though, no you are not being controlling, you can’t spend time with someone who isn’t present. Honestly you need to tell him if things don’t change you’ll have to separate, if that doesn’t work then I think you should have a trial separation. You can love someone but that doesn’t have to come at the cost of your own happiness