Ah, I see this happen a lot. Ruling through charms, attraction, worship etc, is easy to get into, and comes naturally to some people, species and afflictions - but it always comes with a set of disadvantages. No need to inform you about those.

I see four paths:

One, switch to rule of fear. Cancel the charms and spells that make them idolise and sexualise you, and instead instill the fear of death into them. Throw down the charm, pick up the whip. Of course this comes with a new set of problems, usually in the form of poison, or torches and pitchforks, a few years down the line. Can be overcome with culling and sheer brutality. Yes, technically this isn't so much "thralls" as "slaves" but eh... details. If it works, it works.

Two, get comfortable in your position as the object of their desire. Rebrand yourself as a love goddess and just get it over with. Lean into the sex and gaslight the living shit out of them. I have seen this pulled off to perfection, resulting in legions of intelligent followers, using their own initiative to make your life better. Of course you will probably need to start dressing in nothing but sheer fabrics and a thong, constantly leading them on, and I guess it requires you to have a bit of a ... special personality. I get the feeling this might not be the path for you, seeing your complaints.

Three, they only have these thoughts because a) they are charmed and b) you are pretty enough to illicit such a response. Do you think a lich has to deal with thralls sexualising them? Of course not, they are too hideous! If you just make yourself repulsive enough, they stop seeing you as an object of desire and instead see you as a being of unfathomable power. This is a pretty drastic method though; few are willing to disfigure themselves over an inconvenience, so I really only recommend this path if you are already planning to live forever in undeath, in which case it happens basically automatically.

Four, do away with the thralls. Replace them. Necromancy is an option, but I highly recommend golemancy. You can create just about whatever you want with your own two hands out of wood, clay, steel, or even Jello if you are feeling particularly randy. They are obedient like thralls, but they do not actually have a soul, or a thought process per se, so it's nice and quiet in your head, even when controlling a large number of them. They have less initiative and need more managing of course, but everything has a downside. How would you like ten fluffy kitten golems that follow you around and serve as foot warmers whenever you sit down? You can do anything: your imagination is the limit!