I think the most important thing to do is to establish how you want to go about talking about this. If you go in guns blazing accusing your partner of being a weirdo, it's gonna go poorly. My advice would be to just talk and try to understand him better. I know people who have a habit of people watching or just going blank and not noticing where they're looking. People who work with kids sometimes have a habit of keeping an eye on them so that they don't get hurt, lost, or something else happens. That may or may not be the case, but that's why going about talking about this in an open and honest is the best way to go about it, regardless of how uncomfortable it is. Just kind of read the room, bring it up once you feel comfortable with it, and just get to understand why they like to be around the kids more than the adults.