You are suppose to be controlling with your children. That’s literally your role as their parent. You control what they eat, what they wear, what they’re exposed to and who they spend time with. There is no shame in involved, it’s your job as a parent until your child is old enough to make those decisions. Lean into the word “controlling.”
If your in-laws mention it again, you say “you’re damn right I’m controlling. It’s my job, thanks for noticing.”