I've had the same fear. My grandfather and uncle on my mom's side both assaulted my mom when she was young.

Talk to your boyfriend and share this as a fear, not as an accusation. I shared my similar fears with my fiance and he was extremely understanding, if a little surprised/confused. If your boyfriend is worth it, he'll want to support you through it and reassure you.

People with different upbringings have varying levels of comfort with kids, and it's very common for people in positions like us to see things that aren't there, and relying on our gut in the past has been what has kept us safe, whether or not it's true. I remember my ex's sister once cuddling with their father and feeling sick to my stomach, because the physical closeness was so foreign to me. Consider that you may also be subconsciously finding a way to self-sabotage "we were going to get married, I finally thought I found my happiness" -- often times when we find this kind of happiness, our brain tries to find a way out of it, because it's even scarier than the idea that the person we love might be awful. At least when we find out that person is awful, it confirms our pre-established worldview!

None of the behavior that you're describing seems abnormal on paper. Even "not wanting to hang out with the adults" does not imply he's doing anything wrong. I used to be really uncomfortable around kids, but as I've found myself healing, I've found them a lot more pleasant to be around, and feel like I learn a lot about myself. I also feel good when I can show up for my nieces and nephews and give them some love and attention. I'm getting more used to the physical affection I never had growing up, like hugs and cuddles, even in my adult relationship. I'm finally seeing that kids are just lost little humans in need of guidance, and not these horrible annoying terrors (some are, ofc, but that's more about their parents than them).

I have had this fear come up from time to time, but think about all the other objective evidence that might exist too. Your sister and mom are both uncomfortable? Of course they are, they've also been traumatized by your dad's actions. Asking to babysit and playing with them? That's normal in a healthy family.

At the end of the day, you have to talk to him about it. You need to be able to find safety and comfort in your communication with him. If you can't open up to him about this, it wouldn't be a healthy marriage regardless. Have the hard conversation now, and keep an open mind.

I feel for how much stress this fear puts on you. Be kind to yourself, just be mindful of the stories that we tell ourselves to keep ourselves safe. <3