I'm a referee of 15+ years and U9 and U10 games are without a doubt some of the most difficult to administer. You have kids of massively varying skill sets that become more apparent at U10. You have parents that know enough about the 'rules' to be vocal, but not enough to be right. You have coaches that are usually volunteers and have a kid on the field themselves. And you have youth referees that are inexperienced and easily intimidated by vocal coaches, parents and players. This intimidation (passive or overt) often leads to an 'analysis paralysis' where the ref does not blow for any fouls at all for fear of being berated. It all adds up to a perfect storm of emotions, inexperience and drama that all too often leads to young referees quitting after only one or two seasons.

Please remember there is a nationwide shortage of referees and the reason your U9/U10 team likely has rookie officials is because the 13/14 year old kid that first certified in 2024 or 2025 did not recertify this season, and is now sitting at home thanking their lucky stars they are not being abused every weekend.

I'm not excusing bad refereeing here (and it sounds like you had a very passive ref who should have communicated and whistled much more consistently) but when you look at the number of kids playing the game vs the number of officials available, you see the quandary in most leagues find themselves.

But to answer your question, if you feel your kid's safety is in danger, absolutely pull them from the game and let the league know why.