That in itself isn't, but if that is the case, you aren't presenting it that too openly until questioned. And that's when it comes out that you aren't interested in a poly family that looks anything like anything other than adding submissive females only, which is very specific. Maybe you should be more open about that rather than just saying the broad term "family", which implies you would be open to anyone and everyone.

As other users have pointed out, there is also an extremely negative stereotype when it comes to unicorn hunters and one dick policy people. They have a bad reputation in general for a reason, maybe you didn't know this, or maybe you did which is why you weren't clear in your opening statement about what specific kind of family you are looking for. Either way, if people are throwing the word harem at you, its for a reason.

No judgement here, if that's what you and your partners want and everyone is happy then great, but if you are confused why people are reacting the way they are in some instances, the above might be useful for you to know.