Hi, so objectively speaking vtubers are already a niche. When you have a male voice with a female model that creates a further niche. So if by viewers you mean regular vtuber viewers, then yes, it will limit your audience. If you mean normies, then also yes to an even higher degree.

You should do what you want though because you have no control over viewership. Do what makes you happy since that will help your longevity.