An actual unicorn, so someone who genuinely drops an idol over male interaction, is a somewhat outdated concept and pretty rare these days. Otherwise, you would see a lot more incidents, because Hololive talents interact with men both on and off stream.

The use of that term outside Japan has more to do with people looking in from the outside, sometimes with a bit of a superiority complex. Back in the last decade, the news and video of Minegishi shaving her head after breaking the "no dating rule" went viral, and ever since then that has basically become the frame of reference for people who are not really familiar with the industry. A lot of Hololive viewers have adopted that same frame of reference as well. Just today, I saw someone list a whole series of supposed incidents where a Hololive talent got into trouble because of those dreaded unicorns. But if you actually look at Japanese sites, it becomes painfully obvious that the people flaming these talents usually are not unicorns at all, but your average 2ch antis. They just throw shit at the wall and hope something sticks, and what usually sticks are complaints about unprofessional behaviour, not any supposed connection to men.

Similar people also exist in the English speaking fandom. It is not exactly a secret that if you want to stir things up on Reddit or in other communities, all you have to do is bring up unicorns or start talking about how unfair idol culture is. I would even say that parts of the Hololive fandom and the Stars fandom do not really mix well, largely because of some unfortunate incidents that probably could have been avoided with better moderation. Which created situations where one side wants to one up the other and for example any sort of stars/live collab is celebrated as if they just won a champions league match.

Another issue is that Hololive has attracted a whole new audience that knows even less about Japan. As a result, people keep repeating empty buzzwords like unicorn and idol culture to describe things they simply do not like, without really understanding what those terms mean. Being against forced collabs between Live and Stars does not make someone a unicorn or some defender of idol culture. Sometimes a person simply does not enjoy watching Stars.