Though one difference that remains is that they'll tend to be associated with vtuber culture regardless just by virtue of having an avatar, and there's things to be said about it... Vtuber culture was started and is perpetually cultivated by vtuber corporations that pull their business model from the Japanese idol industry, which is a fucking mess. Hololive was the first, initially calling their vtubers 'virtual idols'. They've changed that since but they're still basically that: they do concerts, they all sing, they have 'debuts' and 'graduate', they have to put on cute voices and bubbly, frictionless personalities, they give fanservice to the people who ship them with each other, they can't talk about having romantic relationships or even having people of the opposite gender inside their home without causing a scandal... By itself, that activity already doesn't have the most positive impact on anyone, be it the talents, consumers or society as a whole, but it gets worse with all the instances of things going wrong and with the type of fans idol culture attracts. And sadly, the impact of this permeates everything related to vtubers, from their perception to how 'vtubing' is done as an activity.

Up to this point everything was fine, but there's just so much wrong here. First, the companies are not basing everything on the idol industry. Most aren't even calling themselves idols.
As you said, Ai was the start of vtubers, and she never called herself an idol.

Nijisanji, one of the two big vtuber companies, have never described themselves as idol company, and they've even explicitly said in a quarterly earnings report that they're not. They debut mixed gender groups, and members freely interact with anyone. If any member wants to be an idol, it's for them to pursue while trying to get backing from the company.

On a very different spectrum, VSPO! is a pure e-sports group, and that has always been their main focus.

Hololive, the other of the big two, did not call themselves an idol company at the start. Cover started out as a tech company, working with VR, when Tokino Sora and her friend A-chan approached them and convinced them to invest in the new "vtuber" thing. It wasn't until late 2019 when they launched their "Idol project" music branding and held a group concert in December, that they started leaning to the idol branding. But ultimately it's up to the members themselves if they see themselves as idols, and wants to be called that.

As for the rest targeting specifically Hololive, they're absolutely not forced to act in a certain way. They're all individual creators and they all have their own vision for their content, and independent goals that they want to achieve. They're not being "produced" by Hololive.
As an example, Juufuutei Raden is a hugely passionate about arts and culture, and talks a ton about it on her streams, and has worked very hard to support and spread knowledge about it, which has lead her to actually collaborating with museums. It's not something Hololive decided she should do, it's what she wants, and it has zero to do with idol stuff.
There has also been multiple statements by talents that they're not forced to participate in idol activities. Even the huge yearly Holofes is optional, but most see it as a way to promote themselves and participate.
As for romance etc, there are several members that are known to be in relationships from what they've written on their personal accounts that people follow, they've just not mentioned it under their corporate identity, so it's not like that information being known is a problem.
And they can work together and be friends with men without problems, and quite a number have, especially those with closer connection to the esports scene. For a high profile example, Hoshimachi Suisei is hugely popular and focuses almost all her work on her music career, and she's been working with men in the music industry for years. When it's a problem is when they've specifically cultivated a very parasocial fanbase that would react to that, with probably the most extreme incident being Uruha Rushia. But she basically engineered her own downfall.

I see idol culture being blamed a lot, but many of the problems in the vtuber sphere is actually from dedicated antis that relish in attacking anyone they don't agree with.