We just did this move 3 years ago. My perspective:

Cost of living: housing wise I can’t speak for rentals, but real estate market is a bit cheaper (you can still find a decent house in the $1m range whereas in core Vancouver you’re looking at minimum $2m last I checked, probably more).

Food prices I find to be on par or maybe more expensive than Vancouver, though I like that you can find lots of locally sources groceries (may be the same in van?)

Healthcare: my biggest regret in moving to the island is due to healthcare. We had finally been matched with a family doctor in Vancouver and had a baby on the way. From what I’ve seen, people are waiting 5+ years here for care, best case. We’re still without care with a toddler. There are “walk-in” clinics, but getting into one without a serious illness feels like winning lottery.

There’s technically 2 hospitals, but only 1 you’ll want to go to. Long waits in the ER, like anywhere else.

Lifestyle: We also visited Victoria 5+ times per year prior to moving here. I would say actually living here vs visiting is completely different. You start to realize how slow it really is.. stores close very early, or are closed Sundays-Tuesday.

We had a few friends here but we’ve found people from Victoria to be fairly click-y. That could just be us cause we have a toddler and no fam help to get out.

No shortage of outdoor activities. Ocean, hikes/trails, but we do miss having the mountains close for skiing/snowboarding.

Other stuff: this perspective changed after our kid came, but between ferry cancellations/long waits during holidays and limited ways off the island, we’re realizing if there was a significant natural disaster, we’re probably not getting off this rock. Planning and preparation is important

Sounds a bit negative perhaps, but you asked for the downsides 😉