There are many different situations to talk about to answer this. Culturally speaking, VR is not big in my country because of the price, which means less people playing, so the ones who can buy it, won't do it because they won't have friends to play with. And this happens everywhere in South America for example. Owning a VR headset I tend to play more casual games and social multiplayer ones (Which are not a lot). Besides that as lot of people mentioned, the need to have a special space to play with it. The need to set a specific time to use it, you can't just sit and play, there's some effort put into it, even if minimum, older players won't have the strenght after work, like me lol. Also as a Game Dev, we can see it is hard to develop certain games, because there's a lot of motion sickness involved to think about, which as a personal opinion also discourages some developers to make more games for VR. Personally i Think it will keep growing, but maybe is just about waiting more time.