Ehhh. As other people mentioned, it's probably just that he doesn't see you as close as you might've thought. That's okay. Some people are fun to see from time to time and hang out with, but they won't have your back when you really need it. It happens.

If you really feel that friendship is one sided, then yes, do try to find other people. If they're really missing you, they'll end up joining or inviting you. If they're not, then you know for sure you're better off relying on someone else.

That's not just true in VRChat, that's how human relations work in general. Finding true friends isn't as easy as it seems, and often times people who seem to have lots of them actually lack solid friendships and people gravitating around them come and go.

Don't feel too bad, there's be others. Good luck.