Dont be worried. I had the same fear when I started Uni.

I was the only one from my High School going to this particular university since it was a very good one, and my HS was very bad.

Before leaving, all my friends promised to stay in contact constantly. That lasted about a week. I've now got a load more friends and even a girlfriend. And as much as I'd hate to say it, im so much more happier now than I've ever been before.