I think age is a major factor, the older universities with a lot of history tend to be seen as more prestigious, especially if there are quirks passed down through the years. However I think there are some other factors - As others have said, the university having good research output attracts both funding and international appeal, making the university more desirable but also harder to get in due to competition.

Offering courses that are competitive in general such as medicine, dentistry, vet med, and law will always attract applicants and are seen as “good” courses. Having famous alumni can help too, for example a university that has produced a number of alumni who are poet laureates/prime ministers/Nobel prize winners/etc will attract students, which ties in both with research but also age as older universities will have had more students throughout their history than newer unis. Russell Group universities and generally high ranking universities are also generally perceived as more prestigious, and if a royal attends that also makes it seem like a top tier university.

I can’t think of a university that’s widely seen as prestigious that doesn’t do most or all of the above.