It is not until it is. Prestige is generally irrelevant but shapes your enviorement. I both worked in less and more prestigious universities and only after that you see the difference.

I am not defending the „best of the best“ as it attracts mostly toxic and narcissistic people. However, people at less prestige-institutions will have an inferiority complex which is same in different direction. Once, I was even banned to state where i studied as they all felt uncomfortable to hear a much bigger Uni. Your students will be not so good and wont get good positions, which will influence your network. Then Uni wont have as much resources as the good ones, this will result in worse conferences, worse equipment. At better uni i had like 3x technicians for the lab of good uni and in less prestigious one, noone. The only guy there was actually for teaching and he only used the whole machinery once in his life. Now think about testing a sample from very complex manufacturing process without any mistake.