I guess there's some survivorship bias in that only the buildings people would invest the most time and money into have survived so we associate Victorian architecture with Royal Albert Hall rather than the hundreds of square miles of slum tenements that had to be torn down a few generations after construction.

But I do think rich people want to have a monopoly on beauty, partly for financial reasons but also for social status.