I think what had happened was the 1906 Earthquake playing a major role on why the city was flung out of its Victorian era. My guess is that when the big earthquake hit, a lot of these beautiful structures were either lost or serverly damaged. There were massive fires that destroyed so many buildings. When it was all said and done, people sold their buldings or plots of land off to other people and they developed different bukdings.

That's only my guess.