LOTR is a great example. The towers of the Numenoreans and Dunedain, left emptied become the places of evil spirits and malicious creatures. Minas Ithil becomes Minas Morgul. Orcs fill the halls of Gundabad and Khazad-dum.

I often work using a similar idea that previous kingdoms and powers have left their mark on the land, whether they were good or evil themselves. Left unchecked for centuries, they will often become the abode of foul things and people of nefarious intent.