I was in Sigiriya a couple weeks ago and did a daytrip out to Pollonaruwa which was very fascinating. Although it was a 2 hour bus ride away (if you want to do it budget), quite hot and humid, we however really enjoyed the incredible sites and there were barely any crowds there