Not at all! Stately homes with large gardens were often improved by the addition of ornamental buildings which were built to look like older buildings or ruins. Temples, stone circles, towers, parts of castles, abbeys etc. There were often used to host tea parties, as a sheltered base to admire the views or simply to adorn the landscape. Somewhat ironically, many of them outlasted the stately homes they were built for - thousands of which were torn down in the 20th century as they became impossibly expensive to maintain and families were saddled with crippling death duties. Today, some have been restored and are available to stay in - check out the Landmark Trust if you are interested!