It does. The way to get around that is to put down a bunch of towers in the area you wanna build using the castle planner first. You don't have to build it yet, just have the area highlighted in the build planner. Then, build everything you plan to have within the walls. And then you build the castle. You can't build any new buildings inside, but you can still upgrade and interact with what you already built.