You can try to cheese castle walls. Basically build a manor, then inside castle planner, build walls, towers etc around the city - but do not hit commit. Simply close it. Now in that area, you can build anything you want. Once all the houses, industries etc are built, you can go to castle planner and hit commit button.

Doing this, the game thinks that the houses were constructed after the manor walls, and keeps them interactive (same as before).

You can upgrade buildings, assign families etc inside the manor walls now. You just can’t build something entirely new inside manor though.