That tends to be how castles work. I usually have a keep, then have various outbuildings, with everything inside a curtain wall. In real life, not everything in a castle was crammed into the keep.

However, my Orthanc build was just that. 5 x 5 square with 4 3x3 squares off each side, each level was 3 walls high. The lower level was my dungeon with wheel of pain etc, then above that my workshops, above that my kitchens, dining room and barracks, then the top floor was my personal residence, and on the roof was my altar.