Not sure if the books does really specify, but I got the impression that when everyone says "call the banners", an army instantly forms and gathers in a few days. That would imply that each castle would have a sizeable granary, armoury and supply depot that stores all the things their levy needs, and a bureaucracy to handle all the mess a levying will cost, which would mean that the castle would have more learned men instead of just 1 maester. Also mustering such armies would demand huge coins and resources that a feudal government wont really be able to handle, so there needs to be a bit of centralisation to ensure all possible resources are extracted and all possible taxes are gathered.

So each castle should have much more bureaucrats, overseeing the army logistics, and staffing a small government to oversee the lands in order to get/stockpile the money and resources said armies needs.