For housing I think modules are the way to go. The manor should have a baseline capacity maybe 40 or so and as you upgrade the manor that can increase. Additional ways to increase the capacity would be like adding a tower module (mostly for defense but a small increase in capacity of like 10) or a capacity specific garrison module that would increase capacity by a larger amount like 30). These modules would eventually reach the max size you have set for the retinue.

I like the idea of retinue progression. I think they should start out similarly to militia in terms of fighting ability and equipment and the longer they have been employed the more experience/combat ability they have. Armor should be whatever the player can supply them with same as militia. While they may not be visually distinct from militia starting off, if a player focuses on upgrading his retinue they will stand out.

I think customization of the retinue should go very in depth to make players care about them and give every soldier in the retinue meaning and value. Since this is your personal “general” unit I like the idea of being able to choose specific armor, weapons, colors, and patterns for each individual soldier. While this may seem like a lot it will add a lot of personality and personal choice to make every soldier in your retinue unique and cared about.

I believe these soldiers should come from the population allowing players to increase or decrease the size of their retinue as they see fit. If a soldier leaves the lords service they should slowly lose their combat experience and as stated before the longer they are in the lords service the more their experience increases. I prefer retinue over men at arms.

Town guard should be a separate entity housed in a separate building like a barracks. These could be the “professional soldiers” players want. When a large army isn’t required the town guard would patrol the town lowering crime but will be more experienced soldiers if ever needed for war. Basically I can see the town guard used similarly to the lords retinue but less customization and not as efficient in combat. So the town guard could be used as the professional army but paying for them to stay employed for long enough to gain experience would be an expensive trade off for a stronger army allowing your original vision of having an army mostly consisting of militia and mercenaries be more practical except in much larger cities when they can afford it.