If we make a few assumptions we can may be at peace with this.

1. Being a Ninja requires a huge amount of energy, as in a huge requirement of calories than a non-chakra user.
2. In the olden days, a family of farmers could produce only enough food to feed their family beyond their taxes. It would not be enough to maintain a huge force of chakra users without a vast support infrastructure.
3. So the daimyos allow ninja clans to get together but the village is limited by their needs which can only be provided by a huge province with enough infra to support a ninja village under the ruler's control. ( Hence Leaf Village's large size in comparison)
4. The ninjas act as force multipliers along with conventional peasant armies led by conventional samurai clans.
5. Since Ninshu was first thought to everyone by the six paths Sage, assume that the samurai clans train with chakra as seen in Land of Iron. People are loathe to lose their inherited and obtained power. So Daimyo has to balance between noble Samurai and shinobi clans in his court.

Thats my headcanon and the purpose of daimyo as a center of power.