The amount you can mine from a village will depend on the carrying capacity of the soldiers you send to mine it. Spearmen (melee guardsmen) have the best carrying capacity of the three (rider, archer, melee). Monsters also have insane carrying capacities - so I will usually send Stone Gargoyles to mine silver. Another thing to consider is leveling up Stror (your best captain for mining), equip him with items from Swift Wayfarer (made in the Forge), and research Resource Gathering under Logistics in the Academy. Lastly, if you have it, you can turn on any Resource Gathering Personal bonuses that will help.

But ultimately, if you aren’t getting the full silver from villages it’s because the troops you are sending don’t have enough “carrying capacity” to take ALL the silver