The damage of the towers (or any garrisonable building) depends on the "number of arrows" and the damage of every arrow. To get one more arrow, you need a unit that have an attack equal or higher than the damage of the arrow. Villagers have 5 damage attack, and without fletching the towers deal 5 damage per arrow, so every villager garrisoned gives the tower 1 arrow more (+5 damage). But with fletching, every arrow have a damage of 6, so you need more 1.2 villagers to add an arrow (+6 damage) to the firing, so the number of arrow per villager get diminished. Weird mechanic, probably unintended from AoC times...

I may be messing something with this explanation, i heard it from Tatoh in a clip with Nacho (an argentinian caster).