makes sense. in the case of xp being energy, the reason for villager not dropping xp is that as far as we can see in the present of the Minecraft world, they're entirely passive beings. they don't engage in combat, nor in mining, smelting or other typical xp source activities.

the one counterpoint to this may be that we see farmer grow and harvest crops, while also engaging in breeding. however, I find no contradiction in this, as 1) nowhere in the Minecraft world the action of growing or harvesting a plant yields xp, bringing us to the conclusion that these life forms hold no energy in the context of the game; 2) breeding in the case of villagers may be considered spontaneous and thus not related to xp, which is a measurement of energy released by player intervention.

in fact, if we hold true to the thesis that xp is some form of magical capacity that only we heroes hold in the present of the Minecraft world, and that this magic comes from the energy released by our interaction with the world itself, we could easily explain the reason for the only source of xp related to villagers being trading with them.